Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

macho: speed up the linker by embracing multi-threaded approach #143

Merged
merged 139 commits into from
Jul 2, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
139 commits
Select commit Hold shift + click to select a range
d43ff79
macho: separate adding dyld entries out of Atom.resolveRelocs
kubkon May 25, 2024
38e76ee
macho: simplify scanning relocs logic
kubkon May 26, 2024
4b610f7
macho: write sections in parallel
kubkon May 26, 2024
2679e7d
macho: move updating Rebase size into Rebase.updateSize
kubkon May 26, 2024
2b16311
macho: move adding binding logic into Bind.updateSize
kubkon May 26, 2024
769aaf5
macho: move adding weak-binding logic into WeakBind.updateSize
kubkon May 26, 2024
cbf5be9
macho: move adding lazy-binding logic into LazyBind.updateSize
kubkon May 26, 2024
ebe950c
macho: do the same for export trie
kubkon May 26, 2024
a8d3560
macho: calc dyld info sizes in parallel
kubkon May 26, 2024
99b1aeb
macho: calc data-in-code size in separate pass
kubkon May 27, 2024
b67229f
macho: decouple allocating linkedit sections from writing them out
kubkon May 27, 2024
fbb5388
macho: move updating dyld opcodes to work queue approach
kubkon May 27, 2024
f1e1fb6
macho: move updating data-in-code to worker queue
kubkon May 27, 2024
c85dcd7
macho: rewrite writeAtoms to use work queue
kubkon May 27, 2024
3a67402
macho: dedup some logic
kubkon May 27, 2024
4083c1e
macho: dedup some more logic
kubkon May 27, 2024
edaac31
macho: dedup even more logic
kubkon May 27, 2024
8d11ee7
macho: calc section sizes in parallel
kubkon May 27, 2024
6ad7992
macho: run scan relocs in parallel
kubkon May 28, 2024
4a9b33a
macho: add more tracy hooks
kubkon May 28, 2024
71966bb
macho: write dyld info data in parallel
kubkon May 28, 2024
9f8b97a
macho: make writing symtab more parallel-friendly
kubkon May 28, 2024
c3f2e72
macho: write remaining LINKEDIT sections in parallel
kubkon May 28, 2024
d150528
macho: redo how we create tasks for writing atoms
kubkon May 28, 2024
1b06f51
macho: preinit symbols extra per each symbol that is from alive object
kubkon May 30, 2024
514f29f
macho: put symbol section flags behind atomic fences
kubkon May 31, 2024
b19b714
macho: write synthetic sections to buffer in parallel
kubkon May 31, 2024
3d10615
macho: do not write to a file from multiple threads (yet)
kubkon May 31, 2024
61d50de
macho: write sections while updating linkedit sizes
kubkon May 31, 2024
809fba2
macho: simplify out_n_sect for symbols
kubkon May 31, 2024
2305694
macho: simplify scanning relocs logic
kubkon May 31, 2024
874e3ae
macho: dedup literals in parallel
kubkon Jun 1, 2024
84d32cf
macho: purge file seeks from existence
kubkon Jun 3, 2024
b94149e
macho: refactor logic for validating cpu and platform
kubkon Jun 3, 2024
3ce4a3c
macho: handle multiple build version LCs in input object files
kubkon Jun 3, 2024
6334550
macho: (somewhat) handle "zippered" binary dylib input files
kubkon Jun 4, 2024
0e362b5
macho: put writing errors/warnings behind mutexes
kubkon Jun 4, 2024
a358195
macho: scope atoms to each input file
kubkon Jun 5, 2024
ea7e376
macho: re-enable deduping literals
kubkon Jun 6, 2024
6f83128
macho: introduce Atom.Ref in Symbol
kubkon Jun 7, 2024
6ca8fac
macho: re-enable deduping literals with latest changes
kubkon Jun 7, 2024
a051cfa
macho: dummy symbols for deduped literals can be nameless
kubkon Jun 7, 2024
0c499db
macho: move ownership of unwind info records to each object
kubkon Jun 7, 2024
04d2a3d
macho: fix invalid calculation of output strtab
kubkon Jun 7, 2024
0ca49bb
macho: remove macho_file ref where possible in Object
kubkon Jun 9, 2024
ae8c154
macho: start migrating to object-based symbol ownership
kubkon Jun 10, 2024
7779333
macho: bring back state logging
kubkon Jun 10, 2024
275da48
macho: move ownership of undefined globals and synthetics to Internal…
kubkon Jun 10, 2024
f7a23d5
macho: wip: symbol resolution
kubkon Jun 11, 2024
6b7577f
macho: wip: more symbol resolution shenanigans
kubkon Jun 11, 2024
87407b8
macho: port symbol resolver to new design
kubkon Jun 12, 2024
ed4355b
macho: init defined synthetic symbols in InternalObject
kubkon Jun 12, 2024
351aa54
macho: re-enable converting tentative defs
kubkon Jun 12, 2024
8f8ddee
macho: re-enable deduping literals
kubkon Jun 12, 2024
e5ca168
macho: claim unresolved symbols
kubkon Jun 12, 2024
481f13e
macho: mark imports and exports
kubkon Jun 12, 2024
bb10e45
macho: dead strip dylibs
kubkon Jun 12, 2024
82585f2
macho: wip: scanning relocs
kubkon Jun 12, 2024
69e0029
macho: get relocations going
kubkon Jun 13, 2024
a0f3871
macho: purge symbols from MachO's global scope
kubkon Jun 13, 2024
d374dca
macho: allocate sections, atoms and symbols
kubkon Jun 13, 2024
61ac100
macho: first pass
kubkon Jun 13, 2024
1295fd4
macho: make undefined in InternalObject also actually extern
kubkon Jun 13, 2024
de396e9
macho: update resolver when claiming unresolved symbols
kubkon Jun 13, 2024
156ab7e
macho: remove obsolete Object.init functions
kubkon Jun 13, 2024
653e51f
macho: parse input files in parallel
kubkon Jun 13, 2024
23681bf
macho: slightly improve resolver perf
kubkon Jun 13, 2024
c6b6cd0
macho: clean up
kubkon Jun 13, 2024
060a2e2
macho: partially reintroduce deduping literals
kubkon Jun 13, 2024
c7ca9af
macho: improve symbol resolver
kubkon Jun 14, 2024
4192f8f
macho: correctly merge weak_ref attributes
kubkon Jun 14, 2024
1ed3b04
macho: move claiming unresolved after marking imports/exports
kubkon Jun 14, 2024
895cd64
macho: fix symbol resolution of tentatives in archives
kubkon Jun 14, 2024
c7f0b2b
macho: actually parse dependents
kubkon Jun 14, 2024
221b095
macho: fix resolution of ObjC msgsend symbols
kubkon Jun 14, 2024
ff84b3d
macho: leave a todo for opt when sorting bind entries
kubkon Jun 14, 2024
fa308f0
macho: reinstate dead stripping
kubkon Jun 15, 2024
0653ffb
macho: reinstate boundary symbols handling
kubkon Jun 15, 2024
24a704f
macho: init symbols in dylib when parsing
kubkon Jun 15, 2024
41115ce
macho: reinstate relocatable mode
kubkon Jun 15, 2024
bc21551
macho: fix resolving objc msgSend references
kubkon Jun 16, 2024
785d793
macho: do not compare strings when sorting bind/weak/lazy entries
kubkon Jun 16, 2024
d4b2930
macho: fix conditions for dyld dynamic relocs
kubkon Jun 16, 2024
8455bb5
macho: Trie: clean up wild allocations
kubkon Jun 16, 2024
2d4e031
macho: Trie: generate output byte stream in updateSize
kubkon Jun 16, 2024
9f06f45
macho: atoms: prealloc extras wip
kubkon Jun 17, 2024
5c03e01
macho: thunks: extract relocs scanning logic into a function
kubkon Jun 17, 2024
d916f4a
macho: cache string pos and len for atoms and symbols
kubkon Jun 17, 2024
6ae207e
macho: add more tracy hooks
kubkon Jun 18, 2024
5a3a50b
macho: more atomics
kubkon Jun 18, 2024
16172c9
macho: resolve relocs and write atoms in parallel
kubkon Jun 18, 2024
79050f1
macho: move writeAtoms logic to each Object
kubkon Jun 20, 2024
0c25772
macho: do not call Atom.getCode when resolving literals
kubkon Jun 20, 2024
3897ffe
macho: actually error out after any object unsuccessfully parses
kubkon Jun 20, 2024
4a6570f
macho: put path resolution in its own function
kubkon Jun 20, 2024
7fafb71
macho: put logic for inferring cpu and platform into a function
kubkon Jun 20, 2024
7de8299
macho: convert tentative defs and resolve special symbols in parallel
kubkon Jun 20, 2024
9c813d2
macho: clean up dedupLiterals worker helper
kubkon Jun 20, 2024
b882f46
macho: clean up claimUnresolved
kubkon Jun 20, 2024
bb6a0ae
macho: clean up scanRelocs
kubkon Jun 20, 2024
f0a721c
macho: report error after calcSectionSize
kubkon Jun 20, 2024
9a830bd
macho: report more errors happening across threads
kubkon Jun 20, 2024
774ddcd
macho: measure time to invalidate inode
kubkon Jun 21, 2024
b9bc590
macho: move claimUnresolved for -r into Object
kubkon Jun 21, 2024
211aa82
macho: lock stderr where required
kubkon Jun 21, 2024
ae734f3
macho: relocatable: split section size calc across threads
kubkon Jun 24, 2024
3a6e542
macho: relocatable: calculate symtab size in parallel too
kubkon Jun 24, 2024
efefc85
macho: relocatable: calculate data-in-code size in parallel too
kubkon Jun 24, 2024
8273992
macho: relocatable: do not use stream for writing out relocs
kubkon Jun 24, 2024
f0d3a19
macho: Atom.getExtra is always defined hence no null needed
kubkon Jun 24, 2024
5f63ce6
macho: relocatable: write atoms and relocs in parallel
kubkon Jun 24, 2024
f45fdf4
macho: relocatable: write eh_frame in parallel
kubkon Jun 24, 2024
bd0d821
macho: emit compact unwind records in parallel
kubkon Jun 25, 2024
54600a4
macho: truly calculate compact unwind size in parallel
kubkon Jun 25, 2024
bf3e83e
macho: remove obsolete helper for writing out data-in-code
kubkon Jun 25, 2024
4c10683
macho: relocatable: sort relocs in parallel (per output section)
kubkon Jun 25, 2024
23b52d5
macho: relocatable: write symtab in parallel
kubkon Jun 25, 2024
76597c2
macho: relocatable: create named temp symbols when deduping literals
kubkon Jun 25, 2024
d71468e
macho: skip creating internal object in relocatable mode
kubkon Jun 26, 2024
3fc04c5
macho: remove obsolete mutex on Relocation
kubkon Jun 27, 2024
3490a67
macho: add build options for tsan and single-threaded
kubkon Jun 27, 2024
52adef7
macho: create nlists for each literal upfront
kubkon Jun 28, 2024
f733efd
macho: refactor dedup literals in Object
kubkon Jun 28, 2024
7994300
macho: refactor dedup literals in InternalObject
kubkon Jun 28, 2024
eacffb5
macho: implement missing logic for aarch64
kubkon Jun 28, 2024
ca6111e
macho: simplify reloc parsing logic for literals
kubkon Jun 28, 2024
e0679e1
macho: remove obsolete relocs flag in Atom
kubkon Jun 28, 2024
d7415d8
macho: remove obsolete unwind flag in Atom
kubkon Jun 28, 2024
d1cd935
macho: remove obsolete literal_pool flag in Atom
kubkon Jun 28, 2024
c736940
macho: remove obsolete thunk flag in Atom
kubkon Jun 28, 2024
cd34489
macho: refactor literal indexes naming
kubkon Jun 28, 2024
210d88f
macho: remove obsolete check when updating lazy-bind data
kubkon Jun 29, 2024
824dd2a
macho: generate lazy bind opcodes only if __la_symbol_ptr exists
kubkon Jun 29, 2024
249a57d
ci: remove macos-11 as been deprecated by GH
kubkon Jun 29, 2024
f068eb7
macho: rewrite Trie to use multi-arraylist
kubkon Jun 29, 2024
2dd97ac
macho: trie offset does not need to be an optional
kubkon Jun 29, 2024
b22eebf
macho: refactor Trie.Node
kubkon Jun 29, 2024
c62e005
macho: fix deadlocks
kubkon Jun 30, 2024
5334972
macho: redo deduping literals yet again
kubkon Jul 1, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 0 additions & 11 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,17 +34,6 @@ jobs:
- run: zig fmt --check src
- run: zig build test -Dhas-zig

macos_11:
name: Test macos 11.0
runs-on: macos-11
steps:
- uses: actions/checkout@v3
- uses: goto-bus-stop/setup-zig@v2
with:
version: master
- run: zig version
- run: zig build test -Dhas-static -Dhas-zig

gcc_musl:
name: Test gcc with musl
runs-on: ubuntu-latest
Expand Down
6 changes: 6 additions & 0 deletions build.zig
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ pub fn build(b: *std.Build) void {
};
const use_llvm = b.option(bool, "use-llvm", "Whether to use LLVM") orelse true;
const use_lld = if (builtin.os.tag == .macos) false else use_llvm;
const sanitize_thread = b.option(bool, "sanitize-thread", "Enable thread-sanitization") orelse false;
const single_threaded = b.option(bool, "single-threaded", "Force single-threaded") orelse false;

const yaml = b.dependency("zig-yaml", .{
.target = target,
Expand All @@ -36,6 +38,8 @@ pub fn build(b: *std.Build) void {
.optimize = mode,
.use_llvm = use_llvm,
.use_lld = use_lld,
.sanitize_thread = sanitize_thread,
.single_threaded = single_threaded,
});
exe.root_module.addImport("yaml", yaml.module("yaml"));
exe.root_module.addImport("dis_x86_64", dis_x86_64.module("dis_x86_64"));
Expand Down Expand Up @@ -91,6 +95,8 @@ pub fn build(b: *std.Build) void {
.optimize = mode,
.use_llvm = use_llvm,
.use_lld = use_lld,
.sanitize_thread = sanitize_thread,
.single_threaded = single_threaded,
});
const unit_tests_opts = b.addOptions();
unit_tests.root_module.addOptions("build_options", unit_tests_opts);
Expand Down
Loading
Loading