overflow error when linking big libraries on MacOS

[davidgm94]

Zig Version

0.13.0-dev.7+40118c769

Steps to Reproduce and Observed Behavior

I am using a fork of zig-bootstrap and when I try to link LLVM in debug mode I get the following error:
The fork used: https://github.com/birth-software/zig-bootstrap/tree/make-script-more-configurable

/Users/david/dev/zig/build-debug/stage3/bin/zig c++ -fno-sanitize=all -s -target aarch64-macos-none -mcpu=apple_m1 -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -g -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.4.sdk -Wl,-headerpad_max_install_names -static -Wl,--color-diagnostics utils/TableGen/CMakeFiles/obj.LLVMTableGenCommon.dir/Attributes.cpp.o utils/TableGen/CMakeFiles/obj.LLVMTableGenCommon.dir/CodeGenIntrinsics.cpp.o utils/TableGen/CMakeFiles/obj.LLVMTableGenCommon.dir/DirectiveEmitter.cpp.o utils/TableGen/CMakeFiles/obj.LLVMTableGenCommon.dir/IntrinsicEmitter.cpp.o utils/TableGen/CMakeFiles/obj.LLVMTableGenCommon.dir/RISCVTargetDefEmitter.cpp.o utils/TableGen/CMakeFiles/obj.LLVMTableGenCommon.dir/SDNodeProperties.cpp.o utils/TableGen/CMakeFiles/obj.LLVMTableGenCommon.dir/VTEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/AsmMatcherEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/AsmWriterEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/AsmWriterInst.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CTagsEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CallingConvEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeEmitterGen.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenDAGPatterns.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenHwModes.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenInstAlias.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenInstruction.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenMapTable.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenRegisters.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenSchedule.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenTarget.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DAGISelEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DAGISelMatcherEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DAGISelMatcherGen.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DAGISelMatcherOpt.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DAGISelMatcher.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DecoderEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DFAEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DFAPacketizerEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DisassemblerEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DXILEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/ExegesisEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/FastISelEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/GICombinerEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/GlobalISelCombinerMatchTableEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/GlobalISelEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/GlobalISelMatchTable.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/GlobalISelMatchTableExecutorEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/InfoByHwMode.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/InstrInfoEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/InstrDocsEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/OptEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/OptParserEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/OptRSTEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/PredicateExpander.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/PseudoLoweringEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CompressInstEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/RegisterBankEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/RegisterInfoEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/SearchableTableEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/SubtargetEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/SubtargetFeatureInfo.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/TableGen.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/Types.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/VarLenCodeEmitterGen.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/X86DisassemblerTables.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/X86EVEX2VEXTablesEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/X86FoldTablesEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/X86MnemonicTables.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/X86ModRMFilters.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/X86RecognizableInstr.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/WebAssemblyDisassemblerEmitter.cpp.o -o bin/llvm-tblgen  -Wl,-rpath,@loader_path/../lib  lib/libLLVMCodeGenTypes.a  lib/libLLVMSupport.a  lib/libLLVMTableGen.a  lib/libLLVMTableGenGlobalISel.a  lib/libLLVMCodeGenTypes.a  lib/libLLVMTableGen.a  lib/libLLVMSupport.a  -lm  /Users/david/dev/zig-bootstrap/out/aarch64-macos-none-apple_m1-debug-static/lib/libz.a  /Users/david/dev/zig-bootstrap/out/aarch64-macos-none-apple_m1-debug-static/lib/libzstd.a  lib/libLLVMDemangle.a 
error: Overflow
/Users/david/dev/zig/src/link/aarch64.zig:31:56: 0x10d7aee5f in calcNumberOfPages (zig)
    const pages = math.cast(i21, tpage - spage) orelse return error.Overflow;
                                                       ^
/Users/david/dev/zig/src/link/MachO/Atom.zig:773:45: 0x10d82a667 in resolveRelocInner__anon_84207 (zig)
            const pages = @as(u21, @bitCast(try aarch64.calcNumberOfPages(source, target)));
                                            ^
/Users/david/dev/zig/src/link/MachO/Atom.zig:584:29: 0x10d8262d3 in resolveRelocs (zig)
                else => |e| return e,
                            ^
/Users/david/dev/zig/src/link/MachO.zig:2563:29: 0x10d82bd77 in writeAtoms (zig)
                else => |e| return e,
                            ^
/Users/david/dev/zig/src/link/MachO.zig:638:13: 0x10d432a4f in flushModule (zig)
            return e;
            ^
/Users/david/dev/zig/src/link/MachO.zig:361:5: 0x10d109e4b in flush (zig)
    try self.flushModule(arena, prog_node);
    ^
/Users/david/dev/zig/src/link.zig:566:17: 0x10ce97c3b in flush (zig)
                return @as(*tag.Type(), @fieldParentPtr("base", base)).flush(arena, prog_node);
                ^
/Users/david/dev/zig/src/Compilation.zig:2265:25: 0x10ce97253 in flush (zig)
            else => |e| return e,
                        ^
/Users/david/dev/zig/src/Compilation.zig:2253:13: 0x10ce9a54f in update (zig)
            try flush(comp, arena, main_progress_node);
            ^
/Users/david/dev/zig/src/main.zig:4483:9: 0x10cec6dbb in updateModule (zig)
        try comp.update(main_progress_node);
        ^
/Users/david/dev/zig/src/main.zig:3411:21: 0x10cf2aa87 in buildOutputType (zig)
        else => |e| return e,
                    ^
/Users/david/dev/zig/src/main.zig:291:9: 0x10cd95ba7 in mainArgs (zig)
        return buildOutputType(gpa, arena, args, .cpp);
        ^
/Users/david/dev/zig/src/main.zig:206:5: 0x10cd926eb in main (zig)
    return mainArgs(gpa, arena, args);
    ^

Expected Behavior

To link successfully.

[slimsag]

I've been able to reproduce this with a similarly large pre-built static library, on macOS M3, using a slightly older Zig version:

0.12.0-dev.3180+83e578a18

Steps to reproduce with the above Zig version:

git clone https://github.com/hexops/mach-gpu-dawn
cd mach-gpu-dawn/
git checkout sg/static-library-overflow
zig build

download_dir: /Volumes/data/hexops/tmp/mach-gpu-dawn/zig-cache/mach/gpu-dawn/release-cdd4a1a/aarch64-macos-none/release-fast/download
info(mach_gpu_dawn): downloading https://github.com/hexops/mach-gpu-dawn/releases/download/release-cdd4a1a/libdawn_aarch64-macos-none_release-fast.a.gz

info(mach_gpu_dawn): finished

info(mach_gpu_dawn): extracting /Volumes/data/hexops/tmp/mach-gpu-dawn/zig-cache/mach/gpu-dawn/release-cdd4a1a/aarch64-macos-none/release-fast/download/compressed.gz

info(mach_gpu_dawn): finished

info(mach_gpu_dawn): downloading https://github.com/hexops/mach-gpu-dawn/releases/download/release-cdd4a1a/headers.json.gz

info(mach_gpu_dawn): finished

install
└─ install mach-gpu-dawn-lib
   └─ zig build-lib mach-gpu-dawn-lib Debug native failure
error: error: Overflow

error: the following command exited with error code 1:
/Users/slimsag/zig/0.12.0-dev.3180+83e578a18/files/zig build-lib -ldawn -lobjc -ODebug -iframework /Users/slimsag/.cache/zig/p/122010c1a745ea06dee3012fbd3b311bd3d75ec39ded6bf566b36ebe3cd8da482347/Frameworks -isystem /Users/slimsag/.cache/zig/p/122010c1a745ea06dee3012fbd3b311bd3d75ec39ded6bf566b36ebe3cd8da482347/include -I /Volumes/data/hexops/tmp/mach-gpu-dawn/zig-cache/mach/gpu-dawn/release-cdd4a1a/include -I /Volumes/data/hexops/tmp/mach-gpu-dawn/src/dawn -iframework /Users/slimsag/.cache/zig/p/122010c1a745ea06dee3012fbd3b311bd3d75ec39ded6bf566b36ebe3cd8da482347/Frameworks -isystem /Users/slimsag/.cache/zig/p/122010c1a745ea06dee3012fbd3b311bd3d75ec39ded6bf566b36ebe3cd8da482347/include -iframework /Users/slimsag/.cache/zig/p/122010c1a745ea06dee3012fbd3b311bd3d75ec39ded6bf566b36ebe3cd8da482347/Frameworks -isystem /Users/slimsag/.cache/zig/p/122010c1a745ea06dee3012fbd3b311bd3d75ec39ded6bf566b36ebe3cd8da482347/include -iframework /Users/slimsag/.cache/zig/p/122010c1a745ea06dee3012fbd3b311bd3d75ec39ded6bf566b36ebe3cd8da482347/Frameworks -isystem /Users/slimsag/.cache/zig/p/122010c1a745ea06dee3012fbd3b311bd3d75ec39ded6bf566b36ebe3cd8da482347/include -L /Users/slimsag/.cache/zig/p/122010c1a745ea06dee3012fbd3b311bd3d75ec39ded6bf566b36ebe3cd8da482347/lib -L /Volumes/data/hexops/tmp/mach-gpu-dawn/zig-cache/mach/gpu-dawn/release-cdd4a1a/aarch64-macos-none/release-fast -L /Users/slimsag/.cache/zig/p/122010c1a745ea06dee3012fbd3b311bd3d75ec39ded6bf566b36ebe3cd8da482347/lib -L /Users/slimsag/.cache/zig/p/122010c1a745ea06dee3012fbd3b311bd3d75ec39ded6bf566b36ebe3cd8da482347/lib -L /Users/slimsag/.cache/zig/p/122010c1a745ea06dee3012fbd3b311bd3d75ec39ded6bf566b36ebe3cd8da482347/lib -Mroot=/Volumes/data/hexops/tmp/mach-gpu-dawn/zig-cache/o/30002037093dd2fc6999af4c2f599c8a/empty.c -framework Foundation -framework Metal -framework CoreGraphics -framework IOKit -framework IOSurface -framework QuartzCore -framework CoreFoundation -framework CoreImage -framework CoreVideo -lc++ --cache-dir /Volumes/data/hexops/tmp/mach-gpu-dawn/zig-cache --global-cache-dir /Users/slimsag/.cache/zig --name mach-gpu-dawn-lib -static --listen=-
Build Summary: 2/5 steps succeeded; 1 failed (disable with --summary none)
install transitive failure
└─ install mach-gpu-dawn-lib transitive failure
   └─ zig build-lib mach-gpu-dawn-lib Debug native failure
error: the following build command failed with exit code 1:
/Volumes/data/hexops/tmp/mach-gpu-dawn/zig-cache/o/858dc668c47581a652742ce73c5a60e8/build /Users/slimsag/zig/0.12.0-dev.3180+83e578a18/files/zig /Volumes/data/hexops/tmp/mach-gpu-dawn /Volumes/data/hexops/tmp/mach-gpu-dawn/zig-cache /Users/slimsag/.cache/zig --seed 0x93446939 -Z897b19a522970a87

Running the build-lib command at the end of the output above, I was able to confirm that is the command that produces error: Overflow.

Note that I am phasing out / deprecating this project in a month or two, so I will probably not be able to re-test this. I thought the info was still useful however.

[zhylmzr]

According to the information above, this is a minimal reproducible repository.
zig-link-overflow.tar.gz

Note: If compiled under Windows it will run out of memory.

zig build -Dtarget=aarch64-macos

[kamidev]

According to the information above, this is a minimal reproducible repository. zig-link-overflow.tar.gz

Using this, I can reproduce the overflow error with both zig 0.12.0 and 0.13.0-dev.35+e8f28cda9 on MacOS M3.