RuntimeError: ELF .dynamic section is missing. when building for armv7l-linux and riscv64-linux on NixOS since 254-rc1

[misuzu]

systemd version the issue has been seen with

254-rc1

Used distribution

NixOS

Linux kernel version used

6.1.53

CPU architectures issue was seen on

x86_64

Component

systemd, systemd-boot

Expected behaviour you didn't see

systemd with systemd-boot builds without issues on armv7l-linux and riscv64-linux, both native and cross

Unexpected behaviour you saw

When cross-compiling systemd for armv7l-linux:

[659/1779] Compiling C object src/boot/efi/addonarm.elf.stub.p/addon.c.o
[660/1779] Linking target src/boot/efi/addonarm.elf.stub
[661/1779] Generating src/boot/efi/addonarm.efi.stub with a custom command
FAILED: src/boot/efi/addonarm.efi.stub
/build/source/tools/elf2efi.py --version-major=254 --version-minor=0 --efi-major=1 --efi-minor=1 --subsystem=10 --minimum-sections=7 src/boot/efi/addonarm.elf.stub src/boot/efi/addonarm.efi.stub
Traceback (most recent call last):
  File "/build/source/tools/elf2efi.py", line 593, in <module>
    main()
  File "/build/source/tools/elf2efi.py", line 589, in main
    elf2efi(parser.parse_args())
  File "/build/source/tools/elf2efi.py", line 493, in elf2efi
    pe_reloc_s = convert_elf_relocations(elf, opt, sections)
  File "/build/source/tools/elf2efi.py", line 384, in convert_elf_relocations
    raise RuntimeError("ELF .dynamic section is missing.")
RuntimeError: ELF .dynamic section is missing.
[662/1779] Compiling C object src/boot/efi/linuxarm.elf.stub.p/stub.c.o
[663/1779] Compiling C object src/import/libimport-common.a.p/import-common.c.o
[664/1779] Compiling C object src/boot/efi/systemd-bootarm.elf.p/boot.c.o
ninja: build stopped: subcommand failed.

Full log: systemd-armv7l-unknown-linux-gnueabihf-254-rc1.log

When cross-compiling systemd for riscv64-linux:

[658/1776] Compiling C object src/boot/efi/linuxriscv64.elf.stub.p/stub.c.o
[659/1776] Generating src/boot/efi/addonriscv64.efi.stub with a custom command
FAILED: src/boot/efi/addonriscv64.efi.stub
/build/source/tools/elf2efi.py --version-major=254 --version-minor=0 --efi-major=1 --efi-minor=1 --subsystem=10 --minimum-sections=7 src/boot/efi/addonriscv64.elf.stub src/boot/efi/addonriscv64.efi.stub
Traceback (most recent call last):
  File "/build/source/tools/elf2efi.py", line 593, in <module>
    main()
  File "/build/source/tools/elf2efi.py", line 589, in main
    elf2efi(parser.parse_args())
  File "/build/source/tools/elf2efi.py", line 493, in elf2efi
    pe_reloc_s = convert_elf_relocations(elf, opt, sections)
  File "/build/source/tools/elf2efi.py", line 384, in convert_elf_relocations
    raise RuntimeError("ELF .dynamic section is missing.")
RuntimeError: ELF .dynamic section is missing.
[660/1776] Linking target src/boot/efi/linuxriscv64.elf.stub
[661/1776] Compiling C object src/import/libimport-common.a.p/import-common.c.o
[662/1776] Compiling C object src/boot/efi/systemd-bootriscv64.elf.p/boot.c.o
ninja: build stopped: subcommand failed.

Full log: systemd-riscv64-unknown-linux-gnu-254-rc1.log

Steps to reproduce the problem

1. Use NixOS or install nix on x86_64-linux machine:
   curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install
2. Cross-compile systemd:
   nix build -L -j4 --cores 4 github:NixOS/nixpkgs/8a86b98f0ba1c405358f1b71ff8b5e1d317f5db2#pkgsCross.armv7l-hf-multiplatform.systemd (armv7l-linux, v254.3, fails)
   nix build -L -j4 --cores 4 github:NixOS/nixpkgs/fe6e29938174d1548bdcf02e0100824b272442ed#pkgsCross.armv7l-hf-multiplatform.systemd (armv7l-linux, v254-rc1, fails)
   nix build -L -j4 --cores 4 github:NixOS/nixpkgs/18b500811c2676bb2da63594403e581c7c899206#pkgsCross.armv7l-hf-multiplatform.systemd (armv7l-linux, v253.5, succeeds)
   nix build -L -j4 --cores 4 github:NixOS/nixpkgs/8a86b98f0ba1c405358f1b71ff8b5e1d317f5db2#pkgsCross.riscv64.systemd (riscv64-linux, v254.3, fails)
   nix build -L -j4 --cores 4 github:NixOS/nixpkgs/fe6e29938174d1548bdcf02e0100824b272442ed#pkgsCross.riscv64.systemd (riscv64-linux, v254-rc1, fails)
   nix build -L -j4 --cores 4 github:NixOS/nixpkgs/18b500811c2676bb2da63594403e581c7c899206#pkgsCross.riscv64.systemd (riscv64-linux, v253.5, succeeds)

Related:
#26641
NixOS/nixpkgs#243242
NixOS/nixpkgs@fe6e299

Additional program output to the terminal or log subsystem illustrating the issue

No response

[medhefgo]

Cross compilation works fine for me in a debian chroot (arm and riscv64). Something is causing the linker to not emit a .dynamic section when it should. The contents of the cross file as well as adding -Wl,-v to link flags to get the linker's cmdline would be very helpful.

[misuzu]

Logs with -Wl,-v in CFLAGS and cross files:

systemd-armv7l-unknown-linux-gnueabihf-254.3.log

[properties]
needs_exe_wrapper = true

[host_machine]
system = 'linux'
cpu_family = 'arm'
cpu = 'armv7l'
endian = 'little'

[binaries]
llvm-config = 'llvm-config-native'

systemd-riscv64-unknown-linux-gnu-254.3.log

[properties]
needs_exe_wrapper = true

[host_machine]
system = 'linux'
cpu_family = 'riscv64'
cpu = 'riscv64'
endian = 'little'

[binaries]
llvm-config = 'llvm-config-native'

[lopsided98]

I think I have figured out the cause of this bug, see my explanation here: NixOS/nixpkgs#243242 (comment)

[misuzu]

I think I have figured out the cause of this bug, see my explanation here: NixOS/nixpkgs#243242 (comment)

I can confirm, it builds for armv7l and riscv64 with -pie in efi_c_ld_args. I've used this patch to test:

diff --git a/src/boot/efi/meson.build b/src/boot/efi/meson.build
index 2773eaf286..83a615f646 100644
--- a/src/boot/efi/meson.build
+++ b/src/boot/efi/meson.build
@@ -164,6 +164,7 @@ efi_c_ld_args = [
         # These flags should be passed by -static-pie, but seem to be missing sometimes.
         '-Wl,--no-dynamic-linker',
         '-z', 'text',
+        '-pie',

         # EFI has 4KiB pages.
         '-z', 'common-page-size=4096',

[medhefgo]

I can confirm, it builds for armv7l and riscv64 with -pie in efi_c_ld_args. I've used this patch to test:

We explicitly set pie : true in meson, so why is that not passed to the linker?

[medhefgo]

Can you check whether meson actually passes -pie to the compiler by running ninja with -v?

[lopsided98]

Yeah, that's weird. It's passing -pie to the gcc linking call, but when gcc actually invokes the linker it is missing. My guess is that this is some argument ordering issue and @misuzu's patch works because it puts -pie later in the arguments.

These are the arguments passed with the patch:

  -o
  src/boot/efi/addonarm.elf.stub
  src/boot/efi/addonarm.elf.stub.p/addon.c.o
  -Wl,--as-needed
  -Wl,--no-undefined
  -pie <--------------------- this one is added by Meson
  -fstack-protector
  -fuse-ld=bfd
  -Wl,--start-group
  -lgcc
  -Wl,--end-group
  -nostdlib
  -static-pie
  -Wl,--entry=efi_main
  -Wl,--fatal-warnings
  -Wl,--no-dynamic-linker
  -z
  text
  -pie <---------------------- this one was added by the patch
  -z
  common-page-size=4096
  -z
  max-page-size=4096
  -z
  noexecstack
  -z
  norelro
  -T/build/source/tools/elf2efi.lds
  -fcf-protection=none
  -fno-asynchronous-unwind-tables
  -fno-exceptions
  -fno-unwind-tables
  -fno-sanitize=all
  -Wl,--no-wchar-size-warning

[medhefgo]

That's… interesting. Calling the cross gcc with -dumpspecs might shed a light on why/how it translates that -static-pie.

[lopsided98]

armv6l-unknown-linux-gnueabihf-gcc -dumpspecs

*asm:
%{mbig-endian:-EB} %{mlittle-endian:-EL} %(asm_cpu_spec) %{mapcs-*:-mapcs-%*} %(subtarget_asm_float_spec) %{mthumb-interwork:-mthumb-interwork} %{mfloat-abi=*} %{!mfpu=auto: %{mfpu=*}} %(subtarget_extra_asm_spec)

*asm_debug:
%{%:debug-level-gt(0):%{gstabs*:--gstabs;:%{g*:%{%:dwarf-version-gt(4):--gdwarf-5;%:dwarf-version-gt(3):--gdwarf-4;%:dwarf-version-gt(2):--gdwarf-3;:--gdwarf2}}}} %{fdebug-prefix-map=*:--debug-prefix-map %*}

*asm_debug_option:


*asm_final:
%{gsplit-dwarf: 
       objcopy --extract-dwo 	 %{c:%{o*:%*}%{!o*:%w%b%O}}%{!c:%U%O} 	 %b.dwo 
       objcopy --strip-dwo 	 %{c:%{o*:%*}%{!o*:%w%b%O}}%{!c:%U%O}     }

*asm_options:
%{-target-help:%:print-asm-header()} %{v} %{w:-W} %{I*} %(asm_debug_option) %{gz|gz=zlib:--compress-debug-sections=zlib} %{gz=none:--compress-debug-sections=none} %{gz=zlib-gnu:--compress-debug-sections=zlib-gnu} %a %Y %{c:%W{o*}%{!o*:-o %w%b%O}}%{!c:-o %d%w%u%O}

*invoke_as:
%{!fwpa*:   %{fcompare-debug=*|fdump-final-insns=*:%:compare-debug-dump-opt()}   %{!S:-o %|.s |
 as %(asm_options) %m.s %A }  }

*cpp:
%(subtarget_cpp_spec)

*cpp_options:
%(cpp_unique_options) %1 %{m*} %{std*&ansi&trigraphs} %{W*&pedantic*} %{w} %{f*} %{g*:%{%:debug-level-gt(0):%{g*} %{!fno-working-directory:-fworking-directory}}} %{O*} %{undef} %{save-temps*:-fpch-preprocess}

*cpp_debug_options:
%<dumpdir %<dumpbase %<dumpbase-ext %{d*} %:dumps()

*cpp_unique_options:
%{!Q:-quiet} %{nostdinc*} %{C} %{CC} %{v} %@{I*&F*} %{P} %I %{MD:-MD %{!o:%b.d}%{o*:%.d%*}} %{MMD:-MMD %{!o:%b.d}%{o*:%.d%*}} %{M} %{MM} %{MF*} %{MG} %{MP} %{MQ*} %{MT*} %{Mmodules} %{Mno-modules} %{!E:%{!M:%{!MM:%{!MT:%{!MQ:%{MD|MMD:%{o*:-MQ %*}}}}}}} %{remap} %{%:debug-level-gt(2):-dD} %{!iplugindir*:%{fplugin*:%:find-plugindir()}} %{H} %C %{D*&U*&A*} %{i*} %Z %i %{E|M|MM:%W{o*}}

*trad_capable_cpp:
cc1 -E %{traditional|traditional-cpp:-traditional-cpp}

*cc1:
%{!mandroid|tno-android-cc:%{profile:-p} %{%:sanitize(address):-funwind-tables} ;:%{profile:-p} %{%:sanitize(address):-funwind-tables} %{!mglibc:%{!muclibc:%{!mbionic: -mbionic}}} %{!fno-pic:%{!fno-PIC:%{!fpic:%{!fPIC: -fPIC}}}}}

*cc1_options:
%{pg:%{fomit-frame-pointer:%e-pg and -fomit-frame-pointer are incompatible}} %{!iplugindir*:%{fplugin*:%:find-plugindir()}} %1 %{!Q:-quiet} %(cpp_debug_options) %{m*} %{aux-info*} %{g*} %{O*} %{W*&pedantic*} %{w} %{std*&ansi&trigraphs} %{v:-version} %{pg:-p} %{p} %{f*} %{undef} %{Qn:-fno-ident} %{Qy:} %{-help:--help} %{-target-help:--target-help} %{-version:--version} %{-help=*:--help=%*} %{!fsyntax-only:%{S:%W{o*}%{!o*:-o %w%b.s}}} %{fsyntax-only:-o %j} %{-param*} %{coverage:-fprofile-arcs -ftest-coverage} %{fprofile-arcs|fprofile-generate*|coverage:   %{!fprofile-update=single:     %{pthread:-fprofile-update=prefer-atomic}}}

*cc1plus:
%{!mandroid|tno-android-cc:;:%{!fexceptions:%{!fno-exceptions: -fno-exceptions}} %{!frtti:%{!fno-rtti: -fno-rtti}}}

*link_gcc_c_sequence:
%{static|static-pie:--start-group} %G %{!nolibc:%L}    %{static|static-pie:--end-group}%{!static:%{!static-pie:%G}}

*link_ssp:
%{fstack-protector|fstack-protector-all|fstack-protector-strong|fstack-protector-explicit:}

*endfile:
%{Ofast|ffast-math|funsafe-math-optimizations:crtfastmath.o%s} %{!mandroid|tno-android-ld:%{!static:%{fvtable-verify=none:%s;      fvtable-verify=preinit:vtv_end_preinit.o%s;      fvtable-verify=std:vtv_end.o%s}}    %{static:crtend.o%s;      shared|static-pie|pie:crtendS.o%s;      :crtend.o%s} crtn.o%s ;:%{shared: crtend_so%O%s;: crtend_android%O%s}}

*link:
%{!static|static-pie:--eh-frame-hdr}  %{mcpu=arm8|mcpu=arm810|mcpu=strongarm*|march=armv4|mcpu=fa526|mcpu=fa626:--fix-v4bx}%{!r:%{!mbe32:%:be8_linkopt(%{mlittle-endian:little}			       %{mbig-endian:big}			       %{mbe8:be8}			       %{march=*:arch %*})}}%{!mandroid|tno-android-ld:%{h*}    %{static:-Bstatic}    %{shared:-shared}    %{symbolic:-Bsymbolic}    %{!static:      %{rdynamic:-export-dynamic}      %{!shared:-dynamic-linker %{muclibc:/lib/ld-uClibc.so.0;:%{mbionic:/system/bin/linker;:%{mmusl:/lib/ld-musl-arm%{mbig-endian:eb}%{mfloat-abi=hard:hf}%{mfdpic:-fdpic}.so.1;:%{mfloat-abi=hard:/lib/ld-linux-armhf.so.3}     %{mfloat-abi=soft*:/lib/ld-linux.so.3}     %{!mfloat-abi=*:/lib/ld-linux.so.3}}}}}}    -X    %{mbig-endian:-EB} %{mlittle-endian:-EL} -m armelf_linux_eabi;:%{h*}    %{static:-Bstatic}    %{shared:-shared}    %{symbolic:-Bsymbolic}    %{!static:      %{rdynamic:-export-dynamic}      %{!shared:-dynamic-linker %{muclibc:/lib/ld-uClibc.so.0;:%{mbionic:/system/bin/linker;:%{mmusl:/lib/ld-musl-arm%{mbig-endian:eb}%{mfloat-abi=hard:hf}%{mfdpic:-fdpic}.so.1;:%{mfloat-abi=hard:/lib/ld-linux-armhf.so.3}     %{mfloat-abi=soft*:/lib/ld-linux.so.3}     %{!mfloat-abi=*:/lib/ld-linux.so.3}}}}}}    -X    %{mbig-endian:-EB} %{mlittle-endian:-EL} -m armelf_linux_eabi %{shared: -Bsymbolic}}

*lib:
%{!mandroid|tno-android-ld:%{pthread:-lpthread} %{shared:-lc}    %{!shared:%{profile:-lc_p}%{!profile:-lc}};:%{shared:-lc}    %{!shared:%{profile:-lc_p}%{!profile:-lc}} %{!static: -ldl}}

*link_gomp:


*libgcc:
%{static|static-libgcc|static-pie:-lgcc -lgcc_eh}%{!static:%{!static-libgcc:%{!static-pie:%{!shared-libgcc:-lgcc --push-state --as-needed -lgcc_s --pop-state}%{shared-libgcc:-lgcc_s%{!shared: -lgcc}}}}}

*startfile:
%{!mandroid|tno-android-ld:%{shared:;      pg|p|profile:%{static-pie:grcrt1.o%s;:gcrt1.o%s};      static:crt1.o%s;      static-pie:rcrt1.o%s;      pie:Scrt1.o%s;      :crt1.o%s} crti.o%s    %{static:crtbeginT.o%s;      shared|static-pie|pie:crtbeginS.o%s;      :crtbegin.o%s}    %{fvtable-verify=none:%s;      fvtable-verify=preinit:vtv_start_preinit.o%s;      fvtable-verify=std:vtv_start.o%s} ;:%{shared: crtbegin_so%O%s;:  %{static: crtbegin_static%O%s;: crtbegin_dynamic%O%s}}}

*cross_compile:
1

*version:
12.3.0

*multilib:
. ;

*multilib_defaults:


*multilib_extra:


*multilib_matches:


*multilib_exclusions:


*multilib_options:


*multilib_reuse:


*linker:
collect2

*linker_plugin_file:


*lto_wrapper:


*lto_gcc:


*post_link:


*link_libgcc:
%D

*md_exec_prefix:


*md_startfile_prefix:


*md_startfile_prefix_1:


*startfile_prefix_spec:


*sysroot_spec:
--sysroot=%R

*sysroot_suffix_spec:


*sysroot_hdrs_suffix_spec:


*self_spec:


*subtarget_cpp_spec:
%{posix:-D_POSIX_SOURCE} %{pthread:-D_REENTRANT}

*asm_cpu_spec:
 %{mfpu=auto:%<mfpu=auto %:asm_auto_mfpu(%{march=*: arch %*})} %{mcpu=generic-*:-march=%:rewrite_march(%{mcpu=generic-*:%*});   march=*:-march=%:rewrite_march(%{march=*:%*});   mcpu=*:-mcpu=%:rewrite_mcpu(%{mcpu=*:%*}) }

*subtarget_extra_asm_spec:
%{mabi=apcs-gnu|mabi=atpcs:-meabi=gnu;:-meabi=5} %{mcpu=arm8|mcpu=arm810|mcpu=strongarm*|march=armv4|mcpu=fa526|mcpu=fa626:--fix-v4bx}

*subtarget_asm_float_spec:
%{mapcs-float:-mfloat}

*link_command:
%{!fsyntax-only:%{!c:%{!M:%{!MM:%{!E:%{!S:    %(linker) %{!fno-use-linker-plugin:%{!fno-lto:     -plugin %(linker_plugin_file)     -plugin-opt=%(lto_wrapper)     -plugin-opt=-fresolution=%u.res         %{flinker-output=*:-plugin-opt=-linker-output-known}     %{!nostdlib:%{!nodefaultlibs:%:pass-through-libs(%(link_gcc_c_sequence))}}     }}%{flto|flto=*:%<fcompare-debug*}     %{flto} %{fno-lto} %{flto=*} %l %{static|shared|r:;pie:-pie} %{fuse-ld=*:-fuse-ld=%*}  %{gz|gz=zlib:--compress-debug-sections=zlib} %{gz=none:--compress-debug-sections=none} %{gz=zlib-gnu:--compress-debug-sections=zlib-gnu} %X %{o*} %{e*} %{N} %{n} %{r}    %{s} %{t} %{u*} %{z} %{Z} %{!nostdlib:%{!r:%{!nostartfiles:%S}}}     %{static|no-pie|static-pie:} %@{L*} %(mfwrap) %(link_libgcc) %{fvtable-verify=none:} %{fvtable-verify=std:   %e-fvtable-verify=std is not supported in this configuration} %{fvtable-verify=preinit:   %e-fvtable-verify=preinit is not supported in this configuration} %{!nostdlib:%{!r:%{!nodefaultlibs:%{%:sanitize(address):%{!shared:libasan_preinit%O%s} %{static-libasan:%{!shared:-Bstatic --whole-archive -lasan --no-whole-archive -Bdynamic}}%{!static-libasan:-lasan}}     %{%:sanitize(hwaddress):%{static-libhwasan:%{!shared:-Bstatic --whole-archive -lhwasan --no-whole-archive -Bdynamic}}%{!static-libhwasan:-lhwasan}}     %{%:sanitize(thread):%{!shared:libtsan_preinit%O%s} %{static-libtsan:%{!shared:-Bstatic --whole-archive -ltsan --no-whole-archive -Bdynamic}}%{!static-libtsan:-ltsan}}     %{%:sanitize(leak):%{!shared:liblsan_preinit%O%s} %{static-liblsan:%{!shared:-Bstatic --whole-archive -llsan --no-whole-archive -Bdynamic}}%{!static-liblsan:-llsan}}}}} %o      %{fopenacc|fopenmp|%:gt(%{ftree-parallelize-loops=*:%*} 1):	%:include(libgomp.spec)%(link_gomp)}    %{fgnu-tm:%:include(libitm.spec)%(link_itm)}    %(mflib)  %{fsplit-stack: --wrap=pthread_create}    %{fprofile-arcs|fprofile-generate*|coverage:-lgcov} %{!nostdlib:%{!r:%{!nodefaultlibs:%{%:sanitize(address): %{static-libasan|static:%:include(libsanitizer.spec)%(link_libasan)}    %{static:%ecannot specify -static with -fsanitize=address}}    %{%:sanitize(hwaddress): %{static-libhwasan|static:%:include(libsanitizer.spec)%(link_libhwasan)}	%{static:%ecannot specify -static with -fsanitize=hwaddress}}    %{%:sanitize(thread): %{static-libtsan|static:%:include(libsanitizer.spec)%(link_libtsan)}    %{static:%ecannot specify -static with -fsanitize=thread}}    %{%:sanitize(undefined):%{static-libubsan:-Bstatic} -lubsan %{static-libubsan:-Bdynamic} %{static-libubsan|static:%:include(libsanitizer.spec)%(link_libubsan)}}    %{%:sanitize(leak): %{static-liblsan|static:%:include(libsanitizer.spec)%(link_liblsan)}}}}}     %{!nostdlib:%{!r:%{!nodefaultlibs:%(link_ssp) %(link_gcc_c_sequence)}}}    %{!nostdlib:%{!r:%{!nostartfiles:%E}}} %{T*}  
%(post_link) }}}}}}

Through experimentation, I have determined that -pie must come after -static-pie for it to work.

[medhefgo]

The attached PR should fix this, please try.

It looks pretty much as @lopsided98 says: -static-pie overrides -pie and the distro build does not default to -pie, which means we get a regular binary. The nix vs debian gcc specs are largely the same (and neither contains the -static-pie translation), so this pie default is probably handled in code somewhere else.

[misuzu]

@lopsided98, @medhefgo thank you!