kexec_file is broken with LLVM 18 and newer on x86_64 #2016
Comments
|
ThinLTO and CFI are red herrings, this reproduces for me with a standard distribution configuration (Arch Linux) with LLVM 18 and newer. It is only reproducible on x86_64, aarch64 works fine for me. I bisected LLVM to see what change introduced this and I landed on llvm/llvm-project@239a41e. I have not tried to compare the diff of |
nathanchance
changed the title
I would suggest mainly looking at & OFC inspecting the comment above the line that triggers a warning has useful clues as to the unmet requirements: https://github.com/gregkh/linux/blob/486067966cd308763a6ec811296e2ed4581f4af6/kernel/kexec_file.c#L922 |
|
Thanks for the tip, I realized that once I started digging a little bit. There are two distinct but related issues here. I think I understand the first issue but I am not sure I understand the second one... The first issue is the warning. As you aluded to, this warning is caused by the fact that there are now two text sections with With LLVM 17: With LLVM 18: Resulting in the following section diff: diff --git a/tmp/.psub.bySdEduC68 b/tmp/.psub.dAUlPpwKwM
index 5399ef7a64b6..254217201609 100644
--- a/tmp/.psub.bySdEduC68
+++ b/tmp/.psub.dAUlPpwKwM
@@ -1,17 +1,20 @@
Name
NULL
.text
+.ltext
__patchable_function_entries
-.rela.text
+.rela.ltext
.rela__patchable_function_entries
.kexec-purgatory
.comment
.llvm_addrsig
.data
+.rela.text
.rodata
.rela.rodata
.bss
.modinfo
+.lrodata
.rodata.str1.1
.note.GNU-stack
.symtabThis behavior change is caused by LLVM commit llvm/llvm-project@d8a0439, which adds the use of diff --git a/arch/x86/purgatory/.gitignore b/arch/x86/purgatory/.gitignore
index d2be1500671d..71bd99d98906 100644
--- a/arch/x86/purgatory/.gitignore
+++ b/arch/x86/purgatory/.gitignore
@@ -1 +1,2 @@
purgatory.chk
+purgatory.lds
diff --git a/arch/x86/purgatory/Makefile b/arch/x86/purgatory/Makefile
index bc31863c5ee6..d4669644137f 100644
--- a/arch/x86/purgatory/Makefile
+++ b/arch/x86/purgatory/Makefile
@@ -1,7 +1,7 @@
# SPDX-License-Identifier: GPL-2.0
OBJECT_FILES_NON_STANDARD := y
-purgatory-y := purgatory.o stack.o setup-x86_$(BITS).o sha256.o entry64.o string.o
+purgatory-y := purgatory.o purgatory.lds stack.o setup-x86_$(BITS).o sha256.o entry64.o string.o
targets += $(purgatory-y)
PURGATORY_OBJS = $(addprefix $(obj)/,$(purgatory-y))
@@ -25,9 +25,9 @@ KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_LTO),$(KBUILD_CFLAGS))
# When linking purgatory.ro with -r unresolved symbols are not checked,
# also link a purgatory.chk binary without -r to check for unresolved symbols.
-PURGATORY_LDFLAGS := -e purgatory_start -z nodefaultlib
-LDFLAGS_purgatory.ro := -r $(PURGATORY_LDFLAGS)
-LDFLAGS_purgatory.chk := $(PURGATORY_LDFLAGS)
+PURGATORY_LDFLAGS := -z nodefaultlib
+LDFLAGS_purgatory.ro := -r $(PURGATORY_LDFLAGS) -T
+LDFLAGS_purgatory.chk := -e purgatory_start $(PURGATORY_LDFLAGS)
targets += purgatory.ro purgatory.chk
# Sanitizer, etc. runtimes are unavailable and cannot be linked here.
@@ -80,7 +80,7 @@ CFLAGS_string.o += $(PURGATORY_CFLAGS)
asflags-remove-y += $(foreach x, -g -gdwarf-4 -gdwarf-5, $(x) -Wa,$(x))
-$(obj)/purgatory.ro: $(PURGATORY_OBJS) FORCE
+$(obj)/purgatory.ro: $(obj)/purgatory.lds $(PURGATORY_OBJS) FORCE
$(call if_changed,ld)
$(obj)/purgatory.chk: $(obj)/purgatory.ro FORCE
diff --git a/arch/x86/purgatory/purgatory.lds.S b/arch/x86/purgatory/purgatory.lds.S
new file mode 100644
index 000000000000..0898a221722f
--- /dev/null
+++ b/arch/x86/purgatory/purgatory.lds.S
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#include <asm-generic/vmlinux.lds.h>
+#include <asm/cache.h>
+
+OUTPUT_FORMAT(CONFIG_OUTPUT_FORMAT)
+
+#undef i386
+
+#ifdef CONFIG_X86_64
+OUTPUT_ARCH(i386:x86-64)
+#else
+OUTPUT_ARCH(i386)
+#endif
+
+ENTRY(purgatory_start)
+
+SECTIONS
+{
+ . = 0;
+
+ .kexec-purgatory : {
+ *(.kexec-purgatory)
+ }
+
+ .text : {
+ _text = .;
+ *(.text .text.*)
+ _etext = .;
+ }
+
+ .rodata : {
+ _rodata = .;
+ *(.rodata .rodata.*)
+ _erodata = .;
+ }
+
+ .data : {
+ _data = .;
+ *(.data .data.*)
+ _edata = .;
+ }
+
+ . = ALIGN(L1_CACHE_BYTES);
+ .bss : {
+ _bss = .;
+ *(.bss .bss.*)
+ *(COMMON)
+ . = ALIGN(8); /* For convenience during zeroing */
+ _ebss = .;
+ }
+ _end = .;
+
+ ELF_DETAILS
+
+ DISCARDS
+ /DISCARD/ : {
+ *(.note.GNU-stack .note.gnu.property)
+ *(.llvm_addrsig)
+ }
+}
The second issue is the "Overflow in relocation" error, which is what actually causes the kexec load to fail due to the Using LLVM 17: LLVM 18: It appears that the relocation to At llvm/llvm-project@e692d08 (the parent of the previously fingered change, llvm/llvm-project@239a41e) I am not really sure what is going wrong here though. I attempted moving I suspect ChromeOS will hit this eventually, so we should definitely look into this. |
|
About |
|
The relocation overflow seems like a real issue. Do you have a reduced repro/godbolt link that shows the 32-bit relocation with the large code model? |
I'm not sure there really is a problem with this, it is just something that the kernel has to account for now because it expects the purgatory object to only have one The bigger issue is the relocation overflow error that we experience after llvm/llvm-project@239a41e, which I expand on above starting with "The second issue is", which may not make too much sense. I am little out of my wheelhouse here but I can try to come at it from another angle tomorrow if it does turn out to be nonsense. |
No, I have not been able to reduce anything yet. I'll see if I can come up with something tomorrow. |
I think you meant "now The definition of 22:struct kexec_sha_region purgatory_sha_regions[KEXEC_SEGMENT_MAX] __section(".kexec-purgatory");perhaps that @nathanchance your linker script doesn't seem to mention |
Heh, yes.
Right, it is actually that function that is failing when relocating
Yeah not entirely sure yet... I want to see if replacing the LLVM 18 built
Whoops, forgot this diff, which was a later commit in my series: diff --git a/arch/x86/purgatory/purgatory.lds.S b/arch/x86/purgatory/purgatory.lds.S
index 0898a221722f..4fb155942642 100644
--- a/arch/x86/purgatory/purgatory.lds.S
+++ b/arch/x86/purgatory/purgatory.lds.S
@@ -26,12 +26,14 @@ SECTIONS
.text : {
_text = .;
*(.text .text.*)
+ *(.ltext .ltext.*)
_etext = .;
}
.rodata : {
_rodata = .;
*(.rodata .rodata.*)
+ *(.lrodata .lrodata.*)
_erodata = .;
}
which resolves these orphan section warnings |
|
ah yes, an explicit section on the global and no-PIC large code model does this: https://godbolt.org/z/9x1vhxYYb there are a couple of things in play here: with the medium code model, we have a small/large split for globals. small globals can be considered "near", meaning in the bottom 2GB of the address space. we decided to treat any globals with explicit sections as small because if you have multiple globals in an explicit section and some of them are small, some of them are large, you end up with a mixed small/large data in a section which is bad because there's no proper location to place the section in the binary to avoid relocation overflows, since 32-bit relocations are used for small data, but we want to put large data in a place that doesn't contribute to relocation pressure. for consistency, we did this for the large code model too with the assumption that the user won't have small data sections that sum to over 2GB (causing relocation overflows at that point). so no-PIC large code model functions theoretically could use 32-bit relocations to address small data. however, IIRC the kernel doesn't load these binaries at the low end of the address space. we can make functions in the large code model always use 64-bit relocations regardless of the global, that shouldn't be an issue. I'll work on a patch to do that |
I'd be happy to test it to make sure this issue gets resolved properly once it is available, just ping me. |
This matches other types of relocations, e.g. to constant pool. Some users of the large code model may not place small data in the lower 2GB of the address space (e.g. ClangBuiltLinux/linux#2016), so just unconditionally use 64-bit relocations in the large code model.
can you test llvm/llvm-project#89101? |
Thanks, I can confirm that the relocation overflow error is resolved with this change and |
nathanchance
added
[BUG] linux
This matches other types of relocations, e.g. to constant pool. And makes things more consistent with PIC large code model. Some users of the large code model may not place small data in the lower 2GB of the address space (e.g. ClangBuiltLinux/linux#2016), so just unconditionally use 64-bit relocations in the large code model. So now functions in a section not marked large will use 64-bit relocations to reference everything when using the large code model. This also fixes some lldb tests broken by #88172 (https://lab.llvm.org/buildbot/#/builders/68/builds/72458).
…89101) This matches other types of relocations, e.g. to constant pool. And makes things more consistent with PIC large code model. Some users of the large code model may not place small data in the lower 2GB of the address space (e.g. ClangBuiltLinux/linux#2016), so just unconditionally use 64-bit relocations in the large code model. So now functions in a section not marked large will use 64-bit relocations to reference everything when using the large code model. This also fixes some lldb tests broken by llvm#88172 (https://lab.llvm.org/buildbot/#/builders/68/builds/72458). (cherry picked from commit 6cea7c4)
|
@0n-s would you like a |
Sure. |
Commit 8652d44 ("kexec: support purgatories with .text.hot sections") added a warning when the purgatory has more than one .text section, which is unsupported. A couple of changes have been made to the x86 purgatory's Makefile to prevent the compiler from splitting the .text section as a result: 97b6b9c ("x86/purgatory: remove PGO flags") 75b2f7e ("x86/purgatory: Remove LTO flags") Unfortunately, there may be compiler optimizations that add other text sections that cannot be disabled. For example, starting with LLVM 18, large text is emitted in '.ltext', which happens for the purgatory due to commit e16c298 ("x86/purgatory: Change compiler flags from -mcmodel=kernel to -mcmodel=large to fix kexec relocation errors"), but there are out of line assembly files that use '.text'. $ llvm-readelf -S arch/x86/purgatory/purgatory.ro | rg ' .[a-z]?text' [ 1] .text PROGBITS 0000000000000000 000040 0000d0 00 AX 0 0 16 [ 2] .ltext PROGBITS 0000000000000000 000110 0015a6 00 AXl 0 0 16 To avoid the runtime warning when the purgatory has been built with LLVM 18, add a linker script that explicitly describes the sections of the purgatory.ro and use it to merge '.ltext' and '.lrodata' back into '.text' and '.rodata' to match the behavior of GCC and LLVM prior to the optimization, as the distinction between small and large text is not important in this case. This results in no warnings with '--orphan-handling=warn' with either GNU or LLVM toolchains and the resulting kernels can properly kexec other kernels. This linker script is based on arch/s390/purgatory/purgatory.lds.S and Ricardo Ribalda's prior attempt to add one for arch/x86 [1]. As a consequence of this change, the aforementioned flag changes can be reverted because the '.text.*' sections generated by those options will be combined properly by the linker script, which avoids the only reason they were added in the first place. kexec continues to work with LTO enabled. [1]: https://lore.kernel.org/20230321-kexec_clang16-v5-2-5563bf7c4173@chromium.org/ Reported-by: ns <0n-s@users.noreply.github.com> Closes: ClangBuiltLinux#2016 Signed-off-by: Nathan Chancellor <nathan@kernel.org>
Thanks! I humbly request that this be backported to stable if & when it's accepted. |
|
The solution has shifted to moving away from https://lore.kernel.org/20240418201705.3673200-2-ardb+git@google.com/ |
|
Compiled 6.8.7 with the above patch applied & one flashbang later (I wish GPU drivers didn't set brightness to max by default), I have managed to successfully |
…89101) This matches other types of relocations, e.g. to constant pool. And makes things more consistent with PIC large code model. Some users of the large code model may not place small data in the lower 2GB of the address space (e.g. ClangBuiltLinux/linux#2016), so just unconditionally use 64-bit relocations in the large code model. So now functions in a section not marked large will use 64-bit relocations to reference everything when using the large code model. This also fixes some lldb tests broken by llvm#88172 (https://lab.llvm.org/buildbot/#/builders/68/builds/72458).
|
The patch has been accepted into -tip, although the plan is to merge it during the 6.10 cycle and I guess manually backport it to stable because the cc was stripped during application. https://git.kernel.org/tip/cba786af84a0f9716204e09f518ce3b7ada8555e |
nathanchance
added
[PATCH] Accepted
…89101) This matches other types of relocations, e.g. to constant pool. And makes things more consistent with PIC large code model. Some users of the large code model may not place small data in the lower 2GB of the address space (e.g. ClangBuiltLinux/linux#2016), so just unconditionally use 64-bit relocations in the large code model. So now functions in a section not marked large will use 64-bit relocations to reference everything when using the large code model. This also fixes some lldb tests broken by llvm#88172 (https://lab.llvm.org/buildbot/#/builders/68/builds/72458). (cherry picked from commit 6cea7c4)
On x86, the ordinary, position dependent small and kernel code models only support placement of the executable in 32-bit addressable memory, due to the use of 32-bit signed immediates to generate references to global variables. For the kernel, this implies that all global variables must reside in the top 2 GiB of the kernel virtual address space, where the implicit address bits 63:32 are equal to sign bit 31. This means the kernel code model is not suitable for other bare metal executables such as the kexec purgatory, which can be placed arbitrarily in the physical address space, where its address may no longer be representable as a sign extended 32-bit quantity. For this reason, commit e16c298 ("x86/purgatory: Change compiler flags from -mcmodel=kernel to -mcmodel=large to fix kexec relocation errors") switched to the large code model, which uses 64-bit immediates for all symbol references, including function calls, in order to avoid relying on any assumptions regarding proximity of symbols in the final executable. The large code model is rarely used, clunky and the least likely to operate in a similar fashion when comparing GCC and Clang, so it is best avoided. This is especially true now that Clang 18 has started to emit executable code in two separate sections (.text and .ltext), which triggers an issue in the kexec loading code at runtime. The SUSE bugzilla fixes tag points to gcc 13 having issues with the large model too and that perhaps the large model should simply not be used at all. Instead, use the position independent small code model, which makes no assumptions about placement but only about proximity, where all referenced symbols must be within -/+ 2 GiB, i.e., in range for a RIP-relative reference. Use hidden visibility to suppress the use of a GOT, which carries absolute addresses that are not covered by static ELF relocations, and is therefore incompatible with the kexec loader's relocation logic. [ bp: Massage commit message. ] Fixes: e16c298 ("x86/purgatory: Change compiler flags from -mcmodel=kernel to -mcmodel=large to fix kexec relocation errors") Fixes: https://bugzilla.suse.com/show_bug.cgi?id=1211853 Closes: ClangBuiltLinux/linux#2016 Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de> Reviewed-by: Nathan Chancellor <nathan@kernel.org> Reviewed-by: Fangrui Song <maskray@google.com> Acked-by: Nick Desaulniers <ndesaulniers@google.com> Tested-by: Nathan Chancellor <nathan@kernel.org> Link: https://lore.kernel.org/all/20240417-x86-fix-kexec-with-llvm-18-v1-0-5383121e8fb7@kernel.org/
Don't really think there's much to explain here. Very nice & simple reproducer & splat:
vmlinuz-6.8.6is a bzImage withEFI_STUBenabled, if that matters.Unsurprisingly trying to
kexec -ehere will result in a hang.The text was updated successfully, but these errors were encountered: