ld: final link failed: File truncated

[aliceinwire]

kernel: 4.19.1
kernel_config: https://gist.github.com/aliceinwire/72d090b6ff4ec8cc019882e553fbeacb
patch: https://gist.github.com/aliceinwire/9cc9d852f5a99450f035052474d85044

localhost ~ # gcc --version
gcc (Gentoo 7.3.0-r3 p1.4) 7.3.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

kpatch-build log: https://gist.github.com/aliceinwire/2a3910a958b8809b04c7baae516544c7

[joe-lawrence]

The "sibling call from callable instruction with modified stack frame" are only warning messages. See the note in the kernel documentation on those.

This looks to be the real failure:

ld: warning: dot moved backwards before `.text.__startup_secondary_64'
ld: warning: dot moved backwards before `.text.__startup_secondary_64'
ld: final link failed: File truncated
make: *** [Makefile:1032: vmlinux] Error 1

__startup_secondary_64() is defined in arch/x86/kernel/head64.c, can you attach head64.o or is it empty?

[aliceinwire]

This is the output (warning big file):
elivepatch-b0c6620d-20ae-4bd8-8c18-c16b81a6e90f.tar.gz

I looked a bit into the head64.o file:

2e743432eaf6 /tmp/elivepatch-b0c6620d-20ae-4bd8-8c18-c16b81a6e90f/usr/src/linux/
arch/x86/kernel # objdump -x head64.o  |grep __startup_secondary_64
  9 .text.__startup_secondary_64 00000003  0000000000000000  0000000000000000  00000b38  2**0
0000000000000000 l    d  .text.__startup_secondary_64   0000000000000000 .text.__startup_secondary_64
0000000000000000 g     F .text.__startup_secondary_64   0000000000000003 __startup_secondary_64
000000000000e99b R_X86_64_64       .text.__startup_secondary_64
0000000000000070 R_X86_64_64       .text.__startup_secondary_64
0000000000000160 R_X86_64_64       .text.__startup_secondary_64
0000000000000168 R_X86_64_64       .text.__startup_secondary_64+0x0000000000000003
0000000000000d9f R_X86_64_64       .text.__startup_secondary_64
00000000000000f0 R_X86_64_64       .text.__startup_secondary_64
0000000000000040 R_X86_64_PC32     .text.__startup_secondary_64
0000000000000044 R_X86_64_PC32     .text.__startup_secondary_64+0x0000000000000003

2e743432eaf6 /tmp/elivepatch-b0c6620d-20ae-4bd8-8c18-c16b81a6e90f/usr/src/linux/
arch/x86/kernel # readelf -h head64.o
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              REL (Relocatable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          151984 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           0 (bytes)
  Number of program headers:         0
  Size of section headers:           64 (bytes)
  Number of section headers:         41
  Section header string table index: 37

CFLAGS:
https://paste.pound-python.org/show/jmkJctutf4RRj8wsmdQK/

[sm00th]

I can reproduce it with gcc version 8.2.1 20181105 (Red Hat 8.2.1-5) (GCC) on 4.19/4.20-rc this only happens when optimizing for size and building with function-sections enabled, i.e. KCFLAGS="-ffunction-sections -Os".

[joe-lawrence]

Does this occur when simply building a newish kernel and CONFIG_CC_OPTIMIZE_FOR_SIZE set? Or does it only occur when kpatch-build is building its before/after kernels?

[sm00th]

Does this occur when simply building a newish kernel and CONFIG_CC_OPTIMIZE_FOR_SIZE set? Or does it only occur when kpatch-build is building its before/after kernels?

Because it is only reproducible with -ffunction-sections it does not happen during "normal" kernel builds.

[joe-lawrence]

Because it is only reproducible with -ffunction-sections it does not happen during "normal" kernel builds.

Ah, got it. FWIW, I get a slightly different build error when setting CONFIG_CC_OPTIMIZE_FOR_SIZE and exporting KCFLAGS="-ffunction-sections" with both gcc v7.2.1 and v4.8.5:

% make -j$(nproc)
...
String table index out of bounds
make[2]: *** [arch/x86/boot/compressed/Makefile:127: arch/x86/boot/compressed/vmlinux.relocs] Error 1
make[2]: *** Deleting file 'arch/x86/boot/compressed/vmlinux.relocs'
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [arch/x86/boot/Makefile:112: arch/x86/boot/compressed/vmlinux] Error 2
make: *** [arch/x86/Makefile:283: bzImage] Error 2

So I guess the question is whether -Os and -ffunction-sections are expected to be compatible.

[sm00th]

String table index out of bounds

Uh, I started getting this as well yesterday (no idea what have changed in my environment) and it has nothing to do with -Os, this one is reproducible for me with just KCFLAGS="-ffunction-sections" and CONFIG_X86_NEED_RELOCS=y

[sm00th]

I've been looking at it a bit more and I have a picture of what is going on, but still have no idea who is at fault here.

The difference between perf/size optimized objectfiles that breaks linking is the section alignment (sh_addralign).

$ eu-readelf -S head64-perf-optimized.o | grep __startup_secondary
[16] .text.__startup_secondary_64 PROGBITS     0000000000000000 00000c60 00000003  0 AX     0   0 16

$ eu-readelf -S head64-size-optimized.o | grep __startup_secondary
[16] .text.__startup_secondary_64 PROGBITS     0000000000000000 00000c40 00000003  0 AX     0   0  1

ld does multiple passes, in both objectfile on first pass dot after .iplt section is 0xffffffff814055c8, on second one it is 0xffffffff814055c1 because .iplt has ignored set and thus is not included in the output. In case of size-optimized binary these are the dot values for is .text.__startup_secondary_64 and so it is considered a backwards movement, but with perf-optimized binary this address is getting aligned and dot is set to 0xffffffff814055d0 in both cases.

It looks like in 4.18 this section has proper alignment even with CONFIG_CC_OPTIMIZE_FOR_SIZE so I'll try bisecting it.

[sm00th]

It looks like in 4.18 this section has proper alignment even with CONFIG_CC_OPTIMIZE_FOR_SIZE

No it is reproducible even on 4.15

[joe-lawrence]

String table index out of bounds

Uh, I started getting this as well yesterday (no idea what have changed in my environment) and it has nothing to do with -Os, this one is reproducible for me with just KCFLAGS="-ffunction-sections" and CONFIG_X86_NEED_RELOCS=y

This is really weird -- on older upstream kernels (v4.10, v4.11, etc.), copying RHEL-7 config + make olddefconfig + make localmodconfig and export KCFLAGS="-ffunction-sections" results in the linker segfaulting. Add -fdata-sections and I get the "string table index out of bounds" error. How is this any different than kpatch-build's invocations?

[aliceinwire]

kpatch-build looks relay on the GCC -ffunction-sections flag.

kpatch/kpatch-build/kpatch-build

Line 704 in 7f550f0

export KCFLAGS="-I$DATADIR/patch -ffunction-sections -fdata-sections $ARCH_KCFLAGS"

using instead

export KCFLAGS="-I$DATADIR/patch -fno-function-sections -fdata-sections $ARCH_KCFLAGS"

it compile but i got
/usr/libexec/kpatch/create-diff-object: ERROR: meminfo.o: find_local_syms: 174: find_local_syms for meminfo.c: couldn't find in vmlinux symbol table

kernel: 4.19.2
GCC: gcc (Gentoo Hardened 7.3.0-r3 p1.4) 7.3.0

[sm00th]

How is this any different than kpatch-build's invocations?

The answer is "targets", kpatch-build does make vmlinux modules which is less then default target and thus the 'relocs' testing is skipped.

kpatch-build looks relay on the GCC -ffunction-sections flag

Yes, kpatch-build won't work without this flag as this is how we determine which functions are changed.

[sm00th]

The answer is "targets", kpatch-build does make vmlinux modules which is less then default target and thus the 'relocs' testing is skipped.

posttests are triggered by bzImage target, which is added to default list as a default arch target in arch/x86/Makefile.

[joe-lawrence]

How is this any different than kpatch-build's invocations?

The answer is "targets", kpatch-build does make vmlinux modules which is less then default target and thus the 'relocs' testing is skipped.

kpatch-build looks relay on the GCC -ffunction-sections flag

Yes, kpatch-build won't work without this flag as this is how we determine which functions are changed.

So two different issues:

1. When running kpatch-build, we're hitting the "dot moved backwards" linker error, as described in this comment.

2. when setting CONFIG_X86_NEED_RELOCS=y, manually specifying KCFLAGS="-ffunction-sections", and running make we're blowing up in arch/x86/tools/relocs.c with "string table index out of bounds" error.

In both cases, these are general kernel build problems, not kpatch bugs. @sm00th , since you've been doing the detective work here, do you want to continue pursuing these upstream? Should kpatch check for (1) in kpatch-build and report that the configuration combination is not supported (yet)?

[aliceinwire]

Should kpatch check for (1) in kpatch-build and report that the configuration combination is not supported (yet)?
It would be useful, to warning users for now.

[sm00th]

@sm00th , since you've been doing the detective work here, do you want to continue pursuing these upstream?

The first issue seems to be a linker problem, I've managed to hack up a reproducer for it and bisect it down to a specific commit in ld, filed a bug here: https://sourceware.org/bugzilla/show_bug.cgi?id=23930

Should kpatch check for (1) in kpatch-build and report that the configuration combination is not supported (yet)?

Since it is not kpatch's issue I see no reason to work around it, I think this issue is enough for those who might hit it.

[sm00th]

Proposed a patch for the second issue: https://lore.kernel.org/lkml/20181129135133.31369-1-asavkov@redhat.com/T/#u

[jpoimboe]

I'm getting a linker seg fault when running kpatch-build against a stock Fedora 28 kernel (4.19.8-200.fc28.x86_64)

scripts/link-vmlinux.sh: line 85: 11952 Segmentation fault      (core dumped) ${LD} ${KBUILD_LDFLAGS} ${LDFLAGS_vmlinux} -o ${2} -T ${lds} ${objects}

[sm00th]

Both issues should be resolved now, binutils fix is in 2.32, kernel patch is in kernels 5.2+.

Is anyone willing to retest this or can we just close this issue?