Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unknown pseudo-op #14965

Closed
Dmole opened this issue Jun 10, 2023 · 17 comments · Fixed by #15784
Closed

unknown pseudo-op #14965

Dmole opened this issue Jun 10, 2023 · 17 comments · Fixed by #15784
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)

Comments

@Dmole
Copy link

Dmole commented Jun 10, 2023

System information

Type Version/Name
Distribution Name Ubuntu
Distribution Version 18.04.6
Kernel Version 5.10.183
Architecture aarch64
OpenZFS Version master 70ea484

Describe the problem you're observing

build after copy-builtin from master on aarch64

Describe how to reproduce the problem

build after copy-builtin from master on aarch64

Include any warning/errors/backtraces from the system logs


fs/zfs/icp/asm-aarch64/blake3/b3_aarch64_sse2.S:54: Error: unknown pseudo-op: `.cfi_negate_ra_state'      

fs/zfs/icp/asm-aarch64/blake3/b3_aarch64_sse2.S:558: Error: unknown pseudo-op: `.cfi_negate_ra_state'     

fs/zfs/icp/asm-aarch64/blake3/b3_aarch64_sse2.S:611: Error: unknown pseudo-op: `.cfi_negate_ra_state'     

make[2]: *** [fs/zfs/icp/asm-aarch64/blake3/b3_aarch64_sse2.o] Error 1                                    
make[1]: *** [fs/zfs] Error 2
make: *** [fs] Error 2                               
make: *** Waiting for unfinished jobs....            

@Dmole Dmole added the Type: Defect Incorrect behavior (e.g. crash, hang) label Jun 10, 2023
@FormulaAgile
Copy link

we have the same problem here

@ziggythehamster
Copy link

I can replicate this on EL8 with ZFS 2.2.2, but not with ZFS 2.1.14 and not on EL9. (and obviously only on aarch64 since this is an aarch64 specific optimization)

EL8 ships:

  • GCC 8
  • glibc 2.28
  • binutils 2.30

EL9 ships:

  • GCC 11
  • glibc 2.34
  • binutils 2.35

It is most likely one of those variables causing the issue. A newer toolchain can be installed on EL8, but that feels brittle to me.

@ziggythehamster
Copy link

This was introduced by #14728 and thus all releases of ZFS 2.2 will have this issue.

@Dmole
Copy link
Author

Dmole commented Jan 5, 2024

I encountered the issue with

Type Version/Name
Distribution Name Ubuntu
Distribution Version 18.04.6
GCC Version 7.4

But not with

Type Version/Name
Distribution Name Debian
Distribution Version 12
Kernel Version 5.10.205
Architecture aarch64
OpenZFS Version master cae502c
GCC Version 12.2

So maybe documentation like

https://openzfs.github.io/openzfs-docs/Developer%20Resources/Building%20ZFS.html

should just be updated to list minimum build requirement versions.
Because anywhere this issue is going to happen is going to be EOL soon anyway.

@ziggythehamster
Copy link

Because anywhere this issue is going to happen is going to be EOL soon anyway.

RHEL8 becomes EOL in 2029, so this either needs to be made conditional (I think this is just an assertion?) or officially the newer toolchain has to be made the minimum requirement.

I'm currently starting with GCC 9 to see what the earliest gcc-toolset-* version is that works.

@Dmole
Copy link
Author

Dmole commented Jan 5, 2024

Right; I forgot it was only CentOS 8 that EOL-ed in 2021.

Maybe @mcmilk and @behlendorf can comment.

@ziggythehamster
Copy link

GCC 9 doesn't work, but GCC 10 does work. To avoid having exclusively one version of the package using GCC10 that could break in a weird way, I am unconditionally using GCC10 on EL8 in my build script and both x86_64 and aarch64 work fine on both 2.1 and 2.2 (whereas 2.2 on aarch64 GCC8 doesn't work)

The only issue I see with this is that I will need to patch the zfs-dkms spec to depend on the GCC 10 toolchain.

If you want to try this experiment at home:

dnf install scl-utils gcc-toolset-10
# extract and whatever
scl run gcc-toolset-10 -- ./configure # or whatever args you want
scl run gcc-toolset-10 -- make rpm-utils rpm-dkms # or whatever targets you want

@ziggythehamster
Copy link

I ended up having to use gcc-toolset-11 because gcc-toolset-10 uses the wrong (system) version of annobin if you have both the system and toolset version of annobin installed. gcc-toolset-11 doesn't have this problem. (https://bugzilla.redhat.com/show_bug.cgi?id=1960299)

Also, patching things so that it uses the right toolchain was a nightmare. KERNEL_CC will still run as from $PATH even if you were to set CCAS and AS, so I ended up having to also set PATH and LD_LIBRARY_PATH.

@mcmilk
Copy link
Contributor

mcmilk commented Jan 15, 2024

dnf install scl-utils gcc-toolset-12
scl run gcc-toolset-12 -- ./configure
scl run gcc-toolset-12 -- make rpm-utils rpm-dkms

Yes, it will generate the correct packages, but installation will fail, because users of those packages will likely not have gcc-toolset-12 installed.... and even when, we need to tell the packages, that they rely on gcc-12 :/

@behlendorf - I really think, that we should extend the linux spl layer with some kfpu_begin/kfpu_end wrappers - so we can just use them. The spl module has the correct license and other functions like __alloc_percpu or free_percpu are used there already. What do you think, can I create a PR for this?

For the .cfi_negate_ra_state - I will try to find a fix for that ...

@ziggythehamster
Copy link

Yeah, in order for the built RPMs to install properly (and further for DKMS packages to work), I have to modify the spec files to depend on the right toolset version and also append CC/CCAS/LD/LD_LIBRARY_PATH/NM/PATH (where the path variables are calculated by /opt/rh/*/enable) to configure and make.

This isn’t ideal. :)

I think the cfi_negate_ra_state issue could be resolved by regenerating the .S with an older toolset, and having the knowledge of what the proper machine code is supposed to be. I tried this and got a completely different result. I tried with gcc11 and clang13 and neither of those gave me remotely similar machine code to what’s committed either.

@mcmilk
Copy link
Contributor

mcmilk commented Jan 16, 2024

The source for generating the assembler files is located here:
https://github.com/mcmilk/BLAKE3-tests/blob/master/contrib/simde.sh

@ziggythehamster
Copy link

Yeah, that’s what I tried. I get the expected differences (the function name is different, there’s not a compiler annotation), but also the assembly is very different. I don’t know arm assembly (and for that matter, x86 SIMD is somewhat dark magic to me even though I vaguely remember x86-16 assembly), so I can’t ascertain whether the output is correct or not.

@mcmilk
Copy link
Contributor

mcmilk commented Jan 16, 2024

I have created two new files, they are located here: b3_aarch64_sse2.S and b3_aarch64_sse41.S

They work on my local almalinux-8 with gcc8 ...

mcmilk added a commit to mcmilk/zfs that referenced this issue Jan 16, 2024
Compiling openzfs on aarch64 with gcc-8 and gcc-9 is failing currently.
See issue openzfs#14965 for deeper context.

I have re-generated the assembly without the "-mbranch-protection=standard"
compiling option.

I have tested this on Arm64 FreeBSD 13.2 and AlmaLinux-8.

Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Closes: openzfs#14965
mcmilk added a commit to mcmilk/zfs that referenced this issue Jan 17, 2024
Compiling openzfs on aarch64 with gcc-8 and gcc-9 is failing currently.
See issue openzfs#14965 for deeper context.

On platforms without pointer authentication, .cfi_negate_ra_state can be
defined to a no-op:
https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gdb/aarch64-tdep.c#l1413

I have tested this on Arm64 FreeBSD 13.2 and AlmaLinux-8.

Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Closes: openzfs#14965
mcmilk added a commit to mcmilk/zfs that referenced this issue Jan 17, 2024
Compiling openzfs on aarch64 with gcc-8 and gcc-9 is failing currently.
See issue openzfs#14965 for deeper context.

On platforms without pointer authentication, .cfi_negate_ra_state can be
defined to a no-op:
https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gdb/aarch64-tdep.c#l1413

I have tested this on Arm64 FreeBSD 13.2 and AlmaLinux-8.

Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Closes: openzfs#14965
@q0rban
Copy link

q0rban commented Mar 7, 2024

This error is also occurring on arm64:

$ dpkg --print-architecture
arm64

$ sudo make -s -j$(nproc)
  GEN      gitrev
Making all in include
Making all in tests/zfs-tests/tests
Making all in module
/opt/zfs/zfs-zfs-2.2.3/module/icp/asm-aarch64/blake3/b3_aarch64_sse2.S: Assembler messages:
/opt/zfs/zfs-zfs-2.2.3/module/icp/asm-aarch64/blake3/b3_aarch64_sse2.S:54: Error: unknown pseudo-op: `.cfi_negate_ra_state'
/opt/zfs/zfs-zfs-2.2.3/module/icp/asm-aarch64/blake3/b3_aarch64_sse2.S:558: Error: unknown pseudo-op: `.cfi_negate_ra_state'
/opt/zfs/zfs-zfs-2.2.3/module/icp/asm-aarch64/blake3/b3_aarch64_sse2.S:611: Error: unknown pseudo-op: `.cfi_negate_ra_state'
make[6]: *** [/usr/src/linux-headers-4.19.0-26-common/scripts/Makefile.build:408: /opt/zfs/zfs-zfs-2.2.3/module/icp/asm-aarch64/blake3/b3_aarch64_sse2.o] Error 1
make[6]: *** Waiting for unfinished jobs....
/opt/zfs/zfs-zfs-2.2.3/module/icp/asm-aarch64/blake3/b3_aarch64_sse41.S: Assembler messages:
/opt/zfs/zfs-zfs-2.2.3/module/icp/asm-aarch64/blake3/b3_aarch64_sse41.S:54: Error: unknown pseudo-op: `.cfi_negate_ra_state'
/opt/zfs/zfs-zfs-2.2.3/module/icp/asm-aarch64/blake3/b3_aarch64_sse41.S:568: Error: unknown pseudo-op: `.cfi_negate_ra_state'
make[6]: *** [/usr/src/linux-headers-4.19.0-26-common/scripts/Makefile.build:408: /opt/zfs/zfs-zfs-2.2.3/module/icp/asm-aarch64/blake3/b3_aarch64_sse41.o] Error 1
make[5]: *** [/usr/src/linux-headers-4.19.0-26-common/Makefile:1577: _module_/opt/zfs/zfs-zfs-2.2.3/module] Error 2
make[4]: *** [Makefile:153: sub-make] Error 2
make[3]: *** [Makefile:8: all] Error 2
make[2]: *** [Makefile:56: modules-Linux] Error 2
make[1]: *** [Makefile:12196: all-recursive] Error 1
make: *** [Makefile:4599: all] Error 2

@q0rban
Copy link

q0rban commented Mar 7, 2024

Nevermind, TIL that aarch64 == arm64 😆. I leave my comment though, in case it helps someone else find this issue.

@mcmilk
Copy link
Contributor

mcmilk commented Mar 7, 2024

Nevermind, TIL that aarch64 == arm64 😆. I leave my comment though, in case it helps someone else find this issue.

What compiler and what version of it fails here?

@q0rban
Copy link

q0rban commented Mar 7, 2024

What compiler and what version of it fails here?

$ gcc --version
gcc (Debian 8.3.0-6) 8.3.0

behlendorf pushed a commit that referenced this issue Apr 15, 2024
Compiling openzfs on aarch64 with gcc-8 and gcc-9 is failing currently.
See issue #14965 for deeper context.

On platforms without pointer authentication, .cfi_negate_ra_state can be
defined to a no-op:
https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gdb/aarch64-tdep.c#l1413

I have tested this on Arm64 FreeBSD 13.2 and AlmaLinux-8.

Reviewed-by: Andrew Turner <andrew.turner4@arm.com>
Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Closes #14965
Closes #15784
tonyhutter pushed a commit that referenced this issue May 2, 2024
Compiling openzfs on aarch64 with gcc-8 and gcc-9 is failing currently.
See issue #14965 for deeper context.

On platforms without pointer authentication, .cfi_negate_ra_state can be
defined to a no-op:
https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gdb/aarch64-tdep.c#l1413

I have tested this on Arm64 FreeBSD 13.2 and AlmaLinux-8.

Reviewed-by: Andrew Turner <andrew.turner4@arm.com>
Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Closes #14965
Closes #15784
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants