SIGILL Illegal Instruction on PPC64

[barf]

Unable to install rust on Debian Bookworm PPC64 (POWER5+)

Meta

rustc --version --verbose:

Illegal instruction

When running cargo from rust 1.68.2-powerpc64-unknown-linux-gnu (downloaded from the rust website), I get the following backtrace

Backtrace

#0  std::sys::unix::stack_overflow::imp::make_handler () at library/std/src/sys/unix/stack_overflow.rs:160
#1  0x0000000100eded3c in std::sys::unix::stack_overflow::imp::init () at library/std/src/sys/unix/stack_overflow.rs:123
#2  std::sys::unix::init () at library/std/src/sys/unix/mod.rs:69
#3  std::rt::init () at library/std/src/rt.rs:97
#4  std::rt::lang_start_internal::{closure#1} () at library/std/src/rt.rs:147
#5  std::panicking::try::do_call<std::rt::lang_start_internal::{closure_env#1}, ()> () at library/std/src/panicking.rs:483
#6  std::panicking::try<(), std::rt::lang_start_internal::{closure_env#1}> () at library/std/src/panicking.rs:447
#7  std::panic::catch_unwind<std::rt::lang_start_internal::{closure_env#1}, ()> () at library/std/src/panic.rs:140
#8  std::rt::lang_start_internal () at library/std/src/rt.rs:147
#9  0x000000010033f328 in main ()

To hazard a guess this binary has been built for some specific/newer POWER target and not the generic POWER4/POWER5 supported by Debian

[cuviper]

One relevant piece is the crosstool-ng config for C and C++, which should be POWER4 but tuned for POWER6.

rust/src/ci/docker/README.md

Lines 364 to 365 in e4dae0d

	- Target options > Emit assembly for CPU = power4 -- (+)
	- Target options > Tune for CPU = power6 -- (+)

rust/src/ci/docker/host-x86_64/dist-powerpc64-linux/powerpc64-linux-gnu.config

Lines 138 to 139 in e4dae0d

	CT_ARCH_CPU="power4"
	CT_ARCH_TUNE="power6"

Then for Rust code, the target is set for "ppc64", which looks pretty much the same as "pwr4".

rust/compiler/rustc_target/src/spec/powerpc64_unknown_linux_gnu.rs

Line 6 in e4dae0d

base.cpu = "ppc64".into();

https://github.com/rust-lang/llvm-project/blob/585a6eb3ebf7c40fd7c1b23e3ece557b3cc2aa36/llvm/lib/Target/PowerPC/PPC.td#L657-L662
https://github.com/rust-lang/llvm-project/blob/585a6eb3ebf7c40fd7c1b23e3ece557b3cc2aa36/llvm/lib/Target/PowerPC/PPC.td#L616-L619

[cuviper]

Can you run under gdb, and use the command disassemble after you hit the SIGILL to see what it is?

Do any older versions work? It would help if we can narrow anything that changed.

[barf]

Thanks for the help! Unsure if any version ever worked on this combo, came across this as a dependency build failure for a Python module but can reproduce on both IBM POWER5 and Apple G5 machines FWIW. The PowerPC (32b) binary works ok these same machines.

Offending instruction looks to be a vxor v2,v2,v2 which I cannot find in the AIX assembler reference manual:
https://www.ibm.com/docs/en/ssw_aix_72/assembler/assembler_pdf.pdf

Backtrace:

(gdb) disassemble
Dump of assembler code for function _ZN3std3sys4unix14stack_overflow3imp12make_handler17h970c063dadd51481E:
   0x0000000100efffe4 <+0>:	mflr    r0
   0x0000000100efffe8 <+4>:	std     r0,16(r1)
   0x0000000100efffec <+8>:	stdu    r1,-240(r1)
   0x0000000100effff0 <+12>:	std     r29,216(r1)
   0x0000000100effff4 <+16>:	nop
   0x0000000100effff8 <+20>:	std     r30,224(r1)
   0x0000000100effffc <+24>:	lbz     r3,20608(r2)
   0x0000000100f00000 <+28>:	cmplwi  r3,0
   0x0000000100f00004 <+32>:	beq     0x100f000d8 <_ZN3std3sys4unix14stack_overflow3imp12make_handler17h970c063dadd51481E+244>
=> 0x0000000100f00008 <+36>:	vxor    v2,v2,v2
   0x0000000100f0000c <+40>:	addi    r4,r1,112
   0x0000000100f00010 <+44>:	li      r30,0
   0x0000000100f00014 <+48>:	stvx    v2,0,r4
   0x0000000100f00018 <+52>:	li      r3,0
   0x0000000100f0001c <+56>:	std     r30,128(r1)
   0x0000000100f00020 <+60>:	bl      0x100217f00 <000000d6.plt_call.sigaltstack@@GLIBC_2.3>
   0x0000000100f00024 <+64>:	ld      r2,40(r1)
   0x0000000100f00028 <+68>:	lbz     r3,123(r1)
   0x0000000100f0002c <+72>:	andi.   r3,r3,2
   0x0000000100f00030 <+76>:	beq     0x100f000dc <_ZN3std3sys4unix14stack_overflow3imp12make_handler17h970c063dadd51481E+248>
   0x0000000100f00034 <+80>:	li      r3,30
   0x0000000100f00038 <+84>:	bl      0x100216720 <000000d6.plt_call.sysconf@@GLIBC_2.3>
   0x0000000100f0003c <+88>:	ld      r2,40(r1)
   0x0000000100f00040 <+92>:	addi    r4,r3,16384
   0x0000000100f00044 <+96>:	lis     r3,2
   0x0000000100f00048 <+100>:	ori     r6,r3,34
   0x0000000100f0004c <+104>:	li      r3,0
   0x0000000100f00050 <+108>:	li      r5,3
   0x0000000100f00054 <+112>:	li      r7,-1
   0x0000000100f00058 <+116>:	li      r8,0
   0x0000000100f0005c <+120>:	li      r29,0
   0x0000000100f00060 <+124>:	bl      0x100216680 <000000d6.plt_call.mmap64@@GLIBC_2.3>
   0x0000000100f00064 <+128>:	ld      r2,40(r1)
   0x0000000100f00068 <+132>:	cmpdi   r3,-1
   0x0000000100f0006c <+136>:	beq-    0x100f000f8 <_ZN3std3sys4unix14stack_overflow3imp12make_handler17h970c063dadd51481E+276>
   0x0000000100f00070 <+140>:	mr      r30,r3
   0x0000000100f00074 <+144>:	li      r3,30
   0x0000000100f00078 <+148>:	bl      0x100216720 <000000d6.plt_call.sysconf@@GLIBC_2.3>
   0x0000000100f0007c <+152>:	ld      r2,40(r1)
   0x0000000100f00080 <+156>:	mr      r4,r3
   0x0000000100f00084 <+160>:	mr      r3,r30
   0x0000000100f00088 <+164>:	li      r5,0
   0x0000000100f0008c <+168>:	li      r29,0
   0x0000000100f00090 <+172>:	bl      0x1002167e0 <000000d6.plt_call.mprotect@@GLIBC_2.3>
   0x0000000100f00094 <+176>:	ld      r2,40(r1)
   0x0000000100f00098 <+180>:	cmplwi  r3,0
   0x0000000100f0009c <+184>:	bne-    0x100f00160 <_ZN3std3sys4unix14stack_overflow3imp12make_handler17h970c063dadd51481E+380>
   0x0000000100f000a0 <+188>:	li      r3,30
   0x0000000100f000a4 <+192>:	bl      0x100216720 <000000d6.plt_call.sysconf@@GLIBC_2.3>
   0x0000000100f000a8 <+196>:	ld      r2,40(r1)
   0x0000000100f000ac <+200>:	add     r3,r30,r3
   0x0000000100f000b0 <+204>:	li      r4,0
   0x0000000100f000b4 <+208>:	stw     r29,120(r1)
   0x0000000100f000b8 <+212>:	std     r3,112(r1)
   0x0000000100f000bc <+216>:	li      r3,16384
   0x0000000100f000c0 <+220>:	std     r3,128(r1)
   0x0000000100f000c4 <+224>:	addi    r3,r1,112
   0x0000000100f000c8 <+228>:	bl      0x100217f00 <000000d6.plt_call.sigaltstack@@GLIBC_2.3>
   0x0000000100f000cc <+232>:	ld      r2,40(r1)
   0x0000000100f000d0 <+236>:	ld      r30,112(r1)
   0x0000000100f000d4 <+240>:	b       0x100f000dc <_ZN3std3sys4unix14stack_overflow3imp12make_handler17h970c063dadd51481E+248>
   0x0000000100f000d8 <+244>:	li      r30,0
   0x0000000100f000dc <+248>:	mr      r3,r30
   0x0000000100f000e0 <+252>:	ld      r30,224(r1)
   0x0000000100f000e4 <+256>:	ld      r29,216(r1)
   0x0000000100f000e8 <+260>:	addi    r1,r1,240
   0x0000000100f000ec <+264>:	ld      r0,16(r1)
   0x0000000100f000f0 <+268>:	mtlr    r0
   0x0000000100f000f4 <+272>:	blr
   0x0000000100f000f8 <+276>:	bl      0x1002162a0 <000000d6.plt_call.__errno_location@@GLIBC_2.3>
   0x0000000100f000fc <+280>:	ld      r2,40(r1)
   0x0000000100f00100 <+284>:	addi    r4,r1,208
   0x0000000100f00104 <+288>:	lwz     r3,0(r3)
   0x0000000100f00108 <+292>:	addis   r5,r2,-1
   0x0000000100f0010c <+296>:	std     r4,192(r1)
   0x0000000100f00110 <+300>:	addi    r4,r1,192
   0x0000000100f00114 <+304>:	sldi    r3,r3,32
   0x0000000100f00118 <+308>:	ori     r3,r3,2
   0x0000000100f0011c <+312>:	std     r4,176(r1)
   0x0000000100f00120 <+316>:	addis   r4,r2,-11
   0x0000000100f00124 <+320>:	addi    r4,r4,-5744
   0x0000000100f00128 <+324>:	std     r4,160(r1)
   0x0000000100f0012c <+328>:	addi    r4,r5,5624
   0x0000000100f00130 <+332>:	std     r4,200(r1)
   0x0000000100f00134 <+336>:	li      r4,1
   0x0000000100f00138 <+340>:	std     r29,144(r1)
   0x0000000100f0013c <+344>:	std     r4,168(r1)
   0x0000000100f00140 <+348>:	std     r3,208(r1)
   0x0000000100f00144 <+352>:	std     r4,184(r1)
   0x0000000100f00148 <+356>:	addis   r3,r2,-11
   0x0000000100f0014c <+360>:	addi    r4,r3,-5728
   0x0000000100f00150 <+364>:	addi    r3,r1,144
   0x0000000100f00154 <+368>:	bl      0x10029de40 <_ZN4core9panicking9panic_fmt17h434a0cef292b8897E>
   0x0000000100f00158 <+372>:	nop
   0x0000000100f0015c <+376>:	b       0x100f001c4 <_ZN3std3sys4unix14stack_overflow3imp12make_handler17h970c063dadd51481E+480>
   0x0000000100f00160 <+380>:	bl      0x1002162a0 <000000d6.plt_call.__errno_location@@GLIBC_2.3>
   0x0000000100f00164 <+384>:	ld      r2,40(r1)
   0x0000000100f00168 <+388>:	addi    r4,r1,208
   0x0000000100f0016c <+392>:	lwz     r3,0(r3)
   0x0000000100f00170 <+396>:	addis   r5,r2,-1
   0x0000000100f00174 <+400>:	std     r4,192(r1)
   0x0000000100f00178 <+404>:	addi    r4,r1,192
   0x0000000100f0017c <+408>:	sldi    r3,r3,32
   0x0000000100f00180 <+412>:	ori     r3,r3,2
   0x0000000100f00184 <+416>:	std     r4,176(r1)
   0x0000000100f00188 <+420>:	addis   r4,r2,-11
   0x0000000100f0018c <+424>:	addi    r4,r4,-5704
   0x0000000100f00190 <+428>:	std     r4,160(r1)
   0x0000000100f00194 <+432>:	addi    r4,r5,5624
   0x0000000100f00198 <+436>:	std     r4,200(r1)
   0x0000000100f0019c <+440>:	li      r4,1
   0x0000000100f001a0 <+444>:	std     r29,144(r1)
   0x0000000100f001a4 <+448>:	std     r4,168(r1)
   0x0000000100f001a8 <+452>:	std     r3,208(r1)
   0x0000000100f001ac <+456>:	std     r4,184(r1)
   0x0000000100f001b0 <+460>:	addis   r3,r2,-11
   0x0000000100f001b4 <+464>:	addi    r4,r3,-5688
   0x0000000100f001b8 <+468>:	addi    r3,r1,144
   0x0000000100f001bc <+472>:	bl      0x10029de40 <_ZN4core9panicking9panic_fmt17h434a0cef292b8897E>
   0x0000000100f001c0 <+476>:	nop
   0x0000000100f001c4 <+480>:	trap
   0x0000000100f001c8 <+484>:	mr      r30,r3
   0x0000000100f001cc <+488>:	addi    r3,r1,208
   0x0000000100f001d0 <+492>:	bl      0x100ec6200 <_ZN4core3ptr42drop_in_place$LT$std..io..error..Error$GT$17h660e0984064a8f30E>
   0x0000000100f001d4 <+496>:	b       0x100f001e4 <_ZN3std3sys4unix14stack_overflow3imp12make_handler17h970c063dadd51481E+512>
   0x0000000100f001d8 <+500>:	mr      r30,r3
   0x0000000100f001dc <+504>:	addi    r3,r1,208
   0x0000000100f001e0 <+508>:	bl      0x100ec6200 <_ZN4core3ptr42drop_in_place$LT$std..io..error..Error$GT$17h660e0984064a8f30E>
   0x0000000100f001e4 <+512>:	mr      r3,r30
   0x0000000100f001e8 <+516>:	bl      0x100217940 <000000d6.plt_call._Unwind_Resume@@GCC_3.0>
   0x0000000100f001ec <+520>:	ld      r2,40(r1)
   0x0000000100f001f0 <+524>:	trap
   0x0000000100f001f4 <+528>:	bl      0x10029e024 <_ZN4core9panicking19panic_cannot_unwind17h3cc20fa46c62dff9E>
   0x0000000100f001f8 <+532>:	nop
   0x0000000100f001fc <+536>:	trap
   0x0000000100f00200 <+540>:	.long 0x0
   0x0000000100f00204 <+544>:	.long 0x0
   0x0000000100f00208 <+548>:	.long 0x0
End of assembler dump.

[cuviper]

The PowerPC (32b) binary works ok these same machines.

That uses an even older CPU baseline since #41080.

Offending instruction looks to be a vxor v2,v2,v2

That's Vector Logical XOR, an AltiVec instruction. I can see that instruction in binaries all the way back to Rust 1.8, the first that shipped for this target at all, so at least this seems not to be a regression.

can reproduce on both IBM POWER5 and Apple G5 machines FWIW.

I think those should have AltiVec, but I'm not sure.

Does Debian's own rustc package work for you?

[barf]

Yes I had to use an even older target build to simply build this Python dependency... unfortunately the Debian package is similarly broken, which led me to trying the official rust distribution.

Can only guess this was never tested on actual hardware shrugs

[barf]

Can I please know what the source is for this idea vxor exists in AltiVec on POWER5? It does not appear in any IBM reference material that I can find, maybe that tuning for power6 has broken power4 compatibility?

[cuviper]

I was looking at https://en.wikipedia.org/wiki/Power_ISA#Power_ISA_v.2.03 which includes AltiVec, and lists the 970 (aka G5) and POWER5 as being compliant. There's an archive link to the spec too, with vxor on page 178.
https://web.archive.org/web/20110727183430/http://www.power.org/resources/downloads/PowerISA_203_Final_Public.pdf

maybe that tuning for power6 has broken power4 compatibility?

I think that's only relevant to C and C++ components, while you're seeing this in pure Rust code. Sorry for the red herring.

[barf]

That's the instruction set reference, only certain manufacturers like NXP implemented it AFAIK(?) but it might also be on the PS3 and iBooks etc (due graphics)

The IBM reference omits mention of this instruction, suggesting it does not feature on the POWER5+ (gs). I would not assume vxor exists unless specifically mentioned in the manufacturer's assembly reference manual (and it is not).

[cuviper]

Ok, well I don't know. This is coming from LLVM codegen, which has included FeatureAltivec in pwr3 and up since they were added in 2013: llvm/llvm-project@52742c2

[barf]

Just thinking aloud, you probably know more than me, and thanks again for looking! The Power ISA docs is the complete list of instructions, was under the impression embedded and server platforms omit some.

Interesting to note GCC does not appear to offer accelerated calls that output the AltiVec XOR opcode for the same build target? Can only assume there's some reason for that (shrugs again)
https://gcc.gnu.org/onlinedocs/gcc/PowerPC-AltiVec-Built-in-Functions-on-ISA-2_002e05.html#PowerPC-AltiVec-Built-in-Functions-on-ISA-2_002e05

[cuviper]

GCC lists vec_xor on this page:
https://gcc.gnu.org/onlinedocs/gcc/PowerPC-AltiVec_002fVSX-Built-in-Functions.html

[barf]

Ah thanks, I stand corrected! Will keep digging around here, Debian also broke their kernel package on this machine :-|

[barf]

Hi all, need to bump this issue, the presence of this xvor instruction in the cargo and rustup-init binaries for powerpc64 is AFAIK limiting compatibility to only Apple hardware, more recent IBM POWER hardware (eg the POWER5 9406-520) is throwing this SIGILL. POWER6-7 or newer I would expect to work, but that means skipping a whole POWER generation(?)

Can the installer be built without such AltiVec optimisations, to improve compatibility (like how x86-64 targets a basic flag set)?

[cuviper]

For some context, that model lists Marketing Withdrawn on 2008-07-08 and Service Discontinued on 2019-01-31.
https://www.ibm.com/docs/en/announcements/eserver-i5-model-520?region=US

But we also did allow very old 32-bit powerpc in #41080, so there's some precedent.

Nominating for discussion -- does the compiler team want to support older powerpc64 (big-endian) hardware?
(I don't know which LLVM cpu target would be most compatible...)

@rustbot label +I-compiler-nominated

[workingjubilee]

As far as I know, the aforementioned OS (Debian Linux) does not support ppc64 in recent releases, but rather supports ppc64le. It does support powerpc (32-bit), and POWER5 machines can generally run 32-bit powerpc binaries because the architectures treat bit-width as a mode of a single CPU ISA, unlike, say, aarch64, which treats 64-bit instructions as an entirely different ISA.

I believe Adélie Linux, one of the rare Linux distros which explicitly supports Power ISA in big-endian mode, does in fact assume Altivec support in their builds.

[cuviper]

It seems to be in Debian ports: https://wiki.debian.org/PPC64

[workingjubilee]

I suppose it comes down to your definition of "support", then.

Regarding the PowerMac G5, it should have the Altivec instructions. Because of the conflicting reports in this thread regarding whether it is compatible or not, I am going to obtain a G5 to verify its compatibility myself and remove all doubt.

[barf]

Thanks, this sans-AltiVec IBM (9406-520) hardware is old/uncommon for sure, I guess I'll look into cross building and removing -mabi=altivec or similar too. Grateful for any clarification!

[workingjubilee]

Building a fork which changes the base.cpu here (as was mentioned a while back as a possibility) should be fairly effective, possibly with whatever LLVM-configuration-level changes are necessary to have it also take. That way you don't have to continually reset the compiler flag for all future builds for the host.

rust/compiler/rustc_target/src/spec/powerpc64_unknown_linux_gnu.rs

Line 6 in 734a0d0

base.cpu = "ppc64".into();

[apiraino]

Discussed yesterday in T-compiler meeting (on Zulip). Agreed that we need to discuss this topic once more to be able to make a call. Leaving this issue nominated for now.

[barf]

Thanks for looking into this! I re-tested on an Apple Xserve G5 yesterday (w/AltiVec) and found no issues, sorry for the confusion, I think it can be narrowed down to this IBM hardware (9406-520 and similar POWER5 machines)

[apiraino]

Discusses in T-compiler meeting (on Zulip). There is a bit of unclarity on how to proceed, we're also not sure about the actual user base.

One concrete proposal was perhaps to split this support target and add a new Tier 3 support target for this architecture (POWER5) that is currently not compiling under our PPC64 target. Seems a good task for a contribution.

For context, our tier support targets list.

@rustbot label -I-compiler-nominated