Influxdb 3 Core/Enterprise 'illegal instruction' on x86_64 CPU with CPUs older than Intel Haswell

[jdstrand]

https://github.com/influxdata/influxdb/blob/main/.cargo/config.toml#L21 lists:

[target.x86_64-unknown-linux-gnu]
rustflags = [
    ...
    # Enable all features supported by CPUs more recent than haswell (2013)
    "-C", "target-cpu=haswell",

I have an old test machine (Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz) with up to date Ubuntu 18.04 LTS, but despite having proper glibc compatibility, it fails to run:

$ ./influxdb3 -V
Illegal instruction (core dumped)

$ gdb ./influxdb3
...
Program received signal SIGILL, Illegal instruction.
(gdb) run -V

(gdb) x/i $pc
=> 0x555558233124 <_ZN12clap_builder5error14Error$LT$F$GT$5print17hcabd755f43c3aba1E+8484>:	shrx   %r12d,%eax,%r12d

Googling tells me that the shrx instruction was introduced with the Intel BMI2 (Bit Manipulation Instruction Set 2) extension, which was first supported in Intel's Haswell microarchitecture, released in 2013.

It appears that this was introduced because we wanted the avx2 instruction, first introduced in haswell: https://github.com/influxdata/influxdb_iox/pull/2119 states "I think that the current setting for ROARING_ARCH of x86-64 may not be specific enough to ensure that things are built with the avx2 instructions. Ivy Bridge Haswell has those so I think that this should work better." https://github.com/influxdata/influxdb_iox/pull/2140 has additional information.

It's apparently quite intentional that we don't support CPUs older than haswell. We probably need a documentation note for this. I expect this issue to be closed but wanted to file it for posterity since others will likely run into this.

cc @pauldix, @peterbarnett03, @alamb (since you wrote https://github.com/influxdata/influxdb_iox/pull/2140), @sanderson and @jstirnaman

References:

• https://github.com/influxdata/influxdb_iox/pull/2119
• https://github.com/influxdata/influxdb_iox/pull/2140

[pauldix]

I don't think there's a need so support CPUs older than that. However, if this is due to some dependency on croaring, I don't think we use that anymore. So I'm not sure why we'd still have this limitation. It would take some investigation or someone that knows more about this part than me.

[jdstrand]

From my reading, it seems that we were wanting to support avx2 specifically which is only available in Haswell and later (and croaring just happened to be part of those changes, but not having to do with the desire for avx2 and needing Haswell).

[alamb]

From my reading, it seems that we were wanting to support avx2 specifically which is only available in Haswell and later (and croaring just happened to be part of those changes, but not having to do with the desire for avx2 and needing Haswell).

Yes I think the idea is to let the rust compiler / auto vectorizer take advantage of avx2 SIMD instructions which are significantly faster for many of the lower level arrow kernels

If we want to support older hardware without slowing down for more modern hardware, the standard options I know of are

1. Make a special build targeting older instructions sets (easy for us, less easy for customers)
2. Compile code for multiple instructions sets and then pick the appropriate code version at runtime based on the available hardware

I suggest we go with 1 at first unless there is significant demand for something eaiser

[pauldix]

Ok, I don't think we need to even bother with it at this point. We'll wait until someone comes around asking for the special build before we produce it. Shockingly, anything before 2013 is now long ago, lost to the sands of time. Onwards and upwards into our SIMD accelerated future utopia!

[jdstrand]

Shockingly, anything before 2013 is now long ago, lost to the sands of time

/me gazes at his old test machine... is that a mirage? ;)

(just riffing on the comment; not suggesting we do anything different for my test machine!)

[alamb]

Very meta

[thexeos]

I see that croaring was replaced with arrow in 2022.

I went ahead and compiled main against sandybridge and it seems to be running fine so far.

[schasj]

The box I run Influx on has a Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz which I think is Ivy bridge. This is a very relevant CPU IMO so please support it.

[hpux735]

Right. I was just spinning up an instance on my proxmox host and ran into this. Switching lower-level implementations based on instruction support is something that video games have been doing for ages.

[schasj]

compiled main against sandybridge

I tried to do the same but found that it's not clear how to do this.

[thexeos]

I tried to do the same but found that it's not clear how to do this.

It's rather easy:

1. curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh to install Rust (choose all default options)
2. source "$HOME/.cargo/env" to refresh your Path
3. rustc --version to confirm Rust is installed
4. git clone https://github.com/influxdata/influxdb.git to clone the repo, cd influxdb to enter the directory, you can then choose specific version/tag with git checkout <commit hash> or do nothing and stay on main branch
5. Edit .cargo/config.toml to change target-cpu=haswell to target-cpu=sandybridge
6. Last step (kind of) is to run cargo build -p influxdb3 --profile release

Now, there is a good chance that last command will fail on you multiple times due to missing dependencies. Depending on your OS and already installed packages, you will need to install some extra stuff. Just Google/ChatGPT the errors that you get and it will tell you which packages you are missing. Re-run the cargo build -p influxdb3 --profile release after each error and it will resume the build from the point where it errored out. When it gets to last step, expect it to take 45-60 minutes to complete if you are building it on sandybridge. Once it is done building, you will have the executable in ./target/release/influxdb3 that you can chmod +x ./target/release/influxdb3 to give it the right permissions and then run ./target/release/influxdb3 serve --object-store=file --data-dir=~/.influxdb3 to start the InfluxDB 3 process.

[schasj]

Easy for someone who does this day in and day out!

I tried your instructions and Claude, who had me try 15-20 different build combinations. WAY too much time on this so please just build the Docker image with sandybridge support.