supported_platforms() does not include older x86-64 CPUs

[xianwenchen]

I found out that GDAL.jl does not function well on an older Pentium 4 x86-64 CPU. The problem seems to be that CPU instructions that older x86-64 CPUs do not support, are included. This is typically caused by a setting of march that targets CPUs that are newers.

I posted an issue on GDAL.il [0] and opened a post at Julia Discourse [1].

According to the response from visr, the issue was caused by BinaryBuilder, through this line [2]. Specifically, the supported_platforms() does not seem to include older x86-64 CPUs.

I am not familier with BinaryProvider.jl. A quick search did not reveal which file contains the supported_platforms() setting.

I wonder if you could point me to the file?

Specifically, I am interested in adding generic march setting, x86-64, there as well as older CPUs such as my nocona.

[0] JuliaGeo/GDAL.jl#92
[1] https://discourse.julialang.org/t/illegal-cpu-instructions-when-testing-gdal-jl/41922
[2] https://github.com/JuliaPackaging/Yggdrasil/blob/32f1ae9ecd91998b26cc4f760b74225fca8829ef/G/GDAL/build_tarballs.jl#L61

[visr]

This issue probably needs to be transferred to https://github.com/JuliaPackaging/BinaryBuilder.jl, but I think someone with the right permissions can do that as well.

[xianwenchen]

My bad. I did not notice the difference. Shall I copy the content over to a new ticket for BinaryBuilder.jl and close this one?

[yuyichao]

This is JuliaLang/julia#35215 and someone had already reported it for GDAL on one of the linked issues.

[yuyichao]

Oh, ok, it's you who reported it.... I guess having another issue here is fine by me.......

[xianwenchen]

Thanks, Yuyi. I'm just following the leads and trying to talk to people that may help resolve the issue. Right now it seems that the issue is in the setting of supported_platforms().

[fingolfin]

I may be missing something here, so apologies if I am telling you stuff you already know below, or which for some reason is not applicable...

As you know, there are several "Pentium 4" variants, some with and some without 64bit support (the original Pentium 4 did not have it). The first ones with it are referred to as nocona in the GCC docs and can be selected via -march=nocona; however, as far as I understand, -march=x86-64 should do the essentially the same.

Now, BB these days can actually build different binaries for so-called "microarchitectures" (= sub variants of the primary architectures). E.g. to build binaries supporting the original Pentium4 (32bit only) resp. Prescott (a later Pentium 4 revision; early versions had 64bit support but disabled; it was enabled in a later revision), one can use these:

        Platform("i686", "linux"; libc="glibc", march="pentium4"),
        Platform("i686", "linux"; libc="glibc", march="prescott"),

To support generic x86_64 (which should include any 64bit capable Pentium 4), one can use this (and presumably that's what you are interested int):

        Platform("x86_64", "linux"; libc="glibc", march="x86_64"),

So could try to add these to the list of platforms -- but I've never tried this myself, so there may be pitfalls I am not aware of.

What also makes me wary is that I don't see any information about what x86_64 "microarchitecture" is the default, if any; naively I would hope that "generic x86_64" is already the default. But seeing as you already determined that you get crashes, that means either the default is different (but what is it?) or else that "generic x86_64" is not actually suitable.

In the latter case, the one thing I can think of to do would be to pass lock_microarchitecture=false to build_tarballs and then manually add -march=nocona to the CFLAGS (or whatever).

By the way, BinaryBuilderBase.jl (BBB) also offers a function expand_microarchitectures which takes a list of platforms and "expands" it to cover all supporter microarchitectures; but no packages in Yggdrasil currently uses it -- probably need to ask @giordano or @staticfloat about whether using that is a good idea at this point...

For example, taking a list of all Linux-with-glibc platforms and calling expand_microarchitectures on it currently gives this list (taken from the BBB test suite):

    @test sort(expand_microarchitectures(filter!(p -> Sys.islinux(p) && libc(p) == "glibc", supported_platforms())), by=triplet) == [
        Platform("aarch64", "linux"; libc="glibc", march="armv8_0"),
        Platform("aarch64", "linux"; libc="glibc", march="armv8_1"),
        Platform("aarch64", "linux"; libc="glibc", march="armv8_2_crypto"),
        Platform("aarch64", "linux"; libc="glibc", march="armv8_4_crypto_sve"),
        # Platform("armv6l", "linux"; libc="glibc", call_abi="eabihf", march="arm1176jzfs"),
        Platform("armv7l", "linux"; libc="glibc", call_abi="eabihf", march="armv7l"),
        Platform("armv7l", "linux"; libc="glibc", call_abi="eabihf", march="neonvfpv4"),
        Platform("i686", "linux"; libc="glibc", march="pentium4"),
        Platform("i686", "linux"; libc="glibc", march="prescott"),
        Platform("powerpc64le", "linux"; libc="glibc", march="power8"),
        Platform("x86_64", "linux"; libc="glibc", march="avx"),
        Platform("x86_64", "linux"; libc="glibc", march="avx2"),
        Platform("x86_64", "linux"; libc="glibc", march="avx512"),
        Platform("x86_64", "linux"; libc="glibc", march="x86_64"),
    ]

[staticfloat]

@xianwenchen Thanks for reporting this; I'm going to start from the beginning to help you figure out what's going on.

First off, I'm not certain this is due to a CPU instruction error. If that were the case, you'd be getting SIGILL (4) sent to your computer, rather than SIGABRT (6). This makes me think that this isn't due to the error that you think, and may in fact be due to something else, like a more mundane bug within PROJ itself.

I see that a new version of PROJ_jll was just released recently; can you see if a Pkg.update() installs v800.200.100+0, and if that fixes the issue for you?

If it doesn't, you can also try setting a preference to force PROJ_jll to load a different library more easily than the Overrides.toml stuff above. Just create a LocalPreferences.toml file next to your overall Project.toml file with the following contents:

[PROJ_jll]
libproj_path = "/path/to/my/libproj.so"

Note that you will need to add PROJ_jll as a direct dependency to your project (via Pkg.add("PROJ_jll")) for that to take effect.