Install error: __AVX2__ not defined on AVX2-supporting system

[flying-sheep]

Minimal, reproducible code sample, a copy-pastable example if possible

export CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fno-plt"
mkdir ../instdir
python3 setup.py install --root=../instdir --optimize=1

Problem description

When compiling, __AVX2__ seems to be not defined.

I assume -march=native or -mavx2 should be passed.

[numcodecs] setting up Blosc extension
[numcodecs] compiling Blosc extension with SSE2 support
[numcodecs] compiling Blosc extension with AVX2 support
[…]
running build_ext
building 'numcodecs.blosc' extension
[…]
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall \
    -march=x86-64 -mtune=generic -O3 -pipe -fno-plt \
    -march=x86-64 -mtune=generic -O3 -pipe -fno-plt \
    -march=x86-64 -mtune=generic -O3 -pipe -fno-plt \
    -march=x86-64 -mtune=generic -O2 -pipe -fno-plt \
    -D_FORTIFY_SOURCE=2 -fPIC -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 \
    -Ic-blosc/blosc \
    -Ic-blosc/internal-complibs/zstd-1.4.1 \
    -Ic-blosc/internal-complibs/zlib-1.2.8 \
    -Ic-blosc/internal-complibs/lz4-1.9.1 \
    -Ic-blosc/internal-complibs/snappy-1.1.1 \
    -Ic-blosc/internal-complibs/zstd-1.4.1/decompress \
    -Ic-blosc/internal-complibs/zstd-1.4.1/deprecated \
    -Ic-blosc/internal-complibs/zstd-1.4.1/compress \
    -Ic-blosc/internal-complibs/zstd-1.4.1/dll \
    -Ic-blosc/internal-complibs/zstd-1.4.1/legacy \
    -Ic-blosc/internal-complibs/zstd-1.4.1/dictBuilder \
    -Ic-blosc/internal-complibs/zstd-1.4.1/common \
    -I/usr/include/python3.8 -c c-blosc/blosc/shuffle-avx2.c \
    -o build/temp.linux-x86_64-3.8/c-blosc/blosc/shuffle-avx2.o \
    -DSHUFFLE_SSE2_ENABLED -DSHUFFLE_AVX2_ENABLED
c-blosc/blosc/shuffle-avx2.c:14:4: Error:
    #error AVX2 is not supported by the target architecture/platform and/or this compiler.
[…]
[numcodecs] command 'gcc' failed with exit status 1
Traceback (most recent call last):
  File "/usr/lib/python3.8/distutils/unixccompiler.py", line 117, in _compile
    self.spawn(compiler_so + cc_args + [src, '-o', obj] +
  File "/usr/lib/python3.8/distutils/ccompiler.py", line 910, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/usr/lib/python3.8/distutils/spawn.py", line 36, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/usr/lib/python3.8/distutils/spawn.py", line 157, in _spawn_posix
    raise DistutilsExecError(
distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "setup.py", line 286, in build_extension
    build_ext.build_extension(self, ext)
  File "/usr/lib/python3.8/distutils/command/build_ext.py", line 528, in build_extension
    objects = self.compiler.compile(sources,
  File "/usr/lib/python3.8/distutils/ccompiler.py", line 574, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/usr/lib/python3.8/distutils/unixccompiler.py", line 120, in _compile
    raise CompileError(msg)
distutils.errors.CompileError: command 'gcc' failed with exit status 1

Version and installation information

• numcodecs 0.6.4
• Python 3.8
• Arch Linux (current)
• flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts md_clear flush_l1d

[flying-sheep]

Hmm, no idea what’s happening, if I import setup.py it looks like this:

>>> import setup                                                                                                                                                                                                                                                
>>> setup.base_compile_args                                                                                                                                                                                                                                     
['-msse2', '-mavx2']
>>> setup.blosc_extension()[0].extra_compile_args                                                                                                                                                                                                                                     
['-msse2', '-mavx2', '-DSHUFFLE_SSE2_ENABLED', '-DSHUFFLE_AVX2_ENABLED']

[flying-sheep]

Ah! it’s because of CFLAGS! setup.py doesn’t set -mavx2 if CFLAGS is set.

Since the build needs -mavx2, I don’t think this is intended.

[jakirkham]

Sorry for the slow reply. Is there something we should be following up on here, @flying-sheep?

[flying-sheep]

You should probably warn or error if necessary features aren’t in CFLAGS.