Testsuite failures on selected architectures due to alignment problem

[ghisvail]

I have updated the Debian packaging with release 0.10.4 and the build now fails for some architectures, some of these considered major. On arm64, powerpc, ppc64el and s390x, the failing tests seem all related to alignment. If the package is intended to be not portable, please acknowledge it here and I'll restrict the build to the qualifying architectures.

Thanks,
Ghis

[hgomersall]

Ach, I've no idea why it's suddenly broken. The package is intended to be portable, but I've no mechanism for testing against anything but x86/x86-64. I know the alignment code for ARM is not optimised. What is the failure?

[ghisvail]

What is the failure?

You can click on individual Build-Attempted fields here to read the full build log. All seems to contain this:

======================================================================
ERROR: test_get_alignment (test.test_pyfftw_utils.UtilsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_utils.py", line 84, in test_get_alignment
    if 'avx' in each_cpu['flags']:
KeyError: 'flags'

======================================================================
FAIL: test_alignment (test.test_pyfftw_complex.Complex128FFTWTest)
Test to see if the alignment is returned correctly
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_complex.py", line 293, in test_alignment
    self.assertTrue(fft.input_alignment == 16)
AssertionError: False is not true

======================================================================
FAIL: test_alignment (test.test_pyfftw_complex.Complex64FFTWTest)
Test to see if the alignment is returned correctly
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_complex.py", line 293, in test_alignment
    self.assertTrue(fft.input_alignment == 16)
AssertionError: False is not true

======================================================================
FAIL: test_alignment (test.test_pyfftw_real_backward.RealBackwardDoubleFFTWTest)
Test to see if the alignment is returned correctly
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_complex.py", line 293, in test_alignment
    self.assertTrue(fft.input_alignment == 16)
AssertionError: False is not true

======================================================================
FAIL: test_alignment (test.test_pyfftw_real_backward.RealBackwardSingleFFTWTest)
Test to see if the alignment is returned correctly
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_complex.py", line 293, in test_alignment
    self.assertTrue(fft.input_alignment == 16)
AssertionError: False is not true

======================================================================
FAIL: test_alignment (test.test_pyfftw_real_forward.RealForwardDoubleFFTWTest)
Test to see if the alignment is returned correctly
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_complex.py", line 293, in test_alignment
    self.assertTrue(fft.input_alignment == 16)
AssertionError: False is not true

======================================================================
FAIL: test_alignment (test.test_pyfftw_real_forward.RealForwardSingleFFTWTest)
Test to see if the alignment is returned correctly
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_complex.py", line 293, in test_alignment
    self.assertTrue(fft.input_alignment == 16)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestFFT2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestFFTN)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestIFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestIFFT2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestIFFTN)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestIRFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestIRFFT2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestIRFFTN)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestRFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestRFFT2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestRFFTN)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_aligned_flag (test.test_pyfftw_class_misc.FFTWMiscTest)
Test to see if the aligned flag is correct
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_class_misc.py", line 72, in test_aligned_flag
    self.assertTrue(fft.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_flags (test.test_pyfftw_class_misc.FFTWMiscTest)
Test to see if the flags are correct
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_class_misc.py", line 83, in test_flags
    self.assertEqual(fft.flags, ('FFTW_MEASURE',))
AssertionError: Tuples differ: ('FFTW_MEASURE', 'FFTW_UNALIGN... != ('FFTW_MEASURE',)

First tuple contains 1 additional elements.
First extra element 1:
'FFTW_UNALIGNED'

- ('FFTW_MEASURE', 'FFTW_UNALIGNED')
+ ('FFTW_MEASURE',)

[hgomersall]

Ah, those are very specific tests for x86 like. I suspect it actually works just fine, but the tests should be modified to support alternative architectures - I knew this when I wrote them but didn't have the facility to implement them.

I suggest this issue serves as a rallying point for attempts to solve the problem. PRs gratefully received.

I'll get onto it at some point, but I'm pretty busy at the moment.

[ghisvail]

FYI, pyfftw has been automatically dropped from the next release of Debian due to the failures above. All build logs are still available here, if it can help. I'll keep the packaging updated with new releases but can't commit to fixing these issues myself.

[hgomersall]

@ghisvail Hey, sorry for the delay in getting back to you - I've been away for a couple of weeks. It seems rather extreme to drop to whole thing no? Surely there's some way to only support certain architectures? Is this fixed for the next release now?

[hgomersall]

@ghisvail Any movement on this? I make no claims about supporting anything other than x86/x86-64, can this be reflected?

[ghisvail]

I'll have to ask the release team to remove the packages from the failing architectures. At least I now have a public statement that upstream is not actively supporting anything other than i386 and x86_64, so that should justify it.

[hgomersall]

Thank you. Also for the record, I intend to support ARM at some point, but it is not high priority at the moment and needs a little more thought. As you suggest, it might work currently but YMMV. Patches welcome.

[ghisvail]

Here are the test failures currently on arm:

======================================================================
ERROR: test_get_alignment (test.test_pyfftw_utils.UtilsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_utils.py", line 84, in test_get_alignment
    if 'avx' in each_cpu['flags']:
KeyError: 'flags'

======================================================================
FAIL: test_alignment (test.test_pyfftw_complex.Complex128FFTWTest)
Test to see if the alignment is returned correctly
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_complex.py", line 293, in test_alignment
    self.assertTrue(fft.input_alignment == 16)
AssertionError: False is not true

======================================================================
FAIL: test_alignment (test.test_pyfftw_complex.Complex64FFTWTest)
Test to see if the alignment is returned correctly
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_complex.py", line 293, in test_alignment
    self.assertTrue(fft.input_alignment == 16)
AssertionError: False is not true

======================================================================
FAIL: test_alignment (test.test_pyfftw_real_backward.RealBackwardDoubleFFTWTest)
Test to see if the alignment is returned correctly
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_complex.py", line 293, in test_alignment
    self.assertTrue(fft.input_alignment == 16)
AssertionError: False is not true

======================================================================
FAIL: test_alignment (test.test_pyfftw_real_backward.RealBackwardSingleFFTWTest)
Test to see if the alignment is returned correctly
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_complex.py", line 293, in test_alignment
    self.assertTrue(fft.input_alignment == 16)
AssertionError: False is not true

======================================================================
FAIL: test_alignment (test.test_pyfftw_real_forward.RealForwardDoubleFFTWTest)
Test to see if the alignment is returned correctly
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_complex.py", line 293, in test_alignment
    self.assertTrue(fft.input_alignment == 16)
AssertionError: False is not true

======================================================================
FAIL: test_alignment (test.test_pyfftw_real_forward.RealForwardSingleFFTWTest)
Test to see if the alignment is returned correctly
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_complex.py", line 293, in test_alignment
    self.assertTrue(fft.input_alignment == 16)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestFFT2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestFFTN)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestIFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestIFFT2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestIFFTN)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestIRFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestIRFFT2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestIRFFTN)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestRFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestRFFT2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_auto_align_input (test.test_pyfftw_builders.BuildersTestRFFTN)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_builders.py", line 490, in test_auto_align_input
    self.assertTrue(FFTW_object.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_aligned_flag (test.test_pyfftw_class_misc.FFTWMiscTest)
Test to see if the aligned flag is correct
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_class_misc.py", line 72, in test_aligned_flag
    self.assertTrue(fft.simd_aligned)
AssertionError: False is not true

======================================================================
FAIL: test_flags (test.test_pyfftw_class_misc.FFTWMiscTest)
Test to see if the flags are correct
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_pyfftw_class_misc.py", line 83, in test_flags
    self.assertEqual(fft.flags, ('FFTW_MEASURE',))
AssertionError: Tuples differ: ('FFTW_MEASURE', 'FFTW_UNALIGN... != ('FFTW_MEASURE',)

First tuple contains 1 additional elements.
First extra element 1:
'FFTW_UNALIGNED'

- ('FFTW_MEASURE', 'FFTW_UNALIGNED')
+ ('FFTW_MEASURE',)

----------------------------------------------------------------------

[hgomersall]

@ghisvail thanks. I don't think it's a big effort to fix those, so anybody that wants to try is welcome to do so. It really boils down to adding ARM specific bits where there are x86 specific bits.

[ghisvail]

@hgomersall Just to let you know that pyfftw has been removed from the archive to clear the RC bug it was affected by. If this issue ever gets fixed, I am happy to reconsider this decision. But for now, the non-portability of the package makes it unfit for release as part of Debian.

[mhvk]

Got here as I noticed my python3-pyfftw package had ended up in "obsolete": It seems a pity not to have this package in Debian, especially if it is because x86 tests are executed on architectures where they should not. Sadly, I also don't have access to arm to help, but as a stop-gap measure, could one simply do an pytest.skipif(<architecture test>)?

[hgomersall]

Well, I would have thought so, but apparently not. Does this mean all code that goes into Debian needs to support ARM?

[ghisvail]

The rationales behind the removal are publicly listed here. pyfftw remains pip installable, so it has not gone anywhere.

[drew-parsons]

These alignment test failures still occur in pyFFTW 0.11.1, see the red build logs at https://buildd.debian.org/status/package.php?p=pyfftw. What scope of work is needed to fix the arch-specific alignment code in pyFFTW?

With regards to Debian support, we can deactivate specific arches if necessary, but the standard set of arches that Debian packages should normally support are amd64 arm64 armel armhf i386 mips mips64el mipsel ppc64el s390x. So normally support for ARM is expected. Support on any of the other arches (alpha, hppa, etc) is bonus.

[grlee77]

Can you try with PR #201? I didn't test it on ARM, so it has not been merged. If it does work, we can make a new release with the fix.

[drew-parsons]

The PR#201 patch helps in part, but some errors remain.

On mipsel, test_auto_align_input fails on builders:
test_auto_align_input (test.test_pyfftw_builders.BuildersTestFFT) ... FAIL
test_auto_align_input (test.test_pyfftw_builders.BuildersTestFFT2) ... FAIL
test_auto_align_input (test.test_pyfftw_builders.BuildersTestFFTN) ... FAIL
test_auto_align_input (test.test_pyfftw_builders.BuildersTestIFFT) ... FAIL
test_auto_align_input (test.test_pyfftw_builders.BuildersTestIFFT2) ... FAIL
test_auto_align_input (test.test_pyfftw_builders.BuildersTestIFFTN) ... FAIL
test_auto_align_input (test.test_pyfftw_builders.BuildersTestIRFFT) ... FAIL
test_auto_align_input (test.test_pyfftw_builders.BuildersTestIRFFT2) ... FAIL
test_auto_align_input (test.test_pyfftw_builders.BuildersTestIRFFTN) ... FAIL
test_auto_align_input (test.test_pyfftw_builders.BuildersTestRFFT) ... FAIL
test_auto_align_input (test.test_pyfftw_builders.BuildersTestRFFT2) ... FAIL
test_auto_align_input (test.test_pyfftw_builders.BuildersTestRFFTN) ... FAIL

but test_auto_align_input passes other interfaces.

test_alignment now passes, but test_get_alignment gives an error (also on armel):

ERROR: test_get_alignment (test.test_pyfftw_utils.UtilsTest)

Traceback (most recent call last):
  File "test/test_pyfftw_utils.py", line 83, in test_get_alignment
    cpus_info = get_cpus_info()
  File "test/test_pyfftw_utils.py", line 57, in get_cpus_info
    key, values = [each.strip() for each in line.split(':')]
ValueError: too many values to unpack

I'm attaching the mipsel build log.
pyfftw-build-mipsel.log

[drew-parsons]

On armel the #201 patch passes all test_auto_align_input (including builders) and all test_alignment.

test_get_alignment fails (rather than giving an "error" as on mipsel):

FAIL: test_get_alignment (test.test_pyfftw_utils.UtilsTest)
Traceback (most recent call last):
File "test/test_pyfftw_utils.py", line 91, in test_get_alignment
self.assertTrue(pyfftw.simd_alignment == 4)
AssertionError: False is not true

Build log attached
pyfftw-build-armel.log

[hgomersall]

I think there might be a bit of fiddling to get this working, though I suspect it isn't so hard - there is a bit of CPU inspection that goes on in the alignment code. What's the best hardware to test all this on? One really needs shell access to something to play around with the code.

[drew-parsons]

I've access to Debian armel and mipsel machines (and others) so I can proxy. I guess it's easier for you to fiddle directly, so I'll ask about access.

[drew-parsons]

Shouldn't be so hard to set up. https://dsa.debian.org/doc/guest-account/ I'll send you an email.

[mirabilos]

x32 (amd64 with ILP32 instead of LP64 model) is also affected:

======================================================================
FAIL: test_byte_align_set_dtype (test.test_pyfftw_nbyte_align.ByteAlignTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_pyfftw/build/test/test_pyfftw_nbyte_align.py", line 322, in test_byte_align_set_dtype
    self.assertTrue(d.ctypes.data % expected_alignment == 0)
AssertionError: False is not true

======================================================================
FAIL: test_n_byte_align_set_dtype (test.test_pyfftw_nbyte_align.ByteAlignTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_pyfftw/build/test/test_pyfftw_nbyte_align.py", line 55, in new_function
    return function(*args, **kwargs)
  File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_pyfftw/build/test/test_pyfftw_nbyte_align.py", line 342, in test_n_byte_align_set_dtype
    self.assertTrue(d.ctypes.data%n == 0)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 1603 tests in 835.438s

FAILED (failures=2, skipped=8)

Unfortunately, the assertTrue makes it hard to test this; on 0.11.1-4 the first test seems to pass.

You can test x32 on an amd64 machine in an x32 chroot, if you boot your amd64 kernel with the option syscall.x32=y added to GRUB_CMDLINE_LINUX.

[mirabilos]

Interesting… failures seem to depend on the system it’s built on (which means the package is dependent on the build system, which is a complete no-go for binary distributions).

On my x32 box I get:

======================================================================
FAIL: test_byte_align (test.test_pyfftw_nbyte_align.ByteAlignTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/pyfftw-0.12.0/.pybuild/cpython3_3.8_pyfftw/build/test/test_pyfftw_nbyte_align.py", line 157, in test_byte_align
    self.assertTrue(b.ctypes.data % expected_alignment == 0)
AssertionError: False is not true

======================================================================
FAIL: test_byte_align_different_dtypes (test.test_pyfftw_nbyte_align.ByteAlignTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/pyfftw-0.12.0/.pybuild/cpython3_3.8_pyfftw/build/test/test_pyfftw_nbyte_align.py", line 286, in test_byte_align_different_dtypes
    self.assertTrue(d.ctypes.data % expected_alignment == 0)
AssertionError: False is not true

======================================================================
FAIL: test_byte_align_integer_shape (test.test_pyfftw_nbyte_align.ByteAlignTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/pyfftw-0.12.0/.pybuild/cpython3_3.8_pyfftw/build/test/test_pyfftw_nbyte_align.py", line 175, in test_byte_align_integer_shape
    self.assertTrue(b.ctypes.data % expected_alignment == 0)
AssertionError: False is not true

======================================================================
FAIL: test_byte_align_set_dtype (test.test_pyfftw_nbyte_align.ByteAlignTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/pyfftw-0.12.0/.pybuild/cpython3_3.8_pyfftw/build/test/test_pyfftw_nbyte_align.py", line 326, in test_byte_align_set_dtype
    self.assertTrue(d.ctypes.data % expected_alignment == 0)
AssertionError: False is not true

======================================================================
FAIL: test_empty_aligned (test.test_pyfftw_nbyte_align.ByteAlignTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/pyfftw-0.12.0/.pybuild/cpython3_3.8_pyfftw/build/test/test_pyfftw_nbyte_align.py", line 129, in test_empty_aligned
    self.assertTrue(b.ctypes.data%n == 0)
AssertionError: False is not true

======================================================================
FAIL: test_n_byte_align (test.test_pyfftw_nbyte_align.ByteAlignTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/pyfftw-0.12.0/.pybuild/cpython3_3.8_pyfftw/build/test/test_pyfftw_nbyte_align.py", line 55, in new_function
    return function(*args, **kwargs)
  File "/tmp/buildd/pyfftw-0.12.0/.pybuild/cpython3_3.8_pyfftw/build/test/test_pyfftw_nbyte_align.py", line 166, in test_n_byte_align
    self.assertTrue(b.ctypes.data%n == 0)
AssertionError: False is not true

======================================================================
FAIL: test_n_byte_align_different_dtypes (test.test_pyfftw_nbyte_align.ByteAlignTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/pyfftw-0.12.0/.pybuild/cpython3_3.8_pyfftw/build/test/test_pyfftw_nbyte_align.py", line 55, in new_function
    return function(*args, **kwargs)
  File "/tmp/buildd/pyfftw-0.12.0/.pybuild/cpython3_3.8_pyfftw/build/test/test_pyfftw_nbyte_align.py", line 306, in test_n_byte_align_different_dtypes
    self.assertTrue(d.ctypes.data%n == 0)
AssertionError: False is not true

======================================================================
FAIL: test_n_byte_align_empty (test.test_pyfftw_nbyte_align.ByteAlignTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/pyfftw-0.12.0/.pybuild/cpython3_3.8_pyfftw/build/test/test_pyfftw_nbyte_align.py", line 55, in new_function
    return function(*args, **kwargs)
  File "/tmp/buildd/pyfftw-0.12.0/.pybuild/cpython3_3.8_pyfftw/build/test/test_pyfftw_nbyte_align.py", line 147, in test_n_byte_align_empty
    self.assertTrue(b.ctypes.data%n == 0)
AssertionError: False is not true

======================================================================
FAIL: test_n_byte_align_integer_shape (test.test_pyfftw_nbyte_align.ByteAlignTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/pyfftw-0.12.0/.pybuild/cpython3_3.8_pyfftw/build/test/test_pyfftw_nbyte_align.py", line 55, in new_function
    return function(*args, **kwargs)
  File "/tmp/buildd/pyfftw-0.12.0/.pybuild/cpython3_3.8_pyfftw/build/test/test_pyfftw_nbyte_align.py", line 184, in test_n_byte_align_integer_shape
    self.assertTrue(b.ctypes.data%n == 0)
AssertionError: False is not true

======================================================================
FAIL: test_n_byte_align_set_dtype (test.test_pyfftw_nbyte_align.ByteAlignTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/pyfftw-0.12.0/.pybuild/cpython3_3.8_pyfftw/build/test/test_pyfftw_nbyte_align.py", line 55, in new_function
    return function(*args, **kwargs)
  File "/tmp/buildd/pyfftw-0.12.0/.pybuild/cpython3_3.8_pyfftw/build/test/test_pyfftw_nbyte_align.py", line 342, in test_n_byte_align_set_dtype
    self.assertTrue(d.ctypes.data%n == 0)
AssertionError: False is not true

======================================================================
FAIL: test_ones_aligned (test.test_pyfftw_nbyte_align.ByteAlignTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/pyfftw-0.12.0/.pybuild/cpython3_3.8_pyfftw/build/test/test_pyfftw_nbyte_align.py", line 92, in test_ones_aligned
    self.assertTrue(b.ctypes.data%n == 0)
AssertionError: False is not true

======================================================================
FAIL: test_zeros_aligned (test.test_pyfftw_nbyte_align.ByteAlignTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/pyfftw-0.12.0/.pybuild/cpython3_3.8_pyfftw/build/test/test_pyfftw_nbyte_align.py", line 111, in test_zeros_aligned
    self.assertTrue(b.ctypes.data%n == 0)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 1603 tests in 865.954s

FAILED (failures=12, skipped=8)
One run: 9.81 ms (versus 27.02 ms for numpy.fft)
One run: 9.78 ms (versus 27.00 ms for numpy.fft)
One run: 4.70 ms (versus 26.28 ms for numpy.fft)
One run: 4.74 ms (versus 26.26 ms for numpy.fft)
One run: 2.82 ms (versus 9.76 ms for singled threaded)
One run: 5.12 ms (versus 9.75 ms for singled threaded)
One run: 2.61 ms (versus 9.73 ms for singled threaded)
One run: 3.09 ms (versus 9.72 ms for singled threaded)
One run: 1.36 ms (versus 4.71 ms for singled threaded)
One run: 2.56 ms (versus 4.70 ms for singled threaded)
One run: 1.26 ms (versus 4.71 ms for singled threaded)
One run: 1.53 ms (versus 4.69 ms for singled threaded)
One run: 11.17 ms (versus 42.23 ms for singled threaded)
One run: 21.39 ms (versus 42.23 ms for singled threaded)
One run: 10.78 ms (versus 42.24 ms for singled threaded)
One run: 12.46 ms (versus 42.23 ms for singled threaded)
One run: 4.81 ms (versus 17.05 ms for numpy.fft)
One run: 4.83 ms (versus 17.07 ms for numpy.fft)
One run: 3.05 ms (versus 19.93 ms for numpy.fft)
One run: 3.06 ms (versus 19.91 ms for numpy.fft)
One run: 5.15 ms (versus 13.00 ms for numpy.fft)
One run: 5.17 ms (versus 13.00 ms for numpy.fft)
One run: 3.26 ms (versus 15.25 ms for numpy.fft)
One run: 3.24 ms (versus 15.25 ms for numpy.fft)
E: pybuild pybuild:352: test: plugin distutils failed with: exit code=1: cd /tmp/buildd/pyfftw-0.12.0/.pybuild/cpython3_3.8_pyfftw/build; python3.8 -m unittest discover -v

[mirabilos]

Okaaaaay… it gets better.

When I ignore the test failure, let it create a python3-pyfftw_0.12.0-1_x32.deb package, install it, and then run the script below (which mirrors the test but with more output), it passes!

from pyfftw import (byte_align, is_byte_aligned, ones_aligned,
                    empty_aligned, zeros_aligned, simd_alignment,)
# Test the deprecated functions.
from pyfftw import n_byte_align, n_byte_align_empty, is_n_byte_aligned
import numpy

def get_expected_alignment(n):
    if n is None:
        return simd_alignment
    else:
        return n

shape = (10,10)
a = numpy.int16(numpy.random.randn(*shape)*16000)
b = numpy.float64(numpy.random.randn(*shape))
c = numpy.int8(numpy.random.randn(*shape)*255)
# Test a few alignments
for n in [None, 3, 7, 9, 16, 24, 23, 63, 64]:
    print("working on", n)
    expected_alignment = get_expected_alignment(n)
    print("expected alignment", expected_alignment)
    d = byte_align(a, dtype='float32', n=n)
    print("data", d.ctypes.data)
    print("alignment modulo", d.ctypes.data % expected_alignment)
    #self.assertTrue(d.ctypes.data % expected_alignment == 0)
    if d.ctypes.data % expected_alignment != 0:
        print("FAIL")
    print("float32?", d.dtype)
    #self.assertTrue(d.dtype == 'float32')

    d = byte_align(b, dtype='float32', n=n)
    print("data", d.ctypes.data)
    print("alignment modulo", d.ctypes.data % expected_alignment)
    if d.ctypes.data % expected_alignment != 0:
        print("FAIL")
    print("float32?", d.dtype)

    d = byte_align(c, dtype='float64', n=n)
    print("data", d.ctypes.data)
    print("alignment modulo", d.ctypes.data % expected_alignment)
    if d.ctypes.data % expected_alignment != 0:
        print("FAIL")
    print("float64?", d.dtype)

This is still in the clean/minimal build chroot.

WTF‽

[hgomersall]

Weird... Is the alignment lookup code doing something odd? It probes the CPU and isn't very well tested (it's rather hard to test such super platform specific stuff). Are you able to play around with that a bit?

[mirabilos]

I can install the built package and run a couple of test scripts, but I’m not very involved in Python and have no idea what fftw even is (just trying to keep the Debian/x32 port alive).

I do know that alignment can be tricky: for example, an int is normally 4-byte aligned, but only 2-byte aligned on m68k, and some x86 SSE operations need 16-byte alignment.

[hgomersall]

The lookup is done inside a c header, which is the potential problem point. It tries to detect capability (SSE or AVX) by probing the CPU registers and returns the correct alignment. That is exposed through simd_alignment. That should always be the same on the same machine.

[hgomersall]

@ghisvail What do we want to support? It would be good to extend to ARM, but do we want the other more obscure platforms?

[drew-parsons]

Debian build logs are collated at https://buildd.debian.org/status/package.php?p=pyfftw
The failing arches are armel armhf mipsel . These are the major supported arches.
Failing minor arches are alpha hppa riscv64 sparc64 (and of course x32).
Weird, that status page says x32 is fine, but the arch build log at https://buildd.debian.org/status/logs.php?pkg=pyfftw&arch=x32 shows it failing.

List of debian test machines is at https://db.debian.org/machines.cgi (look for porterbox), which has
major: armhf armel (via arm64 porterboxen) mipsel
minor: hppa sparc64

[hgomersall]

It also says the same for mips64el which is also failing.

[drew-parsons]

hmm, strange that the mips64el build didn't catch it as a failure

[mirabilos]

Drew Parsons dixit:

Weird, that status page says x32 is fine, but the arch build log at https://buildd.debian.org/status/logs.php?pkg=pyfftw&arch=x32 shows it failing.

That’s because I uploaded the package manually with tests disabled. Henry Gomersall dixit:

The lookup is done inside a [c header](https://github.com/pyFFTW/pyFFTW/blob/master/include/cpu.h),

It does that only for x86. ARM is not x86 and so would not trigger that code (statically return 4, which might not be correct either). x32 *is* x86… just 64-bit x86 with 32-bit pointers. (There’s an arm64ilp32 architecture in nascence, which is the equivalent for arm64.) bye, //mirabilos -- „Cool, /usr/share/doc/mksh/examples/uhr.gz ist ja ein Grund, mksh auf jedem System zu installieren.“ -- XTaran auf der OpenRheinRuhr, ganz begeistert (EN: “[…]uhr.gz is a reason to install mksh on every system.”)

[hgomersall]

The lookup is done inside a c header,
It does that only for x86. ARM is not x86 and so would not trigger that code (statically return 4, which might not be correct either). x32 is x86… just 64-bit x86 with 32-bit pointers. (There’s an arm64ilp32 architecture in nascence, which is the equivalent for arm64.)

Yeah right, so it (hopefully) just needs an equivalent bit of code for the other architectures.

[ghisvail]

@ghisvail What do we want to support? It would be good to extend to ARM, but do we want the other more obscure platforms?

You can't support architectures you don't test against.

However, you can ensure that architecture-sensitive code paths and tests are appropriately guarded so that builds on more exotic architectures don't accidentally fail when they should not or succeed when they should not have.

[fubingting]

@ghisvail What do we want to support? It would be good to extend to ARM, but do we want the other more obscure platforms?

You can't support architectures you don't test against.

However, you can ensure that architecture-sensitive code paths and tests are appropriately guarded so that builds on more exotic architectures don't accidentally fail when they should not or succeed when they should not have.

When I tested the arm64, I had the same error. Is there a solution now? Does it support arm64?

The lookup is done inside a c header,
It does that only for x86. ARM is not x86 and so would not trigger that code (statically return 4, which might not be correct either). x32 is x86… just 64-bit x86 with 32-bit pointers. (There’s an arm64ilp32 architecture in nascence, which is the equivalent for arm64.)

Yeah right, so it (hopefully) just needs an equivalent bit of code for the other architectures.

When I tested the arm64, I had the same error. Is there a solution now? Does it support arm64? @hgomersall

[hgomersall]

Not explicitly. If you want to fix it, it would be well received. I think it should be pretty simple.