New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue running the cython 0.28.1 tests on aarch64 (numpy_memoryview) #2308

Open
etu opened this Issue Jun 2, 2018 · 11 comments

Comments

Projects
None yet
5 participants
@etu

etu commented Jun 2, 2018

I don't know if this affects any versions prior to 0.28.1, because I haven't tested it.

But I'm having issues building and running the tests of cython 0.28.1 (python3) on aarch64 (specifically on an Raspberry Pi running NixOS).

I have reported this in the distribution bugtracker here: NixOS/nixpkgs#41347

But it's probably something upstream.

I have managed to disable the execution of tests which results in a cython that "works for what I use it for" but probably not entirely since the tests fails 😄

Here's a complete build log of me building cython and running the tests on my Raspberry Pi: https://ptpb.pw/jkjC

@scoder

This comment has been minimized.

Contributor

scoder commented Jun 3, 2018

@etu

This comment has been minimized.

etu commented Jun 4, 2018

I'm not familiar with the different architectures instructions sets or which features are available. I understand that ARM uses an entirely different instruction set than x86, and that different features may be available etc.

I'm not sure how to look this up or how that looking it up would help.

But can I then assume that you don't have any intention of finding some workarounds for architectures that doesn't support such datatypes?

Can I also assume that it's always been like this?

Is it just for running the tests that this datatype is needed or is it also in the actual software?

etu added a commit to etu/nixpkgs that referenced this issue Jun 4, 2018

python36Packages.cython: Disable tests on aarch64
The tests doesn't run on aarch64 and it's reported to upstream here:
cython/cython#2308

@etu etu referenced this issue Jun 4, 2018

Merged

python36Packages.cython: Disable tests on aarch64 #41453

2 of 8 tasks complete

etu added a commit to etu/nixpkgs that referenced this issue Jun 4, 2018

python36Packages.cython: Disable tests on aarch64
The tests doesn't run on aarch64 and it's reported to upstream here:
cython/cython#2308
@scoder

This comment has been minimized.

Contributor

scoder commented Jun 4, 2018

But can I then assume that you don't have any intention of finding some workarounds for architectures that doesn't support such datatypes?

We usually adapt tests when we notice that they are not cross-platform. And generating cross-platform code is obviously a project goal.

In this case, it's not immediately clear if the test is simply inappropriate for the platform, if it's a C compiler bug, or if there is an actual Cython bug hidden behind the test failure. Only further investigation can answer this.

But for unusual platforms that the core developers do not have access to (and/or do not have an interest in themselves), we rely on users of those platforms to do the debugging for us and/or tell us what we need to improve in what way. If no user of this platform has an interest in solving this problem, then it might simply not be worth solving. Just saying.

Can I also assume that it's always been like this?

Probably since the test was added.

Is it just for running the tests that this datatype is needed or is it also in the actual software?

It's only needed when used by user code, such as this test.

@etu

This comment has been minimized.

etu commented Jun 4, 2018

Cool, thanks for those clarifications. I'll try to adopt the nix build to skip the tests that fails on aarch64 when building on that architecture to get those packages working :)

@robertwb

This comment has been minimized.

Contributor

robertwb commented Jun 7, 2018

@etu

This comment has been minimized.

etu commented Jun 7, 2018

I have now confirmed that disabling the test named numpy_memoryview makes it build and run the tests on aarch64. I wouldn't say that it does it "fine" because building of the program takes about an hour on one core on my raspberry. But running the tests takes about 14.5 hours (52068 seconds) for the last run. Which sounds like it hits timeouts and stuff somewhere.

@tobihan

This comment has been minimized.

tobihan commented Jun 7, 2018

On Debian with cython 0.28.2 we have this test failing on many architectures:
arm64, ppc64el, ppc64, powerpc, alpha
See https://buildd.debian.org/status/package.php?p=cython&suite=experimental&ver=0.28.2-2 for the build logs.

The list of architectures is similar to the numpy issue with float128, so it might be just this issue:
numpy/numpy#11103

@volth

This comment has been minimized.

volth commented Jun 7, 2018

Also, "future_division" and "overflow_check_longlong" fail on i686
NixOS/nixpkgs#41633

@etu etu changed the title from Issue running the cython 0.28.1 tests on aarch64 to Issue running the cython 0.28.1 tests on non x86_64 architectures Jun 7, 2018

@tobihan

This comment has been minimized.

tobihan commented Jun 8, 2018

We were all talking about a specific test failing in this issue. If you have other tests failing on other architectures, you should probably open a new issue for that.

For the record, we were talking about this:

======================================================================
FAIL: test_coerce_to_numpy (numpy_memoryview)
Doctest: numpy_memoryview.test_coerce_to_numpy
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/nix/store/qdcj04rnk8kb34zjrfzg8gfgl6svfcr8-python3-3.6.5/lib/python3.6/doctest.py", line 2199, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for numpy_memoryview.test_coerce_to_numpy
  File "/tmp/nix-build-python3.6-Cython-0.28.1.drv-0/Cython-0.28.1/TEST_TMP/memoryview/c/numpy_memoryview/numpy_memoryview.cpython-36m-aarch64-linux-gnu.so", line unknown line number, in test_coerce_to_numpy

----------------------------------------------------------------------
File "/tmp/nix-build-python3.6-Cython-0.28.1.drv-0/Cython-0.28.1/TEST_TMP/memoryview/c/numpy_memoryview/numpy_memoryview.cpython-36m-aarch64-linux-gnu.so", line ?, in numpy_memoryview.test_coerce_to_numpy
Failed example:
    test_coerce_to_numpy()
Expected:
    [97, 98, 600, 700, 800]
    deallocating...
    (600, 700)
    deallocating...
    ((100, 200), (300, 400), 500)
    deallocating...
    (97, 900)
    deallocating...
    99
    deallocating...
    111
    deallocating...
    222
    deallocating...
    333
    deallocating...
    11.1
    deallocating...
    12.2
    deallocating...
    13.25
    deallocating...
    (14.4+15.5j)
    deallocating...
    (16.5+17.7j)
    deallocating...
    (18.8125+19.9375j)
    deallocating...
    22
    deallocating...
    33.33
    deallocating...
    44
    deallocating...
Got:
    [97, 98, 600, 700, 800]
    deallocating...
    (600, 700)
    deallocating...
    ((100, 200), (300, 400), 500)
    deallocating...
    (97, 900)
    deallocating...
    99
    deallocating...
    111
    deallocating...
    222
    deallocating...
    333
    deallocating...
    11.1
    deallocating...
    12.2
    deallocating...
    0.0
    deallocating...
    (14.4+15.5j)
    deallocating...
    (16.5+17.7j)
    deallocating...
    (0+0j)
    deallocating...
    22
    deallocating...
    33.33
    deallocating...
    44
    deallocating...

@etu etu changed the title from Issue running the cython 0.28.1 tests on non x86_64 architectures to Issue running the cython 0.28.1 tests on aarch64 (numpy_memoryview) Jun 8, 2018

@tobihan

This comment has been minimized.

tobihan commented Jun 10, 2018

I just tried with Numpy 1.14.4 rather than 1.14.3 on ppc64el. This version was supposed to fix numpy/numpy#11103

The output indeed changed. Now it just fails because the "+" is missing for the complex number.

======================================================================
FAIL: test_coerce_to_numpy (numpy_memoryview.__test__)
Doctest: numpy_memoryview.__test__.test_coerce_to_numpy
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/doctest.py", line 2226, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for numpy_memoryview.__test__.test_coerce_to_numpy
  File "/home/thansen/cython-0.28.2/build/work-dir/4/memoryview/c/numpy_memoryview/numpy_memoryview.so", line unknown line number, in test_coerce_to_numpy

----------------------------------------------------------------------
File "/home/thansen/cython-0.28.2/build/work-dir/4/memoryview/c/numpy_memoryview/numpy_memoryview.so", line ?, in numpy_memoryview.__test__.test_coerce_to_numpy
Failed example:
    test_coerce_to_numpy()
Expected:
    [97, 98, 600, 700, 800]
    deallocating...
    (600, 700)
    deallocating...
    ((100, 200), (300, 400), 500)
    deallocating...
    (97, 900)
    deallocating...
    99
    deallocating...
    111
    deallocating...
    222
    deallocating...
    333
    deallocating...
    11.1
    deallocating...
    12.2
    deallocating...
    13.25
    deallocating...
    (14.4+15.5j)
    deallocating...
    (16.5+17.7j)
    deallocating...
    (18.8125+19.9375j)
    deallocating...
    22
    deallocating...
    33.33
    deallocating...
    44
    deallocating...
Got:
    [97, 98, 600, 700, 800]
    deallocating...
    (600, 700)
    deallocating...
    ((100, 200), (300, 400), 500)
    deallocating...
    (97, 900)
    deallocating...
    99
    deallocating...
    111
    deallocating...
    222
    deallocating...
    333
    deallocating...
    11.1
    deallocating...
    12.2
    deallocating...
    13.25
    deallocating...
    (14.4+15.5j)
    deallocating...
    (16.5+17.7j)
    deallocating...
    (18.812519.9375j)
    deallocating...
    22
    deallocating...
    33.33
    deallocating...
    44
    deallocating...
@tobihan

This comment has been minimized.

tobihan commented Jun 10, 2018

DOUBLE_DOUBLE printing on powerpc was improved in numpy/numpy#11104 but they know that it's still broken.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment