New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1.0rc1: test_read failures on s390x, powerpc, and mips #3415

Open
olebole opened this Issue Feb 1, 2015 · 19 comments

Comments

Projects
None yet
4 participants
@olebole
Contributor

olebole commented Feb 1, 2015

On S390x, Powerpc, and mips, I found the following tests failing:

  • TestSingleTable.test_read_from_fileobj
  • TestMultipleHDU.test_read
  • TestMultipleHDU.test_read_with_hdu_1[1]
  • TestMultipleHDU.test_read_with_hdu_1[first]

They all seem to have the same cause, so I show only the first trace:

self = <astropy.io.fits.tests.test_connect.TestSingleTable object at 0x3fff70a2e48>
tmpdir = local('/tmp/pytest-0/test_read_from_fileobj0')

    def test_read_from_fileobj(self, tmpdir):
        filename = str(tmpdir.join('test_read_from_fileobj.fits'))
        hdu = BinTableHDU(self.data)
        hdu.writeto(filename)
        with open(filename, 'rb') as f:
            t = Table.read(f)
>       assert equal_data(t, self.data)
E       assert equal_data(<Table masked=False length=4>\n  a      b        c   \nint64 string32 float64\n--...a     2.3\n    2     b     4.5\n    3     c     6.7\n    4     d     8.9, array([(1, u'a', 2.3), (2, u'b', 4.5), (3, u'c', 6.7), (4, u'd', 8.9)], \n      dtype=[('a', '>i8'), ('b', '>U1'), ('c', '>f8')]))
E        +  where array([(1, u'a', 2.3), (2, u'b', 4.5), (3, u'c', 6.7), (4, u'd', 8.9)], \n      dtype=[('a', '>i8'), ('b', '>U1'), ('c', '>f8')]) = <astropy.io.fits.tests.test_connect.TestSingleTable object at 0x3fff70a2e48>.data

astropy/io/fits/tests/test_connect.py:130: AssertionError

Full log here for s390x, here for powerpc, and here for mips.

@olebole olebole changed the title from 1.0rc1: test_read failures on s390x and powerpc to 1.0rc1: test_read failures on s390x, powerpc, and mips Feb 1, 2015

@olebole

This comment has been minimized.

Show comment
Hide comment
@olebole

olebole Feb 12, 2015

Contributor

Not fixed with rc2 (just for reference, not that I'd expected it since the issue is still open)

Contributor

olebole commented Feb 12, 2015

Not fixed with rc2 (just for reference, not that I'd expected it since the issue is still open)

@astrofrog

This comment has been minimized.

Show comment
Hide comment
@astrofrog

astrofrog Feb 12, 2015

Member

@olebole - since I don't have access to one of these machines, could you add a debug print statement on line 21 of connect.py, so that the function looks like:

def equal_data(a, b):
    for name in a.dtype.names:
        print(a[name], b[name])
        if not np.all(a[name] == b[name]):
            return False
    return True
Member

astrofrog commented Feb 12, 2015

@olebole - since I don't have access to one of these machines, could you add a debug print statement on line 21 of connect.py, so that the function looks like:

def equal_data(a, b):
    for name in a.dtype.names:
        print(a[name], b[name])
        if not np.all(a[name] == b[name]):
            return False
    return True
@astrofrog

This comment has been minimized.

Show comment
Hide comment
@astrofrog

astrofrog Feb 12, 2015

Member

I'm currently trying to set up Qemu to emulate these systems, so may be able to debug interactively later today.

Member

astrofrog commented Feb 12, 2015

I'm currently trying to set up Qemu to emulate these systems, so may be able to debug interactively later today.

@olebole

This comment has been minimized.

Show comment
Hide comment
@olebole

olebole Feb 12, 2015

Contributor

I also hope to find some time tonight to have a look on a Debian porterbox.

Contributor

olebole commented Feb 12, 2015

I also hope to find some time tonight to have a look on a Debian porterbox.

@astrofrog

This comment has been minimized.

Show comment
Hide comment
@astrofrog

astrofrog Feb 12, 2015

Member

I finally managed to get an emulated machine running (and it's veeeery slow). Investigating.

Member

astrofrog commented Feb 12, 2015

I finally managed to get an emulated machine running (and it's veeeery slow). Investigating.

@astrofrog

This comment has been minimized.

Show comment
Hide comment
@astrofrog

astrofrog Feb 12, 2015

Member

So the issue is that for the failing comparison a[name] has dtype S4 and b[name] has dtype U1.

Member

astrofrog commented Feb 12, 2015

So the issue is that for the failing comparison a[name] has dtype S4 and b[name] has dtype U1.

@astrofrog

This comment has been minimized.

Show comment
Hide comment
@astrofrog

astrofrog Feb 12, 2015

Member

cc @embray in case you have ideas, still looking into it.

Member

astrofrog commented Feb 12, 2015

cc @embray in case you have ideas, still looking into it.

@astrofrog

This comment has been minimized.

Show comment
Hide comment
@astrofrog

astrofrog Feb 12, 2015

Member

I've been looking into this but am stumped - if I try and reproduce the error outside of the astropy tests and I don't get anywhere.

@olebole - let me know if you have any ideas!

Member

astrofrog commented Feb 12, 2015

I've been looking into this but am stumped - if I try and reproduce the error outside of the astropy tests and I don't get anywhere.

@olebole - let me know if you have any ideas!

@astrofrog

This comment has been minimized.

Show comment
Hide comment
@astrofrog

astrofrog Feb 12, 2015

Member

In any case, I classify this as non-critical for 1.0 since it doesn't seem to be a major issue, just some strange bug when comparing the arrays. But the data isn't corrupt or anything.

Member

astrofrog commented Feb 12, 2015

In any case, I classify this as non-critical for 1.0 since it doesn't seem to be a major issue, just some strange bug when comparing the arrays. But the data isn't corrupt or anything.

@olebole

This comment has been minimized.

Show comment
Hide comment
@olebole

olebole Feb 12, 2015

Contributor

I just made them xfail during the Debian build; however from the affected architectures this looks like an endianess problem.

Contributor

olebole commented Feb 12, 2015

I just made them xfail during the Debian build; however from the affected architectures this looks like an endianess problem.

@astrofrog

This comment has been minimized.

Show comment
Hide comment
@astrofrog

astrofrog Feb 12, 2015

Member

@olebole - the weird thing is that if I create arrays by hand with types |S4 and >U1 I don't get any issue.

Member

astrofrog commented Feb 12, 2015

@olebole - the weird thing is that if I create arrays by hand with types |S4 and >U1 I don't get any issue.

@astrofrog

This comment has been minimized.

Show comment
Hide comment
@astrofrog

astrofrog Feb 13, 2015

Member

For anyone debugging this, here is the output (on powerpc) of various diagnostic print statements on a[name] and b[name] (just a screenshot to save time, otherwise need to figure out transferring data out of the emulator):

screen shot 2015-02-13 at 9 48 04 am

Member

astrofrog commented Feb 13, 2015

For anyone debugging this, here is the output (on powerpc) of various diagnostic print statements on a[name] and b[name] (just a screenshot to save time, otherwise need to figure out transferring data out of the emulator):

screen shot 2015-02-13 at 9 48 04 am

@astrofrog

This comment has been minimized.

Show comment
Hide comment
@astrofrog

astrofrog Feb 13, 2015

Member

I should say that in the above log I also changed the assert in test_read_with_hdu_1 to use t.as_array() instead of t to get rid of any column types to simplify things.

Member

astrofrog commented Feb 13, 2015

I should say that in the above log I also changed the assert in test_read_with_hdu_1 to use t.as_array() instead of t to get rid of any column types to simplify things.

@astrofrog

This comment has been minimized.

Show comment
Hide comment
@astrofrog

astrofrog Feb 13, 2015

Member

It probably is related to the \x00\x00\x00a

Member

astrofrog commented Feb 13, 2015

It probably is related to the \x00\x00\x00a

@astrofrog

This comment has been minimized.

Show comment
Hide comment
@astrofrog

astrofrog Feb 13, 2015

Member

By the way, this may be related to #3419 which has failures in the same tests with numpy dev on MacOS X - but to clarify, the failures here are with Numpy stable.

Member

astrofrog commented Feb 13, 2015

By the way, this may be related to #3419 which has failures in the same tests with numpy dev on MacOS X - but to clarify, the failures here are with Numpy stable.

@astrofrog

This comment has been minimized.

Show comment
Hide comment
@astrofrog

astrofrog Feb 13, 2015

Member

Ok so on Mac, with a passing test, the S4 array is:

'a\x00\x00\x00b\x00\x00\x00c\x00\x00\x00d\x00\x00\x00'

while on the PowerPC machine it is

'\x00\x00\x00a\x00\x00\x00b\x00\x00\x00c\x00\x00\x00d'

However, I didn't think that endianness changed the order of the characters:

In [1]: import numpy as np

In [2]: x = np.array(['a','b','c','d'], dtype='<S4')

In [3]: y = np.array(['a','b','c','d'], dtype='>S4')

In [4]: x.tobytes()
Out[4]: b'a\x00\x00\x00b\x00\x00\x00c\x00\x00\x00d\x00\x00\x00'

In [5]: y.tobytes()
Out[5]: b'a\x00\x00\x00b\x00\x00\x00c\x00\x00\x00d\x00\x00\x00'

(this is on Mac)

so I'm not sure why the strings are right-aligned on the PowerPC.

@embray - do you have any ideas, with all the information above?

Member

astrofrog commented Feb 13, 2015

Ok so on Mac, with a passing test, the S4 array is:

'a\x00\x00\x00b\x00\x00\x00c\x00\x00\x00d\x00\x00\x00'

while on the PowerPC machine it is

'\x00\x00\x00a\x00\x00\x00b\x00\x00\x00c\x00\x00\x00d'

However, I didn't think that endianness changed the order of the characters:

In [1]: import numpy as np

In [2]: x = np.array(['a','b','c','d'], dtype='<S4')

In [3]: y = np.array(['a','b','c','d'], dtype='>S4')

In [4]: x.tobytes()
Out[4]: b'a\x00\x00\x00b\x00\x00\x00c\x00\x00\x00d\x00\x00\x00'

In [5]: y.tobytes()
Out[5]: b'a\x00\x00\x00b\x00\x00\x00c\x00\x00\x00d\x00\x00\x00'

(this is on Mac)

so I'm not sure why the strings are right-aligned on the PowerPC.

@embray - do you have any ideas, with all the information above?

@embray

This comment has been minimized.

Show comment
Hide comment
@embray

embray Feb 13, 2015

Member

@astrofrog In your test in the last comment you were making arrays of dtype S4, where endianness is irrelevant. But the array the test is failing on is of dtype U1 and for unicode strings endianness does matter. The bug here has to do with assuming endianness of unicode arrays.

Member

embray commented Feb 13, 2015

@astrofrog In your test in the last comment you were making arrays of dtype S4, where endianness is irrelevant. But the array the test is failing on is of dtype U1 and for unicode strings endianness does matter. The bug here has to do with assuming endianness of unicode arrays.

@bsipocz

This comment has been minimized.

Show comment
Hide comment
@bsipocz

bsipocz May 10, 2017

Member

@olebole - This is a rather old issue. Do you, or anyone else with access to such platform, still see these failures?

Member

bsipocz commented May 10, 2017

@olebole - This is a rather old issue. Do you, or anyone else with access to such platform, still see these failures?

@olebole

This comment has been minimized.

Show comment
Hide comment
@olebole

olebole May 10, 2017

Contributor

Please give me a week or so, I will test.

Contributor

olebole commented May 10, 2017

Please give me a week or so, I will test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment