Autopkgtest failure on big-endian (s390x) #48

ginggs · 2019-10-12T13:19:48Z

I see the autokpgtest failure below in Ubuntu 19.10 on s390x, with meshio 3.1.6,

From http://autopkgtest.ubuntu.com/packages/p/pygalmesh/eoan/s390x

___________________________ test_volume_from_surface ___________________________

    def test_volume_from_surface():
        this_dir = os.path.dirname(os.path.abspath(__file__))
        mesh = pygalmesh.generate_volume_mesh_from_surface_mesh(
            os.path.join(this_dir, "meshes", "elephant.vtu"),
            facet_angle=25.0,
            facet_size=0.15,
            facet_distance=0.008,
            cell_radius_edge_ratio=3.0,
>           verbose=False,
        )

test/test_volume_from_surface.py:15: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3/dist-packages/pygalmesh/main.py:188: in generate_volume_mesh_from_surface_mesh
    mesh = meshio.read(filename)
/usr/lib/python3/dist-packages/meshio/_helpers.py:192: in read
    return format_to_reader[file_format].read(filename)
/usr/lib/python3/dist-packages/meshio/_vtu.py:321: in read
    reader = VtuReader(filename)
/usr/lib/python3/dist-packages/meshio/_vtu.py:194: in __init__
    pts = self.read_data(data_array)
/usr/lib/python3/dist-packages/meshio/_vtu.py:308: in read_data
    data = self.read_binary(c.text.strip(), c.attrib["type"])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <meshio._vtu.VtuReader object at 0x3ffa2b92948>
data = 'AwAAAACAAAAoBAAAeXEAAA9yAAAzBAAAeJw8XHc81f/3r09GRdGS0k5pSSkl4aBSKZJVWmiIhMhOJSHZe8/svfc4d7ruNa+9ZZSsprSM373q++sPPR7XX...uFv5c5XRYTm8O/FHr9SXzuyj+SBUzg1t2MvzofniXICMS/m44AbhYvyj/JjjKHSV2Xv0cAN4sXC8O/zbBR1m8mvr8hWcAEbt3Jv/Flogip28k/XHIkXw=='
data_type = 'Float64'

    def read_binary(self, data, data_type):
        # first read the the block size; it determines the size of the header
        dtype = vtu_to_numpy_type[self.header_type]
        num_bytes_per_item = numpy.dtype(dtype).itemsize
        num_chars = num_bytes_to_num_base64_chars(num_bytes_per_item)
        byte_string = base64.b64decode(data[:num_chars])[:num_bytes_per_item]
        num_blocks = numpy.frombuffer(byte_string, dtype)[0]
    
        # read the entire header
        num_header_items = 3 + num_blocks
        num_header_bytes = num_bytes_per_item * num_header_items
        num_header_chars = num_bytes_to_num_base64_chars(num_header_bytes)
        byte_string = base64.b64decode(data[:num_header_chars])
>       header = numpy.frombuffer(byte_string, dtype)
E       ValueError: buffer size must be a multiple of element size

/usr/lib/python3/dist-packages/meshio/_vtu.py:270: ValueError

The text was updated successfully, but these errors were encountered:

nschloe · 2019-10-13T11:38:03Z

Hm, interesting. Looks like a rare bug in meshio. Do you have ssh-access to any of those machines?

ginggs · 2019-10-13T11:44:22Z

I do. Please let me know if there's anything I can test.

nschloe · 2019-10-13T11:47:06Z

Yep. In test/meshes/ you'll find elephant.vtu. What's the result of

import meshio
meshio.read("elephant.vtu")

on that machine?

ginggs · 2019-10-13T12:02:56Z

I tried with meshio 3.2.0.

$ python3
Python 3.7.5rc1 (default, Oct  8 2019, 16:47:45) 
[GCC 9.2.1 20190909] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import meshio
>>> meshio.read("elephant.vtu")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ginggs/meshio-3.2.0/meshio/_helpers.py", line 192, in read
    return format_to_reader[file_format].read(filename)
  File "/home/ginggs/meshio-3.2.0/meshio/_vtu.py", line 321, in read
    reader = VtuReader(filename)
  File "/home/ginggs/meshio-3.2.0/meshio/_vtu.py", line 194, in __init__
    pts = self.read_data(data_array)
  File "/home/ginggs/meshio-3.2.0/meshio/_vtu.py", line 308, in read_data
    data = self.read_binary(c.text.strip(), c.attrib["type"])
  File "/home/ginggs/meshio-3.2.0/meshio/_vtu.py", line 270, in read_binary
    header = numpy.frombuffer(byte_string, dtype)
ValueError: buffer size must be a multiple of element size

nschloe · 2019-10-13T12:29:24Z

Okay, so this is really a meshio bug. Can you print all of the involved variables plus

print(len(byte_string))

etc?

ginggs · 2019-10-13T12:52:15Z

For the first block, on s390x:

dtype: uint32
num_bytes_per_item: 4
num_chars: 8
num_blocks: 50331648
num_header_items: 50331651
num_header_bytes: 201326604
num_header_chars: 268435472
len(byte_string): 59347

on amd64:

dtype: uint32
num_bytes_per_item: 4
num_chars: 8
num_blocks: 3
num_header_items: 6
num_header_bytes: 24
num_header_chars: 32
len(byte_string): 24

nschloe · 2019-10-13T13:08:39Z

Okay, so it's the numblocks. What's the output of

dtype.byteorder
sys.byteorder
byte_string
numpy.frombuffer(byte_string, dtype)

? (Both s390x and amd64 would be nice.)

ginggs · 2019-10-13T13:17:01Z

s390x:

dtype.byteorder: =
sys.byteorder: big
byte_string: b'\x03\x00\x00\x00'
numpy.frombuffer(byte_string, dtype): [50331648]

amd64:

dtype.byteorder: =
sys.byteorder: little
byte_string: b'\x03\x00\x00\x00'
numpy.frombuffer(byte_string, dtype): [3]

nschloe · 2019-10-13T13:21:49Z

Possible fix: Add

        if self.byte_order is not None:
            dtype = dtype.newbyteorder(
                "<" if self.byte_order == "LittleEndian" else ">"
            )

after the dtype definition. Could you try that out?

ginggs · 2019-10-13T13:28:27Z

Now it seems to read the entire file, then:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ginggs/meshio-3.2.0/meshio/_helpers.py", line 192, in read
    return format_to_reader[file_format].read(filename)
  File "/home/ginggs/meshio-3.2.0/meshio/_vtu.py", line 338, in read
    reader = VtuReader(filename)
  File "/home/ginggs/meshio-3.2.0/meshio/_vtu.py", line 252, in __init__
    point_offsets, cells, cell_data_raw
  File "/home/ginggs/meshio-3.2.0/meshio/_vtu.py", line 50, in _organize_cells
    cls["connectivity"], cls["offsets"], cls["types"], cdr
  File "/home/ginggs/meshio-3.2.0/meshio/_vtu.py", line 32, in _cells_from_data
    meshio_type = vtk_to_meshio_type[vtk_type]
KeyError: 360287970189639680

nschloe · 2019-10-13T13:29:31Z

print(vtk_type)

?

nschloe · 2019-10-13T13:33:11Z

Ah, wait. The error is elsewhere.

ginggs · 2019-10-13T13:33:32Z

s390x:

vtk_type: 360287970189639680

amd64:

vtk_type: 5

nschloe · 2019-10-13T13:35:32Z

print(cells)

before the call to _organize_cells (line 250)?

ginggs · 2019-10-13T13:42:56Z

s390x:

cells: [{'connectivity': array([ 4540191374342881280, -4682617712558473216, -3962041772179193856,
       ...,  1298162592589545472,  7711006986988421120,
       -3383892170015506432]), 'offsets': array([ 216172782113783808,  432345564227567616,  648518346341351424, ...,
       2035908506548174848, 2252081288661958656, 2468254070775742464]), 'types': array([360287970189639680, 360287970189639680, 360287970189639680, ...,
       360287970189639680, 360287970189639680, 360287970189639680])}]

amd64:

cells: [{'connectivity': array([ 575, 1215, 1225, ..., 1042,  875, 2769]), 'offsets': array([    3,     6,     9, ..., 16668, 16671, 16674]), 'types': array([5, 5, 5, ..., 5, 5, 5])}]

nschloe · 2019-10-13T15:58:27Z

I really don't get it. I think the data is read alright, but I don't understand why the cell data is converted into the wrong endianness now.

Can I get a login on the respective machine or can I send you my public key to temporarily use your account?

ginggs · 2019-10-13T16:27:02Z

I'm using a Debian porterbox, but you can apply for guest access:
https://dsa.debian.org/doc/guest-account/

nschloe · 2019-10-13T19:10:04Z

Okay, did apply.

nschloe mentioned this issue May 21, 2020

Big endian swap nschloe/meshio#814

Merged

nschloe mentioned this issue Sep 15, 2020

test_from_array_with_subdomain_sizing segfaults on s390x #111

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Autopkgtest failure on big-endian (s390x) #48

Autopkgtest failure on big-endian (s390x) #48

ginggs commented Oct 12, 2019

nschloe commented Oct 13, 2019

ginggs commented Oct 13, 2019

nschloe commented Oct 13, 2019

ginggs commented Oct 13, 2019

nschloe commented Oct 13, 2019

ginggs commented Oct 13, 2019

nschloe commented Oct 13, 2019 •

edited

ginggs commented Oct 13, 2019

nschloe commented Oct 13, 2019

ginggs commented Oct 13, 2019

nschloe commented Oct 13, 2019

nschloe commented Oct 13, 2019

ginggs commented Oct 13, 2019

nschloe commented Oct 13, 2019 •

edited

ginggs commented Oct 13, 2019

nschloe commented Oct 13, 2019

ginggs commented Oct 13, 2019

nschloe commented Oct 13, 2019

Autopkgtest failure on big-endian (s390x) #48

Autopkgtest failure on big-endian (s390x) #48

Comments

ginggs commented Oct 12, 2019

nschloe commented Oct 13, 2019

ginggs commented Oct 13, 2019

nschloe commented Oct 13, 2019

ginggs commented Oct 13, 2019

nschloe commented Oct 13, 2019

ginggs commented Oct 13, 2019

nschloe commented Oct 13, 2019 • edited

ginggs commented Oct 13, 2019

nschloe commented Oct 13, 2019

ginggs commented Oct 13, 2019

nschloe commented Oct 13, 2019

nschloe commented Oct 13, 2019

ginggs commented Oct 13, 2019

nschloe commented Oct 13, 2019 • edited

ginggs commented Oct 13, 2019

nschloe commented Oct 13, 2019

ginggs commented Oct 13, 2019

nschloe commented Oct 13, 2019

nschloe commented Oct 13, 2019 •

edited

nschloe commented Oct 13, 2019 •

edited