MRG, BUG: Explicit read endianness #8618

larsoner · 2020-12-07T14:06:23Z

Let's just be explicit about the encoding on disk everywhere by naming some constants for each reader.

@Aniket-Pradhan can you see if this branch fixes the issues? FYI we're going to release 0.22 in the next couple of weeks if it matters for packaging purposes. so it might be worth

Aniket-Pradhan · 2020-12-07T15:13:20Z

First of all, thanks for the speedy response <3

I have a small doubt regarding the tests. The tests are first writing the test data in the specific encoding and then the same test data is being read during the tests? If this is case, then it makes sense to fix the encoding while writing the files.

Other than that, the tests are currently failing, mainly because certain functions try to read the files directly in little-endian encoding. For example: egi.py, unpack in kit

I also ran a build with this branch which failed which I can see from the above reasons.

Below are the details about the build. The full log is here

=================================== FAILURES ===================================
_______________________________ test_find_layout _______________________________
mne/channels/tests/test_layout.py:255: in test_find_layout
    raw_kit = read_raw_kit(fname_kit_157)
mne/io/kit/kit.py:904: in read_raw_kit
    return RawKIT(input_fname=input_fname, mrk=mrk, elp=elp, hsp=hsp,
<decorator-gen-213>:24: in __init__
    ???
mne/io/kit/kit.py:139: in __init__
    info, kit_info = get_kit_info(
<decorator-gen-215>:24: in get_kit_info
    ???
mne/io/kit/kit.py:570: in get_kit_info
    sqd['dtype'] = np.dtype(f'<i{adc_allocated // 8}')
E   TypeError: data type '<i33554432' not understood
----------------------------- Captured stdout call -----------------------------
    Read a total of 3 projection items:
        PCA-v1 (1 x 102)  idle
        PCA-v2 (1 x 102)  idle
        PCA-v3 (1 x 102)  idle
    Read 5 compensation matrices
Extracting SQD Parameters from /builddir/build/BUILD/mne-python-0.21.2/mne/channels/tests/../../io/kit/tests/data/test.sqd...
Creating Raw.info structure...
____________________________ test_parse_annotation _____________________________
mne/io/edf/tests/test_edf.py:213: in test_parse_annotation
    assert_equal(np.column_stack((onset, duration, description)),
E   AssertionError: 
E   Arrays are not equal
E   
E   (shapes (3, 3), (6, 3) mismatch)
E    x: array([['1.0', '4402.0', '0onhtni'],
E          ['1.0', '4402.0', 'g\x00\x00\x00\x001+08.0\x152525.A'],
E          ['1.0', '4402.0', 'npae']], dtype='>U32')
E    y: array([['180.0', '0.0', 'Lights off'],
E          ['180.0', '0.0', 'Close door'],
E          ['180.0', '0.0', 'Lights off'],...
_______________________________ test_read_annot ________________________________
mne/io/edf/tests/test_edf.py:308: in test_read_annot
    _assert_annotations_equal(annotation, EXPECTED_ANNOTATIONS)
mne/io/edf/tests/test_edf.py:267: in _assert_annotations_equal
    assert_array_equal(a.onset, b.onset)
E   AssertionError: 
E   Arrays are not equal
E   
E   (shapes (3,), (6,) mismatch)
E    x: array([1., 1., 1.])
E    y: array([   3.14,  180.  ,  180.  ,  180.  ,  180.  , 1800.2 ])
_________________________________ test_io_egi __________________________________
mne/io/egi/tests/test_egi.py:146: in test_io_egi
    raw = read_raw_egi(egi_fname, include=None)
<decorator-gen-189>:24: in read_raw_egi
    ???
mne/io/egi/egi.py:153: in read_raw_egi
    return RawEGI(input_fname, eog, misc, include, exclude, preload,
<decorator-gen-190>:24: in __init__
    ???
mne/io/egi/egi.py:170: in __init__
    egi_info = _read_header(fid)
mne/io/egi/egi.py:52: in _read_header
    raise RuntimeError('Floating point precision is undefined.')
E   RuntimeError: Floating point precision is undefined.
----------------------------- Captured stdout call -----------------------------
Reading EGI header from /builddir/build/BUILD/mne-python-0.21.2/mne/io/egi/tests/data/test_egi.raw...
_________________________________ test_io_mrk __________________________________
mne/io/kit/tests/test_coreg.py:26: in test_io_mrk
    pts = read_mrk(mrk_fname)
mne/io/kit/coreg.py:40: in read_mrk
    mrk_count = unpack('i', fid.read(KIT.INT))[0]
E   struct.error: unpack requires a buffer of 4 bytes
_________________________________ test_epochs __________________________________
mne/io/kit/tests/test_kit.py:205: in test_epochs
    raw = read_raw_kit(sqd_path, stim=None)
mne/io/kit/kit.py:904: in read_raw_kit
    return RawKIT(input_fname=input_fname, mrk=mrk, elp=elp, hsp=hsp,
<decorator-gen-213>:24: in __init__
    ???
mne/io/kit/kit.py:139: in __init__
    info, kit_info = get_kit_info(
<decorator-gen-215>:24: in get_kit_info
    ???
mne/io/kit/kit.py:570: in get_kit_info
    sqd['dtype'] = np.dtype(f'<i{adc_allocated // 8}')
E   TypeError: data type '<i33554432' not understood
----------------------------- Captured stdout call -----------------------------
Extracting SQD Parameters from /builddir/build/BUILD/mne-python-0.21.2/mne/io/kit/tests/data/test.sqd...
Creating Raw.info structure...
_______________________________ test_raw_events ________________________________
mne/io/kit/tests/test_kit.py:222: in test_raw_events
    raw = read_raw_kit(sqd_path)
mne/io/kit/kit.py:904: in read_raw_kit
    return RawKIT(input_fname=input_fname, mrk=mrk, elp=elp, hsp=hsp,
<decorator-gen-213>:24: in __init__
    ???
mne/io/kit/kit.py:139: in __init__
    info, kit_info = get_kit_info(
<decorator-gen-215>:24: in get_kit_info
    ???
mne/io/kit/kit.py:570: in get_kit_info
    sqd['dtype'] = np.dtype(f'<i{adc_allocated // 8}')
E   TypeError: data type '<i33554432' not understood
----------------------------- Captured stdout call -----------------------------
Extracting SQD Parameters from /builddir/build/BUILD/mne-python-0.21.2/mne/io/kit/tests/data/test.sqd...
Creating Raw.info structure...
_________________________________ test_ch_loc __________________________________
mne/io/kit/tests/test_kit.py:246: in test_ch_loc
    raw_py = read_raw_kit(sqd_path, mrk_path, elp_txt_path, hsp_txt_path,
mne/io/kit/kit.py:904: in read_raw_kit
    return RawKIT(input_fname=input_fname, mrk=mrk, elp=elp, hsp=hsp,
<decorator-gen-213>:24: in __init__
    ???
mne/io/kit/kit.py:139: in __init__
    info, kit_info = get_kit_info(
<decorator-gen-215>:24: in get_kit_info
    ???
mne/io/kit/kit.py:570: in get_kit_info
    sqd['dtype'] = np.dtype(f'<i{adc_allocated // 8}')
E   TypeError: data type '<i33554432' not understood
----------------------------- Captured stdout call -----------------------------
Extracting SQD Parameters from /builddir/build/BUILD/mne-python-0.21.2/mne/io/kit/tests/data/test.sqd...
Creating Raw.info structure...
_________________________________ test_hsp_elp _________________________________
mne/io/kit/tests/test_kit.py:275: in test_hsp_elp
    raw_txt = read_raw_kit(sqd_path, mrk_path, elp_txt_path, hsp_txt_path)
mne/io/kit/kit.py:904: in read_raw_kit
    return RawKIT(input_fname=input_fname, mrk=mrk, elp=elp, hsp=hsp,
<decorator-gen-213>:24: in __init__
    ???
mne/io/kit/kit.py:139: in __init__
    info, kit_info = get_kit_info(
<decorator-gen-215>:24: in get_kit_info
    ???
mne/io/kit/kit.py:570: in get_kit_info
    sqd['dtype'] = np.dtype(f'<i{adc_allocated // 8}')
E   TypeError: data type '<i33554432' not understood
----------------------------- Captured stdout call -----------------------------
Extracting SQD Parameters from /builddir/build/BUILD/mne-python-0.21.2/mne/io/kit/tests/data/test.sqd...
Creating Raw.info structure...
________________________________ test_decimate _________________________________
mne/io/kit/tests/test_kit.py:308: in test_decimate
    raw = read_raw_kit(sqd_path, mrk_path, elp_txt_path, sphere_hsp_path)
mne/io/kit/kit.py:904: in read_raw_kit
    return RawKIT(input_fname=input_fname, mrk=mrk, elp=elp, hsp=hsp,
<decorator-gen-213>:24: in __init__
    ???
mne/io/kit/kit.py:139: in __init__
    info, kit_info = get_kit_info(
<decorator-gen-215>:24: in get_kit_info
    ???
mne/io/kit/kit.py:570: in get_kit_info
    sqd['dtype'] = np.dtype(f'<i{adc_allocated // 8}')
E   TypeError: data type '<i33554432' not understood
----------------------------- Captured stdout call -----------------------------
Extracting SQD Parameters from /builddir/build/BUILD/mne-python-0.21.2/mne/io/kit/tests/data/test.sqd...
Creating Raw.info structure...
________________________________ test_pick_refs ________________________________
mne/io/tests/test_pick.py:99: in test_pick_refs
    raw_kit = read_raw_kit(sqd_path, mrk_path, elp_path, hsp_path)
mne/io/kit/kit.py:904: in read_raw_kit
    return RawKIT(input_fname=input_fname, mrk=mrk, elp=elp, hsp=hsp,
<decorator-gen-213>:24: in __init__
    ???
mne/io/kit/kit.py:139: in __init__
    info, kit_info = get_kit_info(
<decorator-gen-215>:24: in get_kit_info
    ???
mne/io/kit/kit.py:570: in get_kit_info
    sqd['dtype'] = np.dtype(f'<i{adc_allocated // 8}')
E   TypeError: data type '<i33554432' not understood
----------------------------- Captured stdout call -----------------------------
Extracting SQD Parameters from /builddir/build/BUILD/mne-python-0.21.2/mne/io/tests/../kit/tests/data/test.sqd...
Creating Raw.info structure...
_______________________ test_read_raw_supported[fname3] ________________________
mne/io/tests/test_read_raw.py:35: in test_read_raw_supported
    read_raw(fname)
mne/io/_read_raw.py:82: in read_raw
    return readers[ext](fname, preload=preload, verbose=verbose, **kwargs)
mne/io/kit/kit.py:904: in read_raw_kit
    return RawKIT(input_fname=input_fname, mrk=mrk, elp=elp, hsp=hsp,
<decorator-gen-213>:24: in __init__
    ???
mne/io/kit/kit.py:139: in __init__
    info, kit_info = get_kit_info(
<decorator-gen-215>:24: in get_kit_info
    ???
mne/io/kit/kit.py:570: in get_kit_info
    sqd['dtype'] = np.dtype(f'<i{adc_allocated // 8}')
E   TypeError: data type '<i33554432' not understood
----------------------------- Captured stdout call -----------------------------
Extracting SQD Parameters from /builddir/build/BUILD/mne-python-0.21.2/mne/io/kit/tests/data/test.sqd...
Creating Raw.info structure...
______________________________ test_other_systems ______________________________
mne/preprocessing/tests/test_maxwell.py:235: in test_other_systems
    raw_kit = read_raw_kit(sqd_path, mrk_path, elp_path, hsp_path)
mne/io/kit/kit.py:904: in read_raw_kit
    return RawKIT(input_fname=input_fname, mrk=mrk, elp=elp, hsp=hsp,
<decorator-gen-213>:24: in __init__
    ???
mne/io/kit/kit.py:139: in __init__
    info, kit_info = get_kit_info(
<decorator-gen-215>:24: in get_kit_info
    ???
mne/io/kit/kit.py:570: in get_kit_info
    sqd['dtype'] = np.dtype(f'<i{adc_allocated // 8}')
E   TypeError: data type '<i33554432' not understood
----------------------------- Captured stdout call -----------------------------
Extracting SQD Parameters from /builddir/build/BUILD/mne-python-0.21.2/mne/preprocessing/tests/../../io/kit/tests/data/test.sqd...
Creating Raw.info structure...
=============================== warnings summary ===============================
mne/utils/docs.py:1946
  /builddir/build/BUILD/mne-python-0.21.2/mne/utils/docs.py:1946: DeprecationWarning: `unindent_dict` is deprecated!
  scipy.misc.unindent_dict is deprecated in SciPy 1.3.0
    docdict = unindent_dict(docdict)
mne/utils/docs.py:1947
  /builddir/build/BUILD/mne-python-0.21.2/mne/utils/docs.py:1947: DeprecationWarning: `filldoc` is deprecated!
  scipy.misc.filldoc is deprecated in SciPy 1.3.0
    fill_doc = filldoc(docdict, unindent_params=False)
mne/datasets/utils.py:62
  /builddir/build/BUILD/mne-python-0.21.2/mne/datasets/utils.py:62: DeprecationWarning: `docformat` is deprecated!
  scipy.misc.docformat is deprecated in Scipy 1.3.0
    _data_path_doc = docformat(_data_path_doc, docdict)
mne/datasets/utils.py:63
  /builddir/build/BUILD/mne-python-0.21.2/mne/datasets/utils.py:63: DeprecationWarning: `docformat` is deprecated!
  scipy.misc.docformat is deprecated in Scipy 1.3.0
    _data_path_doc_accept = docformat(_data_path_doc_accept, docdict)
mne/conftest.py:459
  /builddir/build/BUILD/mne-python-0.21.2/mne/conftest.py:459: PytestUnknownMarkWarning: Unknown pytest.mark.slowtest - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/mark.html
    @pytest.mark.slowtest
-- Docs: https://docs.pytest.org/en/stable/warnings.html
- generated xml file: /builddir/build/BUILD/mne-python-0.21.2/junit-results.xml -

larsoner · 2020-12-07T15:20:28Z

I have a small doubt regarding the tests. The tests are first writing the test data in the specific encoding and then the same test data is being read during the tests? If this is case, then it makes sense to fix the encoding while writing the files.

In general, no, we don't write files then read them for all formats. We only really do this for FIF. I see the unpack calls I missed, will push a commit shortly...

larsoner · 2020-12-07T15:42:40Z

Can you try again @Aniket-Pradhan ?

Aniket-Pradhan · 2020-12-07T16:11:11Z

There's one more unpack call in mne/io/kit/coreg.py.

The tests are passing now, I guess once the above unpack call is fixed, it should be ready to go then.

Aniket-Pradhan · 2020-12-07T17:10:20Z

Almost all the failing tests seem to be working well now. Some tests still fail on the s390x arch, which I guess again might be because of the endian-ness, but am not sure. Can you have a look at them?

Here's the complete build log.

Below's the info on the failing tests:

=================================== FAILURES ===================================
_________________________________ test_ch_loc __________________________________
mne/io/kit/tests/test_kit.py:257: in test_ch_loc
    assert_array_almost_equal(raw_py.info['dev_head_t']['trans'],
E   AssertionError: 
E   Arrays are not almost equal to 4 decimals
E   
E   Mismatched elements: 12 / 16 (75%)
E   Max absolute difference: 3.28560431e+289
E   Max relative difference: 4.49241617e+292
E    x: array([[ 4.5494e-001, -7.9305e-001,  4.0509e-001, -3.2856e+289],
E          [ 7.7721e-001,  5.7565e-001,  2.5411e-001,  2.3849e+289],
E          [-4.3471e-001,  1.9923e-001,  8.7825e-001,  8.2542e+288],
E          [ 0.0000e+000,  0.0000e+000,  0.0000e+000,  1.0000e+000]])
E    y: array([[ 9.9821e-01, -3.7228e-02, -4.6765e-02, -7.3137e-04],
E          [ 5.3944e-02,  8.9808e-01,  4.3652e-01,  1.6010e-02],
E          [ 2.5748e-02, -4.3826e-01,  8.9848e-01,  6.1303e-02],
E          [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  1.0000e+00]])
----------------------------- Captured stdout call -----------------------------
Extracting SQD Parameters from /builddir/build/BUILD/mne-python-0.21.2/mne/io/kit/tests/data/test.sqd...
Creating Raw.info structure...
Setting channel info structure...
Creating Info structure...
Ready.
Opening raw data file /builddir/build/BUILD/mne-python-0.21.2/mne/io/kit/tests/data/test_bin_raw.fif...
    Range : 0 ... 1999 =      0.000 ...     1.999 secs
Ready.
_________________________________ test_hsp_elp _________________________________
mne/io/kit/tests/test_kit.py:286: in test_hsp_elp
    assert_array_almost_equal(trans_elp, trans_txt, decimal=5)
E   AssertionError: 
E   Arrays are not almost equal to 5 decimals
E   
E   Mismatched elements: 9 / 16 (56.2%)
E   Max absolute difference: 1.63461466e+284
E   Max relative difference: 2.28222862
E    x: array([[-5.83337e-001, -7.93050e-001, -1.75469e-001, -3.28559e+289],
E          [-8.11881e-001,  5.75654e-001,  9.73246e-002,  2.38492e+289],
E          [ 2.38260e-002,  1.99233e-001, -9.79662e-001,  8.25418e+288],
E          [ 0.00000e+000,  0.00000e+000,  0.00000e+000,  1.00000e+000]])
E    y: array([[ 4.54940e-001, -7.93053e-001,  4.05088e-001, -3.28560e+289],
E          [ 7.77210e-001,  5.75650e-001,  2.54111e-001,  2.38491e+289],
E          [-4.34713e-001,  1.99233e-001,  8.78255e-001,  8.25419e+288],
E          [ 0.00000e+000,  0.00000e+000,  0.00000e+000,  1.00000e+000]])
----------------------------- Captured stdout call -----------------------------
Extracting SQD Parameters from /builddir/build/BUILD/mne-python-0.21.2/mne/io/kit/tests/data/test.sqd...
Creating Raw.info structure...
Setting channel info structure...
Creating Info structure...
Ready.
Extracting SQD Parameters from /builddir/build/BUILD/mne-python-0.21.2/mne/io/kit/tests/data/test.sqd...
Creating Raw.info structure...
Setting channel info structure...
Creating Info structure...
Ready.
______________________________ test_other_systems ______________________________
mne/preprocessing/tests/test_maxwell.py:239: in test_other_systems
    raw_sss = maxwell_filter(raw_kit, origin=(0., 0., 0.04),
<decorator-gen-407>:22: in maxwell_filter
    ???
mne/preprocessing/maxwell.py:214: in maxwell_filter
    params = _prep_maxwell_filter(
<decorator-gen-408>:24: in _prep_maxwell_filter
    ???
mne/preprocessing/maxwell.py:346: in _prep_maxwell_filter
    S_recon = _trans_sss_basis(exp, all_coils, recon_trans, coil_scale)
mne/preprocessing/maxwell.py:1922: in _trans_sss_basis
    S_tot = _sss_basis(exp, all_coils)
mne/preprocessing/maxwell.py:1229: in _sss_basis
    L = _tabular_legendre(rmags, max_order)
mne/preprocessing/maxwell.py:1346: in _tabular_legendre
    r_n = np.sqrt(np.sum(r * r, axis=1))
E   RuntimeWarning: overflow encountered in multiply
----------------------------- Captured stdout call -----------------------------
Extracting SQD Parameters from /builddir/build/BUILD/mne-python-0.21.2/mne/preprocessing/tests/../../io/kit/tests/data/test.sqd...
Creating Raw.info structure...
Setting channel info structure...
Creating Info structure...
Ready.
Maxwell filtering raw data
    No bad MEG channels
    Processing 0 gradiometers and 160 magnetometers (of which 157 are actually KIT gradiometers)
    Automatic origin fit: head of radius 89.7 mm
    Using origin 155.8, -28.3, 102.2 mm in the head frame

larsoner · 2020-12-07T20:26:51Z

Okay caught one more, can you try again @Aniket-Pradhan ?

Aniket-Pradhan · 2020-12-08T07:16:36Z

This fixes everything!

Thanks a lot for providing the fixes @larsoner 🥳

I'm guessing these fixes will be available in 0.22.* release?

agramfort · 2020-12-08T08:27:26Z

thanks @larsoner !

feel free to backport this and make a minor release if you want

BUG: Explicit read endianness

bddb5cd

larsoner added this to the 0.22 milestone Dec 7, 2020

FIX: Move

5872216

FIX: A few more

8292530

FIX: Unpack

3880e96

FIX: Missed one

9ec3b3b

agramfort approved these changes Dec 8, 2020

View reviewed changes

agramfort merged commit 2ae5c40 into mne-tools:master Dec 8, 2020
2 checks passed

larsoner deleted the endian branch December 8, 2020 12:45

MRG, BUG: Explicit read endianness #8618

MRG, BUG: Explicit read endianness #8618

larsoner commented Dec 7, 2020

Aniket-Pradhan commented Dec 7, 2020

larsoner commented Dec 7, 2020

larsoner commented Dec 7, 2020

Aniket-Pradhan commented Dec 7, 2020

Aniket-Pradhan commented Dec 7, 2020

larsoner commented Dec 7, 2020

Aniket-Pradhan commented Dec 8, 2020

agramfort commented Dec 8, 2020

MRG, BUG: Explicit read endianness #8618

MRG, BUG: Explicit read endianness #8618

Conversation

larsoner commented Dec 7, 2020

Aniket-Pradhan commented Dec 7, 2020

larsoner commented Dec 7, 2020

larsoner commented Dec 7, 2020

Aniket-Pradhan commented Dec 7, 2020

Aniket-Pradhan commented Dec 7, 2020

larsoner commented Dec 7, 2020

Aniket-Pradhan commented Dec 8, 2020

agramfort commented Dec 8, 2020