Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix endianness issues in unit tests #615

Merged
merged 2 commits into from
Oct 8, 2021
Merged

fix endianness issues in unit tests #615

merged 2 commits into from
Oct 8, 2021

Conversation

obormot
Copy link
Collaborator

@obormot obormot commented Oct 7, 2021

No description provided.

@obormot obormot marked this pull request as draft October 7, 2021 20:42
@obormot obormot mentioned this pull request Oct 7, 2021
Closed
@coveralls
Copy link

coveralls commented Oct 7, 2021 •
edited
Loading

Coverage Status

Coverage increased (+0.0001%) to 99.815% when pulling 9547a51 on issue#505 into 04bd8ec on master.

@obormot
Copy link
Collaborator Author

obormot commented Oct 7, 2021

All pcapng unit tests now PASS on both big endian and little endian platforms.
The 2 loopback.py tests still fail.

# uname -m
s390x

# tox -e py38
GLOB sdist-make: /opt/dpkt/setup.py
py38 inst-nodeps: /opt/dpkt/.tox/.tmp/package/1/dpkt-1.9.7.2.zip
py38 installed: attrs==21.2.0,coverage==6.0.1,dpkt==1.9.7.2,iniconfig==1.1.1,packaging==21.0,pluggy==1.0.0,py==1.10.0,pyparsing==2.4.7,pytest==6.2.5,pytest-cov==3.0.0,toml==0.10.2
py38 run-test-pre: PYTHONHASHSEED='1413270695'
py38 run-test: commands[0] | py.test --cov=dpkt dpkt
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /opt/dpkt/.tox/py38/bin/python
cachedir: .tox/py38/.pytest_cache
rootdir: /opt/dpkt, configfile: setup.cfg
plugins: cov-3.0.0
collected 425 items
...
dpkt/loopback.py::test_ethernet_unpack FAILED                                                                                                                                                        [ 54%]
dpkt/loopback.py::test_ip_unpack PASSED                                                                                                                                                              [ 54%]
dpkt/loopback.py::test_ip6_unpack FAILED                                                                                                                                                             [ 54%]
...
dpkt/pcapng.py::test_shb PASSED                                                                                                                                                                      [ 61%]
dpkt/pcapng.py::test_idb PASSED                                                                                                                                                                      [ 61%]
dpkt/pcapng.py::test_epb PASSED                                                                                                                                                                      [ 62%]
dpkt/pcapng.py::test_simple_write_read PASSED                                                                                                                                                        [ 62%]
dpkt/pcapng.py::test_pcapng_header PASSED                                                                                                                                                            [ 62%]
dpkt/pcapng.py::test_filter PASSED                                                                                                                                                                   [ 62%]
dpkt/pcapng.py::test_dispatch PASSED                                                                                                                                                                 [ 63%]
dpkt/pcapng.py::test_loop PASSED                                                                                                                                                                     [ 63%]
dpkt/pcapng.py::test_writer_validate_instance PASSED                                                                                                                                                 [ 63%]
dpkt/pcapng.py::test_writepkt_epb_ts PASSED                                                                                                                                                          [ 63%]
dpkt/pcapng.py::test_writer_validate_le PASSED                                                                                                                                                       [ 64%]
dpkt/pcapng.py::test_writer_validate_be PASSED                                                                                                                                                       [ 64%]
dpkt/pcapng.py::test_writepkt_no_time PASSED                                                                                                                                                         [ 64%]
dpkt/pcapng.py::test_writepkt_snaplen PASSED                                                                                                                                                         [ 64%]
dpkt/pcapng.py::test_writepkt_with_time PASSED                                                                                                                                                       [ 64%]
dpkt/pcapng.py::test_writepkts PASSED                                                                                                                                                                [ 65%]
dpkt/pcapng.py::test_shb_header PASSED                                                                                                                                                               [ 65%]
dpkt/pcapng.py::test_shb_bom PASSED                                                                                                                                                                  [ 65%]
dpkt/pcapng.py::test_shb_version PASSED                                                                                                                                                              [ 65%]
dpkt/pcapng.py::test_no_idb PASSED                                                                                                                                                                   [ 66%]
dpkt/pcapng.py::test_idb_opt_offset PASSED                                                                                                                                                           [ 66%]
dpkt/pcapng.py::test_idb_linktype PASSED                                                                                                                                                             [ 66%]
dpkt/pcapng.py::test_readpkts PASSED                                                                                                                                                                 [ 66%]
dpkt/pcapng.py::test_next PASSED                                                                                                                                                                     [ 67%]
dpkt/pcapng.py::test_repr PASSED                                                                                                                                                                     [ 67%]
dpkt/pcapng.py::test_idb_opt_err PASSED                                                                                                                                                              [ 67%]
dpkt/pcapng.py::test_custom_read_write PASSED                                                                                                                                                        [ 67%]
dpkt/pcapng.py::test_multi_idb_writer PASSED                                                                                                                                                         [ 68%]
dpkt/pcapng.py::test_pcapng_block_pack PASSED                                                                                                                                                        [ 68%]
dpkt/pcapng.py::test_pcapng_block_unpack PASSED                                                                                                                                                      [ 68%]
dpkt/pcapng.py::test_epb_unpack PASSED                                                                                                                                                               [ 68%]
dpkt/pcapng.py::test_epb_unpack_length_mismatch PASSED                                                                                                                                               [ 68%]
dpkt/pcapng.py::test_pcapng_block_len_no_opts PASSED                                                                                                                                                 [ 69%]
dpkt/pcapng.py::test_reader_file_descriptor PASSED                                                                                                                                                   [ 69%]
dpkt/pcapng.py::test_posttest PASSED                                                                                                                                                                 [ 69%]
...
TOTAL                      10251     33    99%

========================================================================================= short test summary info ==========================================================================================
FAILED dpkt/loopback.py::test_ethernet_unpack - assert 131074 == 33554944
FAILED dpkt/loopback.py::test_ip6_unpack - assert 402653184 == 24
========================================================================== 2 failed, 423 passed, 4 warnings in 150.12s (0:02:30) ===========================================================================

@obormot obormot marked this pull request as ready for review October 8, 2021 02:15
@obormot
Copy link
Collaborator Author

obormot commented Oct 8, 2021

All unit tests now pass

# uname -m
s390x

# tox -e py38
...
=============================================================================== 425 passed, 4 warnings in 134.03s (0:02:14) ================================================================================
_________________________________________________________________________________________________ summary __________________________________________________________________________________________________
  py38: commands succeeded
  congratulations :)

@obormot obormot merged commit 8830f44 into master Oct 8, 2021
@mcepl
Copy link

mcepl commented Oct 8, 2021

Hmm, I have apparently missed something yet. With this patch on 1.9.7.2 tarball, I get:

[   77s] ____________________________ test_custom_read_write ____________________________
[   77s] 
[   77s]     def test_custom_read_write():
[   77s]         """Test a full pcapng file with 1 ICMP packet"""
[   77s]         buf = define_testdata().valid_pcapng
[   77s]         fobj = BytesIO(buf)
[   77s]     
[   77s]         # test reading
[   77s]         reader = Reader(fobj)
[   77s]         assert reader.snaplen == 0x40000
[   77s]         assert reader.datalink() == DLT_EN10MB
[   77s]     
[   77s]         assert reader.idb.opts[0].data.decode('utf-8') == '\\Device\\NPF_{3BBF21A7-91AE-4DDB-AB2C-C782999C22D5}'
[   77s]         assert reader.idb.opts[2].data.decode('utf-8') == '64-bit Windows 8.1, build 9600'
[   77s]     
[   77s]         ts, buf1 = next(iter(reader))
[   77s]         assert ts == 1442984653.2108380
[   77s]         assert len(buf1) == 74
[   77s]     
[   77s]         assert buf1.startswith(b'\x08\x00\x27\x96')
[   77s]         assert buf1.endswith(b'FGHI')
[   77s]         fobj.close()
[   77s]     
[   77s]         # test pcapng customized writing
[   77s]         shb, idb, epb = define_testdata().shb_idb_epb
[   77s]     
[   77s]         fobj = BytesIO()
[   77s]         writer = Writer(fobj, shb=shb, idb=idb)
[   77s]         writer.writepkt(epb, ts=1442984653.210838)
[   77s]         # .valid_pcapng buf was collected on a little endian system
[   77s] >       assert fobj.getvalue() == buf
[   77s] E       assert b"\n\r\r\n\x0...0\x00\x00\x84" == b"\n\r\r\n|\x...4\x00\x00\x00"
[   77s] E         At index 4 diff: b'\x00' != b'|'
[   77s] E         Full diff:
[   77s] E           (
[   77s] E         -  b'\n\r\r\n|\x00\x00\x00M<+\x1a\x01\x00\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff'
[   77s] E         ?            -            ^^^        ----
[   77s] E         +  b'\n\r\r\n\x00\x00\x00|\x1a+<M\x00\x01\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff'
[   77s] E         ?                        ^    +++++++...
[   77s] E         
[   77s] E         ...Full output truncated (36 lines hidden), use '-vv' to show
[   77s]

Fortunately, it is just the only test which fails now on s390x. Full build log illustrates all packages used and steps taken.

What I am missing?

@obormot
Copy link
Collaborator Author

obormot commented Oct 8, 2021 •
edited
Loading

Hmm, I have apparently missed something yet. With this patch on 1.9.7.2 tarball, I get:

[....]
[   77s]         # .valid_pcapng buf was collected on a little endian system
[   77s] >       assert fobj.getvalue() == buf
[   77s] E       assert b"\n\r\r\n\x0...0\x00\x00\x84" == b"\n\r\r\n|\x...4\x00\x00\x00"
[....]

Fortunately, it is just the only test which fails now on s390x. Full build log illustrates all packages used and steps taken.

What I am missing?

Looks like the diff wasn't fully applied? The assert above (assert fobj.getvalue() == buf) shouldn't have been run on the big endian system. I've added an if condition there but it's missing in the pytest output above

This: https://github.com/kbandla/dpkt/pull/615/files#diff-b7ac13862bd7e7717d0e8da77c39e82c27cd64af621798741ec74ae3df39c9a6R1151

@mcepl
Copy link

mcepl commented Oct 8, 2021

This: https://github.com/kbandla/dpkt/pull/615/files#diff-b7ac13862bd7e7717d0e8da77c39e82c27cd64af621798741ec74ae3df39c9a6R1151

You are right. Thank you, that helped.

bmwiedemann pushed a commit to bmwiedemann/openSUSE that referenced this pull request Oct 20, 2021
@bmwiedemann
Update python-dpkt to version 1.9.7.2 / rev 9 via SR 924285
d2094fe 
https://build.opensuse.org/request/show/924285
by user mcepl + dimstar_suse
- Update skip-BE-tests.patch so that it actually fixes things,
  instead of just skipping tests (gh#kbandla/dpkt#615).
- Update to 1.9.7.2:
  - Fixed performance regression (#611)
  - Moved the project documentation from Read the Docs(RST) to
    github.io(MarkDown)
  - Added a new mechanism for creating bit-sized field
    definitions in the protocol parsers (Packet.__bit_fields__)
  - Added pretty printing capability aka Packet.pprint(),
    Packet.__pprint_funcs__
  - Added documentation on developing protocol parsers in dpkt
    (creating_parsers.md)
  - Added a universal pcap+pcapng reader
    (dpkt.pcap.UniversalReader)
  - Improved TLS ClientHello and ServerHello parsing: return an
    "Unknown" ciphersuite instead of raising an exception, add
    codes for rfc8701, GREASE ciphersute
@obormot obormot deleted the issue#505 branch April 10, 2022 06:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@brifordwylie brifordwylie brifordwylie approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@obormot @coveralls @mcepl @brifordwylie