tests/gnrc_rpl_srh failures

[kb2ma]

Description

Working on 2019.10 release, I found three errors in tests/gnrc_rpl_srh. I confirmed they also exist in master. Does anyone have any insight?

Steps to reproduce the issue

Build for native board.

cd tests/gnrc_rpl_srh
make
sudo make test

Expected results

We expect the test to run cleanly, and finish with a 'SUCCESS' result.

Actual results

Traceback (most recent call last):
  File "/home/kbee/dev/riot/repo/tests/gnrc_rpl_srh/tests/01-run.py", line 376, in <module>
    sys.exit(run(testfunc, timeout=1, echo=False))
  File "/home/kbee/dev/riot/repo/dist/pythonlibs/testrunner/__init__.py", line 29, in run
    testfunc(child)
  File "/home/kbee/dev/riot/repo/tests/gnrc_rpl_srh/tests/01-run.py", line 357, in testfunc
    run(test_multicast_dst)
  File "/home/kbee/dev/riot/repo/tests/gnrc_rpl_srh/tests/01-run.py", line 353, in run
    raise e
  File "/home/kbee/dev/riot/repo/tests/gnrc_rpl_srh/tests/01-run.py", line 349, in run
    func(child, tap, hwaddr_dst, lladdr_dst, lladdr_src)
  File "/home/kbee/dev/riot/repo/tests/gnrc_rpl_srh/tests/01-run.py", line 204, in test_multicast_dst
    assert(len(p) > 0)
AssertionError

also

Traceback (most recent call last):
  File "/home/kbee/dev/riot/repo/tests/gnrc_rpl_srh/tests/01-run.py", line 376, in <module>
    sys.exit(run(testfunc, timeout=1, echo=False))
  File "/home/kbee/dev/riot/repo/dist/pythonlibs/testrunner/__init__.py", line 29, in run
    testfunc(child)
  File "/home/kbee/dev/riot/repo/tests/gnrc_rpl_srh/tests/01-run.py", line 360, in testfunc
    run(test_forward_uncomp)
  File "/home/kbee/dev/riot/repo/tests/gnrc_rpl_srh/tests/01-run.py", line 353, in run
    raise e
  File "/home/kbee/dev/riot/repo/tests/gnrc_rpl_srh/tests/01-run.py", line 349, in run
    func(child, tap, hwaddr_dst, lladdr_dst, lladdr_src)
  File "/home/kbee/dev/riot/repo/tests/gnrc_rpl_srh/tests/01-run.py", line 257, in test_forward_uncomp
    assert(IPv6ExtHdrRouting in p)
AssertionError

also

Traceback (most recent call last):
  File "/home/kbee/dev/riot/repo/tests/gnrc_rpl_srh/tests/01-run.py", line 376, in <module>
    sys.exit(run(testfunc, timeout=1, echo=False))
  File "/home/kbee/dev/riot/repo/dist/pythonlibs/testrunner/__init__.py", line 29, in run
    testfunc(child)
  File "/home/kbee/dev/riot/repo/tests/gnrc_rpl_srh/tests/01-run.py", line 361, in testfunc
    run(test_forward_uncomp_not_first_ext_hdr)
  File "/home/kbee/dev/riot/repo/tests/gnrc_rpl_srh/tests/01-run.py", line 353, in run
    raise e
  File "/home/kbee/dev/riot/repo/tests/gnrc_rpl_srh/tests/01-run.py", line 349, in run
    func(child, tap, hwaddr_dst, lladdr_dst, lladdr_src)
  File "/home/kbee/dev/riot/repo/tests/gnrc_rpl_srh/tests/01-run.py", line 285, in test_forward_uncomp_not_first_ext_hdr
    assert(IPv6ExtHdrRouting in p)

Versions

Test host is Ubuntu 19.04.

[miri64]

Will have a look ASAP. Maybe merging fragmentation broke something in the handling of other extension headers. But need to bisect (and thus a machine with a usable git interface ;)) For that.

[miri64]

I ran

git bisect good 2019.10-devel
git bisect bad 2019.10-RC1
git bisect run ./test_i12436.sh

with ./test_i12436.sh being

#!/bin/sh
make -C tests/gnrc_rpl_srh -j8 || exit 125
sudo timeout 10 make -C tests/gnrc_rpl_srh test
exit $?

(the timeout was added since the script in some cases did not finish, since the thread in does not... need to fix that..)

[miri64]

Interesting... d13d49d is reported to be the first bad commit. Will investigate if that is really the case, as it seems unrelated.

[miri64]

And yes, it is specifically this merge commit. Both d13d49d^1 and d13d49d^2 succeed reliably oO.

[miri64]

Note, I only was able to reproduce the first case so far, not the others.

After d13d49d the packet sent here

RIOT/tests/gnrc_rpl_srh/tests/01-run.py

Lines 199 to 201 in d13d49d

	sendp(Ether(dst=hw_dst) / IPv6(dst="ff02::1", src=ll_src) /
	IPv6ExtHdrRouting(type=3, segleft=1, addresses=["abcd::1"]),
	iface=iface, verbose=0)

somehow never reaches the native instance's IPv6 layer. Interestingly enough, if I enable all the debug in gnrc_ipv6.c (not just the two ipv6: Received (src = ... ones) I get a success in d13d49d, but not in 2019.10-RC1...

[miri64]

Ok now it succeeds for d13d49d also without modifications oO.

Given your different assertion errors and my observations so far, I believe this is some kind of timing issue :-/.

[miri64]

(I've seen the others occur now as well)

[miri64]

485dbd1 (#12175) caused this. While the fix is sound in the regard that it checks if src is multicast, but I'm pretty sure dst can be a multicast address, at least for non-UDP packets. I'll research a bit, before I provide a fix.

[miri64]

The other issue (both instances of assert(IPv6ExtHdrRouting in p)) is caused by the fix provided in #11970... For normal forwarding the "fill header" function is not called, for source routing however, the packet is passed through there. Since this forwarding however, the source address is of course not assigned to the interface, so the "fill header" function fails.

[miri64]

See #12440 for a first fix for this test (there are more needed, but have a different topic).

[miri64]

See #12442 for the second and last part.

[cgundogan]

See #12440 for a first fix for this test (there are more needed, but have a different topic).

See #12442 for the second and last part.

Just to keep a note on that: both (merged) issues do not resolve the assertion problems, the test itself (without scapy) runs successful, though.

[miri64]

Or to state it more positively: previous level of stability was restored ;)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want me to ignore this issue, please mark it with the "State: don't stale" label. Thank you for your contributions.

[miri64]

@leandrolanzieri did you encounter this?

[miri64]

Ah, this should have been fixed by #13836! Re-open, if I am wrong with this.