Deterministic testing

[ebfull]

There's no real way to use travis to do this, that I know. Solutions for this issue would be very nice as I could move most of our "examples" into tests.

[aldanor]

Any ideas on how this could be done?

[ebfull]

Probably by launching the tests in a docker container or something, and having an external program do some deterministic network stuff. :)

[stappersg]

This update is for making a link to #284.
And a request: Please add the label testing to the "Recommendations on how to mock Rust pcap" issue.

[robs-zeynet]

(Apologies in advance if this is already known; hoping to help and there's low context here)

A question for the group and potentially an offer to help:

If you think of the software stack as four layers: App <--> rust pcap <--> libpcap/npcap <--> OS Kernel + NIC

... which layer(s) are you trying to test? Where do you typically find bugs that you'd like to improve the testing to better squash?

One answer could be just in the main "rust pcap" layer, which would suggest creating a mock for the libpcap layer, and then you could run the code in 'examples' against the mock layer and make sure all of the right libpcap calls were made.

A second answer could be "the whole stack", which would suggest using a TAP interface which is supported easily on Linux and MacOS and with some work on Windows. This creates a virtual NIC that you could attach pcap to with a file description on the other side, and sending packets to the file descriptor triggers Capture::next_packet() and packets send via Capture::sendpacket() should be received on the file descriptor. All of this obviously requires special permissions which may not be available in a Travis container, but might be worked around with qemu or similar. Also, it seems that github's actions allow a custom VM so that might be an easier route to go.

FYI:: other projects have similar problems - travis-ci/travis-ci#1503

Which type of testing were you thinking? If it were the first, I could try to take a pass at mocking out the libpcap layer which is more than exists now.

Thougths?

Also, FYI: the "mockability" of the rust pcap code itself (e.g., "can we test the App itself, not rust pcap") is IMHO completely orthogonal to this issue.

[Wojtek242]

Testing of the rust pcap layer alone with mocks isn't that worthwhile for this package, because that way it does not test if the various calls into pcap are made correctly. Which means full stack is the way to go.

In addition to TAP one could also use veth pairs which is something I at least have some more experience with.

Furthermore, for full stack tests, I think that having them run on Linux only at first would be okay as well. No need to jump head in into full platform support given that no such testing exists at the moment.

[robs-zeynet]

Thanks for the reply - makes sense.

Agreed re: veth pairs, but with veth pairs, you would need something that could raw read and raw write the packets, which is exactly the functionality that one (or at least I :-) ) would use pcap for, so that's why I think the TAP (which is really half a veth pair + a file descriptor) is the right way to go. Otherwise you'll need to replicate all of the pcap functionality without actually using pcap because that's the code under test.

IMHO the difficult thing is really working around trying to create a veth pair or TAP in travis which doesn't let you do that.

I can throw together a quick integration test PR if folks are interested. I would probably use https://docs.rs/tun-tap/latest/tun_tap/struct.Iface.html if folks don't mind the extra dependency. Thoughts?

[robs-zeynet]

FYI: I just threw together #285 as an example of what I was thinking. Let me know what people think.

[Wojtek242]

Testing of the rust pcap layer alone with mocks isn't that worthwhile for this package, because that way it does not test if the various calls into pcap are made correctly. Which means full stack is the way to go.

FWIW, I completely changed my mind about this. Testing full stack with pcap is nice, but ultimately, libpcap itself is not under test. As long as there is confidence that the right calls are being made, unit tests against a mocked raw module are sufficient. I have now done this with #330.

With that said, that is not complete and sufficient for testing as some features (such as capture-stream) currently need a runtime to be properly tested.

[FeldrinH]

FWIW, I completely changed my mind about this. Testing full stack with pcap is nice, but ultimately, libpcap itself is not under test. As long as there is confidence that the right calls are being made, unit tests against a mocked raw module are sufficient. I have now done this with #330.

What about testing safety invariants? The interface between libpcap and this crate is almost exclusively unsafe. I have to imagine there is a sizable number of safety invariants that matter to libpcap but are not checked when using a mocked raw module.

[robs-zeynet]

@FeldrinH : I agree that there's pieces inside the rust-pcap layer that could use testing but honestly my main interest was in mocking out the rust-pcap layer to make testing in the calling app easier. It didn't seem like there was interest in either so I just hacked around it on my own. YMMV.