Integration of pytest

[polybassa]

This is just a PoC. A *.uts file will be transformed to a pytest like file. The exported file usually needs manual modifications, but this exporter can be a first step towards pytest

[codecov]

Codecov Report

Merging #2550 (0dc7496) into master (2f07f21) will increase coverage by 2.91%.
The diff coverage is n/a.

❗ Current head 0dc7496 differs from pull request most recent head 8e24fdf. Consider uploading reports for the commit 8e24fdf to get more accurate results

@@            Coverage Diff             @@
##           master    #2550      +/-   ##
==========================================
+ Coverage   84.57%   87.49%   +2.91%     
==========================================
  Files         296      267      -29     
  Lines       62137    55824    -6313     
==========================================
- Hits        52555    48845    -3710     
+ Misses       9582     6979    -2603     

Impacted Files	Coverage Δ
scapy/contrib/automotive/gm/gmlan_logging.py	73.33% <0.00%> (-26.67%)	⬇️
scapy/layers/l2tp.py	88.88% <0.00%> (-11.12%)	⬇️
scapy/themes.py	80.98% <0.00%> (-10.19%)	⬇️
scapy/modules/p0f.py	64.72% <0.00%> (-9.93%)	⬇️
scapy/contrib/automotive/scanner/test_case.py	76.00% <0.00%> (-8.89%)	⬇️
scapy/layers/tls/crypto/cipher_aead.py	81.30% <0.00%> (-6.53%)	⬇️
scapy/layers/tls/crypto/groups.py	87.14% <0.00%> (-5.82%)	⬇️
scapy/autorun.py	78.23% <0.00%> (-5.21%)	⬇️
scapy/contrib/geneve.py	92.10% <0.00%> (-5.20%)	⬇️
scapy/layers/tls/session.py	83.78% <0.00%> (-4.10%)	⬇️
... and 229 more

[gpotter2]

Please don't merge this as is.
This is a great script but I'd like to see how well it works (a fully converted file) before doing anything.

I'm still unsure whether it's best to move to pytest because of how massive the work will probably end up being..

[polybassa]

This is absolutely not ready for merge. Sorry for not mentioning this. The produced files will all need manual rework. This is just a starting point. The use of pytest needs to be evaluated for the tests, in my opinion. -- Viele Grüße Nils Weiß

…

On March 26, 2020 5:21:07 PM GMT+01:00, Gabriel ***@***.***> wrote: Please don't merge this as is. This is a great script but I'd like to see how well it works (a fully converted file) before doing anything. I'm still unsure whether it's best to move to pytest because of how massive the work will probably end up being.. -- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: #2550 (comment)

[polybassa]

I've added pytest to tox (just for PoC, not added everywhere). What about supporting pytest and UTscapy in parallel?

[polybassa]

@gpotter2 @guedou @p-l-
I would like to start a discussion. I've made a small PoC to enable pytest for the unit tests.
I respect the individual configurations by parsing for the argument "-K"
A simple dump function in UTscapy allows to transfer a *.uts file to a pytest file. Manual fixing is required here, nevertheless this script should save some time if one wants to port tests.
I've tested codecov if the combination of tests work corretly, if pytest and uts files for the same unit differ.

Would it make sense to open the unit test system to allow contributors to write tests in *.uts or pytest?

[guedou]

This is a really cool PR. It its definitely worth finishing it and merging. If @gpotter2 and @p-l- are OK, I think that we should move to pytest and leave UTScapy behind us. After merging this PR, the next step will be to mandate that new unit tests are written with pytest while we migrate all tests.

Did you try concurrent testing? I will be interested to know if moving to pytest will also make the testing time shorter.

[polybassa]

Thanks for your feedback. I will continue my work on this PR. I will probably start to port regression.uts to pytest, to see if this concept holds.
I didn't tested concurrent execution, yet. Also my tests on CAN will probably fail if they were executed concurrently. But I will test this, after I ported some tests from regression.uts.

[gpotter2]

I agree with @guedou that this is very cool, and it would be great to be able to migrate to pytest, but we must make sure everything works before doing that. Thé UTscapy file is filled with tons of extra code that would need to be integrated somehow to our test process.

[polybassa]

I totally agree, the UTscapy file is pretty much filled with stuff. I did a rough analysis of the current functionality.. in my opinion, some function might not be useful, some functions will be available in pytest. Most important is the tag system, which I could "somehow" port to pytest.
But I will continue on that PoC, than we can see, where leads to.

[guedou]

Before converting moving to pytest, I think that it will make sense to split regression.uts into smaller files that respect scapy/ naming convention i.e. test/config.uts test/layers/inet6.uts test/packet.uts ...

First, it will make your life easier when adding or looking for a unit test, then we will be able to convert to pytest file by file with several simple PR instead of a huge one.

[p-l-]

That is really great. Thanks a lot for this work.

I agree with @guedou that we may want so split regression.uts.

[polybassa]

Hi, I've updated this PR and would like to open it for your review.

I made some first ports to pytest. I used the Co-Author tag to preserve the original authors (git blame filename --porcelain | grep "^author " | sort | uniq was used to identify the original authors).

I would suggest the enforcement of flake8 for unit tests. The possibility of pytest would help us to have much cleaner unit tests, in my opinion.

[polybassa]

And another question. Should there be a copyright notice on unit tests?

[polybassa]

Thanks for your answer. I can understand all your points. In terms of performance you are probably right. The main advantages I see in pytest compared to UTScapy are:

• Tests are python scripts, which means,
  • blank lines and indentations are supported
  • typing can be applied
  • flake8 can be used on tests
  • codespell can be used on tests
• Most IDEs have support of pytest, which allows for extensive refactoring features

In my opinion, these additions to the test system will lead us to cleaner and more maintainable tests in the future.

Furthermore, pytest comes with a whole bunch of extensions.

A main problem, that you also mentioned, is the transition from UTScapy to pytest. If there is no way for you to have both systems coexisting for a certain period, for example one release cycle, I totally agree, that a huge effort is required to make a fast switch. Which is almost not affordable for a open source project.

To summarize my opinions, if coexistence of two test systems for a certain time is a no go, we can stop here with pytest.
If not, I would be happy if we could make a plan for a smooth transition over a fixed period, for example the next release cycle.

[polybassa]

closed, because its broken

[gpotter2]

Honnestly I think it's a decent idea, and maybe one day we'll be happy to have it as a basis.
It's just too much work for now, but who knows 😛

[evverx]

The main advantages I see in pytest compared to UTScapy

I'd add that pytest should make it much easier to catch resource warnings like #3962 (comment) as well as other warnings produced by python3 -Xdev -Werror. It took me about a week to figure out where that particular warning went :-)

UTScapy and pytest should not coexist

I think it would be easier to switch to pytest if it was possible to convert the tests one by one.

[polybassa]

Hello together, I would like to come back to this issue. I'm wondering, if we could start supporting pytest, in order to have to opportunity to start a transition from UTScapy to pytest.

[polybassa]

I've made a few tests, apparently chatGPT is really useful to translate unit tests from UTScapy to pytest. This would allow a transition from UTScapy to pytest with a fraction of the effort, compared to doing everything manually.

[guedou]

I still believe that it is a good idea to move to pytest. However, I am scared by the amount of work and review time as well as the fear that we will stop the migration before it is fully completed.We could start by splitting regression.uts into smaller pieces and use this opportunity to store uts files under the same architecture as scapy/Then, we could start the migration step by step by submitting small PR.What do you think?Sent from my iPhoneOn 6 Jun 2023, at 13:04, Nils Weiss ***@***.***> wrote: I've made a few tests, apparently chatGPT is really useful to translate unit tests from UTScapy to pytest. This would allow a transition from UTScapy to pytest with a fraction of the effort, compared to doing everything manually. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[KhazAkar]

For me @guedou your idea makes sense. I will someday sit, reading tests/ folder contents, search for UTScapy specific behavior and start porting to pytest, folder by folder. Will take time, but I will step up to it :)