Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework the test suite #56

Open
5 of 7 tasks
krizhanovsky opened this issue Nov 15, 2018 · 0 comments
Open
5 of 7 tasks

Rework the test suite #56

krizhanovsky opened this issue Nov 15, 2018 · 0 comments
Labels
good to start Start form this tasks if you're new in the framework Infrastructure The framework reworking and extensions
Milestone

Comments

@krizhanovsky
Copy link
Contributor

krizhanovsky commented Nov 15, 2018

There are several architectural issues in the current test suite (the list is going to be updated):

  • There are 2 versions of the framework: the whole test suite is the old one, while /framework is the new one. Need to finish moving to the new framework.
  • The test framework must use HTTP/2 client connections by default. HTTP/2 has several drawbacks, performance and observability, so it actually doesn't replace HTTP/1 in all the cases (e.g. server management) and we should not throw HTTP/1 out.
  • /framework and /helpers separation is unclear: helpers are expected to contain generic classes, e.g. dmesg or analyzer, useful in different tests directly, while deproxy.Client seems purely dedicated to the framework itself and it's strange to see deproxy.py in /helpers while all the related logic is in /framework.
  • Master branch of tests is not compatible with older releases. There are two sources of the problem: new features and changes in the configuration file format. It'd be better to have one test code base for all the versions. New features can be disabled in tests_disabled.json with a special version field, e.g. "version" : "0.7" means that the test is disabled for Tempesta version below 0.7. In the new framework, the full Tempesta configuration is listed in the test description file, not compiled from parts, so we can give very specific configuration for exact Tempesta version. Some good and generic solution is required. (Duplicate of Tests for stable versions #55)
  • dmesg is analysed after every test case to search for possible oopses and warnings. Current way of analysis (dmesg | tac | grep -m 1 "Start test" -B %i | tac) is not effective for long running tests, so dmesg will be called once again. Instead we can simply use journalctl -k --since=${date} to show all required content with one call. More over, journalctl can be configured to use really big buffers, and full system log will be available at any time.
  • Move to Python3 since Python 2 in supported only until 2020. The only external module which prevented us from migration to the 3rd version was Scapy-TLS, which has been removed in Moving away from Scapy-ssl_tls and fix tls/ tests #179 . 2to3 can be used for the migration. https://github.com/horlabs/scapy-ssl_tls seems made some iteration to port the library to Python 3. Please update https://github.com/tempesta-tech/tempesta-test/blob/master/README.md when this point is done.
  • The last commit in Scapy-TLS was in Sptember 2018, so we copied its file to the test suite . It seems the base Scapy supports TLS. Scapy-TLS also has bugs like UserWarning: Verification of GCM tag failed: MAC check failed tintinweb/scapy-ssl_tls#163 , which affect us, e.g. in Verification of GCM tag failed: MAC check failed tempesta#1310 .

The both the current test frameworks use HTTP/1.1 in plain test mode, w/o TLS, while the most real life traffic is at least encrypted and HTTP/2 grows. It means that out test suite tests the most crucial components in only ad hoc tests. That doesn't make much sense, so I mark the issue as crucial.

@krizhanovsky krizhanovsky added enhancement New feature or request question Further information is requested labels Nov 15, 2018
@krizhanovsky krizhanovsky added this to the 0.2 Beta milestone Nov 15, 2018
@krizhanovsky krizhanovsky added crucial Infrastructure The framework reworking and extensions labels Nov 25, 2018
@krizhanovsky krizhanovsky removed this from the 0.2 Beta milestone Feb 10, 2019
krizhanovsky added a commit that referenced this issue Jun 24, 2019
Fix ugly assertion 'False is not true'.
Print warnings on higest verbose level about deprecated classes (#56).
@krizhanovsky krizhanovsky removed crucial question Further information is requested labels Jul 25, 2019
@krizhanovsky krizhanovsky added this to the Beta milestone Aug 10, 2019
@krizhanovsky krizhanovsky mentioned this issue Jul 23, 2020
33 tasks
krizhanovsky added a commit that referenced this issue Dec 1, 2020
Essencial parts of the library are moved into tls/ directory and
are supposed for further truncation, fixes and migration to Python 3.
krizhanovsky added a commit that referenced this issue Dec 11, 2020
Essencial parts of the library are moved into tls/ directory and
are supposed for further truncation, fixes and migration to Python 3.
@krizhanovsky krizhanovsky modified the milestones: Beta, Tempesta 0.7 Feb 1, 2022
@krizhanovsky krizhanovsky removed enhancement New feature or request crucial labels Apr 4, 2022
@krizhanovsky krizhanovsky added the good to start Start form this tasks if you're new in the framework label Aug 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good to start Start form this tasks if you're new in the framework Infrastructure The framework reworking and extensions
Projects
None yet
Development

No branches or pull requests

5 participants