Add integration tests #598

qleisan · 2021-04-28T15:12:17Z

Add initial pytest based integration tests

rettichschnidi · 2021-04-29T09:33:59Z

@qleisan Would you like some feedback already now? If not, please let us know once you do.

qleisan · 2021-04-30T07:17:23Z

Yes I do!
Some ongoing improvements
I will refactor the helper/fixture code to have a base class and use inheritance (cleaner). Also I'm working on replacing the temporary "setformat" server CLI command with a command line argument for the client.
e.g. lwm2mclient -f 11543 => the client registers with URI=...;ct=11543 and cause the server to use JSON when communicating with this client. However I just noticed some problems with this approach, registration doesn't work with all formats I want to use in testcases (investigating)

qleisan · 2021-04-30T14:07:03Z

Pushed "object oriented" refactoring of test helpers, thx to @mlasch for the suggestion. Odd bug (on my machine) one test executes twice (and fails second time)

rettichschnidi

My feedback in general, not specific to a certain file/line of code:

Please use a linter (pylint being the obvious choice)
Please format the code according to PEP8
Instead of using the example projects, introduce new executable which can be specifically tailored to the testing needs.
If any TODOs remain in the code, please explain them well

examples/server/lwm2mserver.c

examples/client/CMakeLists.txt

tests/integration/test_bootstrap_interface.py

tests/integration/conftest.py

tests/integration/test_device_mgmnt_interface.py

tests/integration/test_registration_interface.py

qleisan · 2021-05-03T09:18:37Z

Thanks for the feedback. I'll fix formatting etc. before I push a non-draft version.

Instead of using the example projects, introduce new executable which can be specifically tailored to the testing needs.

Interesting thought but do we really want this? Any other feedback on the approach to integration testing (anyone) - any conceptual improvements now might save a lot of test development time

rettichschnidi · 2021-05-03T10:00:52Z

Interesting thought but do we really want this?

I think it will become increasingly hard have a single target serving two purposes:

Beeing a simple, easy to understand example
Supporting everything needed to cover many (all) corner cases for integration testing.

I am pretty sure separate binaries are the way to go.

Any other feedback on the approach to integration testing (anyone) - any conceptual improvements now might save a lot of test development time

I like the direction (Tests in pytest instead of C, invoking executables, using test cases from the standard).

qleisan · 2021-05-04T08:06:18Z

I think it will become increasingly hard have a single target serving two purposes:

Beeing a simple, easy to understand example

Supporting everything needed to cover many (all) corner cases for integration testing.

Today we have a separation in client and lightclient. I see your point but not sure that introducing a testclient (and testserver, testbootstrapsever) is the best way to go, at least not now. Shall we revisit this once it is more clear what modifications are actually needed in order to support good testability?

rettichschnidi · 2021-05-04T19:45:29Z

I see your point but not sure that introducing a testclient (and testserver, testbootstrapsever)

I'd aim for one (configurable) client and one (configurable) server target, not many.

One then can build them multiple times, each time with a different set of supported features and run the relevant pytest tests against it.

... is the best way to go, at least not now. Shall we revisit this once it is more clear what modifications are actually needed in order to support good testability?

Feel free to start off with the examples, but when looking at the "Possible improvements" from back then I have the feeling that we will very soon reach a state when dedicated targets, optimized for testing, will be needed.

sbertin-telular · 2021-05-06T16:19:37Z

I'm glad to see this type of automated testing. Much easier and more reliable to see if something got broken than manual testing.

qleisan · 2021-05-07T11:48:38Z

Pushed a slim down version that can be merged one default block size is set to 1024 (done in PR#596)
Feedback on how to make the HelperBase class more pythonic/PEP8 welcome.

qleisan · 2021-05-10T17:38:49Z

I'll add CI integration and resolve comments that are still valid (some are outdated)

tuve · 2021-05-11T08:29:53Z

I haven't had time to dive into all the details but i really like this approach with integration tests in python, good work!

tuve

I have no issues beyond what has already been commented by @rettichschnidi

.github/workflows/build_and_test.yaml

.pylintrc

tests/integration/conftest.py

.pylintrc

qleisan · 2021-05-17T12:10:50Z

@rettichschnidi - please review

qleisan · 2021-05-17T14:19:07Z

Fixed gitlint issue

tests/integration/conftest.py

.github/workflows/build_and_test.yaml

tests/integration/test_device_mgmnt_interface.py

mlasch · 2021-05-20T15:28:57Z

The log-files created during the test run should be in the .gitignore file. Maybe better even a pattern like *_log.txt. Or place them in a separate runlogs folder which is then ignored.
All in all this PR looks very promising. I'm looking forward to have these automated tests included.

outdated

qleisan · 2021-05-21T10:12:12Z

@mlasch pushed update with fixes for review comments (thanks!). Tests running

rettichschnidi

A few things for now, will do a better review (actually looking at the python code) once I can find some more time.

Surely looking much better already!

.github/workflows/pylint.yaml

.github/workflows/build_and_test.yaml

.github/workflows/pylint.yaml

tests/integration/pytest.ini

README.md

.pylintrc

tests/integration/.gitignore

rettichschnidi

I just can not find the time to review the python code itself. Nevertheless, I am willing to approve after this round.

.pylintrc

README.md

File generated with 'pylint --generate-rcfile > .pylintrc' pylint --version => pylint 2.6.2 astroid 2.4.2 Python 3.8.5 (default, Jan 27 2021, 15:41:15) [GCC 9.3.0]

Add initial pytest based integration tests

qleisan force-pushed the integrationtests branch from 3838765 to 90336c2 Compare April 28, 2021 15:19

mlasch mentioned this pull request Apr 29, 2021

🛣 Roadmap 🎯 #597

Open

qleisan changed the title ~~Add integration tests NB! THIS IS A DRAFT~~ Add integration tests Apr 30, 2021

rettichschnidi previously requested changes May 2, 2021

View reviewed changes

qleisan force-pushed the integrationtests branch from 03972a0 to 818d0e4 Compare May 4, 2021 16:01

qleisan force-pushed the integrationtests branch from 818d0e4 to 8cdbf04 Compare May 7, 2021 11:41

qleisan changed the title ~~Add integration tests~~ Add integration tests (requires PR#596) May 7, 2021

This was referenced May 7, 2021

Add integration tests int-201, int-204 (requires PR#598, PR#602) #604

Closed

Add integration test int-104 (requires PR#598 and PR#561) #603

Closed

qleisan marked this pull request as ready for review May 10, 2021 07:42

tuve self-requested a review May 11, 2021 08:47

tuve reviewed May 11, 2021

View reviewed changes

qleisan force-pushed the integrationtests branch from 8cdbf04 to d9358a8 Compare May 11, 2021 13:13

qleisan changed the title ~~Add integration tests (requires PR#596)~~ Add integration tests May 11, 2021

qleisan force-pushed the integrationtests branch 2 times, most recently from 5a4f7dd to a24e669 Compare May 17, 2021 08:05

qleisan commented May 17, 2021

View reviewed changes

.github/workflows/build_and_test.yaml Outdated Show resolved Hide resolved

qleisan commented May 17, 2021

View reviewed changes

.github/workflows/build_and_test.yaml Outdated Show resolved Hide resolved

qleisan force-pushed the integrationtests branch from a24e669 to 0a9e9ed Compare May 17, 2021 08:16

qleisan commented May 17, 2021

View reviewed changes

.pylintrc Outdated Show resolved Hide resolved

qleisan force-pushed the integrationtests branch from 0a9e9ed to 92e2af1 Compare May 17, 2021 11:59

qleisan commented May 17, 2021

View reviewed changes

tests/integration/conftest.py Outdated Show resolved Hide resolved

.pylintrc Outdated Show resolved Hide resolved

qleisan force-pushed the integrationtests branch from 92e2af1 to efdd31d Compare May 17, 2021 14:17

mlasch self-requested a review May 20, 2021 14:07

qleisan requested a review from rettichschnidi May 20, 2021 14:11

mlasch requested changes May 20, 2021

View reviewed changes

tests/integration/conftest.py Outdated Show resolved Hide resolved

.github/workflows/build_and_test.yaml Outdated Show resolved Hide resolved

tests/integration/test_device_mgmnt_interface.py Outdated Show resolved Hide resolved

qleisan force-pushed the integrationtests branch from efdd31d to 9891dc4 Compare May 21, 2021 10:07

qleisan requested review from mlasch and removed request for rettichschnidi May 21, 2021 16:34

rettichschnidi requested changes May 25, 2021

View reviewed changes

qleisan force-pushed the integrationtests branch from 9891dc4 to a99d60a Compare May 25, 2021 09:42

qleisan requested a review from rettichschnidi May 25, 2021 09:43

rettichschnidi requested changes May 25, 2021

View reviewed changes

.pylintrc Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

Leif Sandstrom added 2 commits May 25, 2021 13:56

Add generated .pylintrc file

9326b6b

File generated with 'pylint --generate-rcfile > .pylintrc' pylint --version => pylint 2.6.2 astroid 2.4.2 Python 3.8.5 (default, Jan 27 2021, 15:41:15) [GCC 9.3.0]

Add integration tests

fb266e4

Add initial pytest based integration tests

qleisan force-pushed the integrationtests branch from a99d60a to fb266e4 Compare May 25, 2021 12:08

mlasch approved these changes May 25, 2021

View reviewed changes

rettichschnidi approved these changes May 25, 2021

View reviewed changes

qleisan merged commit a1281ab into eclipse-wakaama:master May 25, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add integration tests #598

Add integration tests #598

qleisan commented Apr 28, 2021

rettichschnidi commented Apr 29, 2021

qleisan commented Apr 30, 2021

qleisan commented Apr 30, 2021

rettichschnidi left a comment

qleisan commented May 3, 2021

rettichschnidi commented May 3, 2021

qleisan commented May 4, 2021

rettichschnidi commented May 4, 2021

sbertin-telular commented May 6, 2021

qleisan commented May 7, 2021

qleisan commented May 10, 2021

tuve commented May 11, 2021

tuve left a comment

qleisan commented May 17, 2021

qleisan commented May 17, 2021

mlasch commented May 20, 2021

qleisan commented May 21, 2021

rettichschnidi left a comment

rettichschnidi left a comment

Add integration tests #598

Add integration tests #598

Conversation

qleisan commented Apr 28, 2021

rettichschnidi commented Apr 29, 2021

qleisan commented Apr 30, 2021

qleisan commented Apr 30, 2021

rettichschnidi left a comment

Choose a reason for hiding this comment

qleisan commented May 3, 2021

rettichschnidi commented May 3, 2021

qleisan commented May 4, 2021

rettichschnidi commented May 4, 2021

sbertin-telular commented May 6, 2021

qleisan commented May 7, 2021

qleisan commented May 10, 2021

tuve commented May 11, 2021

tuve left a comment

Choose a reason for hiding this comment

qleisan commented May 17, 2021

qleisan commented May 17, 2021

mlasch commented May 20, 2021

qleisan commented May 21, 2021

rettichschnidi left a comment

Choose a reason for hiding this comment

rettichschnidi left a comment

Choose a reason for hiding this comment