Move integration tests towards schutzbot/run_tests.sh

[croissanne]

Instead of calling image-builder-tests, let's remove that binary, and just start the container and use curl and jq to tests the endpoints.

[yuxisun1217]

Hi @Gundersanne ,
Greetings! Sorry for late since I just finished the Spring Festival holiday.
I made a very simple example about using pytest to run the integration tests: https://github.com/yuxisun1217/image-builder/blob/pytest/tests/tests/test_image_builder.py

( Start image-builder container)
# pip install pytest
# pytest -v tests/tests/test_image_builder.py --html tests/report.html --self-contained-html

It can output simple test results and generate a pretty html report(http://10.0.155.207/reports/report.html). And it's lite. Could you please check if it meets your expectation to replace the image-builder-tests binary package? Thanks!

[croissanne]

Nice! Thank you for that, always good to have proof of concepts.

That said I think the ideal way to structure the integration tests is to follow what osbuild-composer does as much as possible. Obviously there would a lot fewer tests in image-builder , but since the api.sh tests in composer are testing the same thing as image-builder (that is an HTTP api), I think it makes a lot of sense to aim for a similar structure across osbuild projects.

For this a few steps are necessary though, we'd need to update the spec file to generate a tests subpackage and move the api.sh to /test/. Update the jenkinsfile to have a similar run_project_tests function, and obviously rewrite it with the same tools as in composer.

[yuxisun1217]

Hi @Gundersanne ,

I read the code in api.sh. Removing composer part, it seems that it only have 3 basic validation check points about image-builder:

• get https://localhost/api/composer/v1/version
• get https://localhost/api/composer/v1/openapi.json
• post https://localhost/api/composer/v1/compose

And it only verify these "curl"s can get ok status, but not verify the contents. As you mentioned in #120 we may need more corner cases to make it more like a interface test(positive/negative test, and perhaps secure test). Here is a small group of cases I think we can test image-builder:

• verify get/post to each interface(e.g. version/openapi.json/distributions...) return status and content
• verify other not allowed headers requests can be blocked
• verify get/post something not expected are not allowed
• osbuild url is http
• ...

In such case, IMO language is not important, but having a test framework will be more convenience for running more test cases. (For bash, I also find a framework bats but I didn't use it before.).

[yuxisun1217]

Hi @Gundersanne ,

I've add 3 simple demos of the integration tests:

1. https://github.com/yuxisun1217/image-builder/tree/yuxisun_test/test/api.sh
   This is the simplify of osbuild-composer api.sh, and the output is very long. Besides, if one case failed, the who script will fail.

2. https://github.com/yuxisun1217/image-builder/tree/yuxisun_test/test/api_bats.sh

# dnf install bats
# bats test.sh
 ✓ Verify version 
 ✓ Verify openapi.json 

3. https://github.com/yuxisun1217/image-builder/tree/yuxisun_test/test/test_image_builder.py

# pytest -v test_image_builder.py
============================================================================================================ test session starts =============================================================================================================
platform linux -- Python 3.9.0, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.9.0', 'Platform': 'Linux-5.8.15-301.fc33.x86_64-x86_64-with-glibc2.32', 'Packages': {'pytest': '6.2.2', 'py': '1.10.0', 'pluggy': '0.13.1'}, 'Plugins': {'cov': '2.11.1', 'metadata': '1.11.0', 'html': '3.1.1'}, 'WORKSPACE': '.'}
rootdir: /root/repos/image-builder/test
plugins: cov-2.11.1, metadata-1.11.0, html-3.1.1
collected 1 item                                                                                                                                                                                                                             

test_image_builder.py::TestImageBuilder::test_version PASSED                                                                                                                                                                           [100%]

============================================================================================================= 1 passed in 0.16s ==============================================================================================================

Having a framework will be more convenient to run multiple test cases. But for the consistency, it's also OK to use the api.sh. Which one do you prefer? I'd like to add more cases after confirming the test framework. Thanks!

[croissanne]

Hey, Thank you!

So I like option number 2, this would give us tap output, but lets us keep using tools that we're expecting other people to use when calling the api directly. Furthermore keeps the differences between composer and image-builder small enough I'd say.

I still like option number 1 as well just because it's what composer does.

I think I'd be fine with either 1 or 2 honestly.

Just to note: to have a similar structure to composer this will require some spec file work too but I can do that afterwards, we can just invoke those tests from schutzbot/run_tests.sh for now so I don't think this would be blocked on anything.

[yuxisun1217]

Hi,

Is there any further work needed for this issue? Thanks!

[croissanne]

Hey, I'd say this is done! They're now invoked in the same way as in osbuild-composer. Thank you!