Improve error handling in e2e tests and get container's logs in case of failure

[cmainas]

If a test from the e2e tests fail, we should provide a more meaningful error message of what failed in the test. Moreover, it will be helpful when a fail tests to provide the output of the container.

[IrvingMg]

Hi, I'd like to take this issue.

[IrvingMg]

@cmainas I've been looking into options to improve error handling. The simplest approach would be to add a helper function to capture logs on failure and improve the error messages. This would solve the immediate problem, but given the broader goal of making E2E tests more robust (#200), I was wondering if you'd be open to adding a testing framework for E2E tests, such as Ginkgo.

I think Ginkgo could be a good fit since it's specifically designed for integration and E2E testing, and is widely used by projects like Kubernetes.

If you agree, I can work on it. I can either create an umbrella issue or use #200 to refactor the entire E2E package and tackle it incrementally (for example, by files or test functions). What do you think?

[cmainas]

Hello @IrvingMg thank you for taking time to solve this correctly.

I personally have no problem at all to use a new framework for testing. My only concern is that I think none of the current maintainers are familiar with the Ginkgo framework and thus it might be hard to maintain the end-to-end testing code.

One the other hand, I spent some time to read some Ginkgo's documentation, examples and actual use cases. Indeed it sounds a good fit for some urunc's testing purposes, but I think it would be helpful if you could share some more details of your thoughts regarding the refactor. Would it be possible (if you have time) to provide a sketch of how you think the integration of Ginkgo with the end-to-end testing? It would also be nice to have a small example (just some pseudo code) of how a end-to-end test code would look like.

Furthermore, the overall objectives of the end-to-end testing code are:

• Allow the easy expression of new tests.
• Be extensible regarding container tools, making it easy to add code that handles/interacts with container tools. For example, some kubernetes testing could also be done from the current code, if we simply define the pod deployment as a new container tool (e.g. calling kubectl).
• Be extensible regarding testing functions, making it easy to define new functions to test a specific feature (e.g. network, correct user/namespaces/environment setup etc.)

Therefore, it would also be very interesting if there are any ideas which will help us achieve the above objectives easier with Ginkgo.

[IrvingMg]

Sure, no problem! I’ll prepare a proposal with more details and open an issue so we can continue the discussion there.

[cmainas]

Great, you can also join the community meeting next week (Wednesday 21st) and add it as a discussion item if you want.

[IrvingMg]

I’ve opened issue #381 with the proposal and added it to the community meeting agenda. Happy to discuss it there!