Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

E2E - Isolate native tests into a dedicated workflow and put each test with a separate job in the workflow #3777

Closed
tadayosi opened this issue Oct 26, 2022 · 3 comments
Labels
area/continuous integration Related to CI and automated testing area/test

Comments

@tadayosi
Copy link
Member

tadayosi commented Oct 26, 2022

#3774 (comment)

I think that kind of timeout starts becoming a problem. 2 hours seems to be too long. At least we should try to have the native tests starting before everything else, so, the rest of checks will progress in parallel to this long running one.

With my last commit the native tests native-it is already isolated from the rest of install-it, so they should run in parallel from the start and should not interfere with each other.

Note that 2 hours is the maximum cap of timeouts. When successful they shouldn't take the full 2 hours but the actual time they need to finish (right now it's ~1h20m). That said, as I explained at #3715 (comment), this is what it means to run Quarkus native build tests on GitHub-hosted runners.

Right now, each native build test requires ~40 mins for a successful run and we already have 2 test cases, so it should take at least 80 mins. If we want to add more native tests the time will grow linearly. If we are not happy with it, either we should abandon the native tests from CI, or isolate the native tests into a dedicated workflow and put and run each test case with a separate job in the workflow. (The latter might be something we can consider in the future.)

@tadayosi tadayosi added area/test area/continuous integration Related to CI and automated testing labels Oct 26, 2022
@tadayosi
Copy link
Member Author

tadayosi commented Oct 26, 2022

#3774 (comment)

An alternative could be to have a separated check that run on a nightly basis instead (we can even put beside the nightly releases to force us to look to it when failures happen). We will still have a feedback on early failures, but it won't affect that much the time to build at each PR.

@tadayosi
Copy link
Member Author

Wow, the tests have passed but already close to the timeout 60m. It's a good reason to separate the native tests soon.
https://github.com/apache/camel-k/actions/runs/3329377626/jobs/5506518141

📦 github.com/apache/camel-k/e2e/namespace/native
✅ TestNativeBinding (57m18.75s)
✅ TestNativeBinding/kamelet_binding_with_native_build (55m34.35s)
✅ TestNativeBinding/warm_up_before_native_build_testing (1m29.93s)
✅ TestNativeIntegrations (54m30.12s)
✅ TestNativeIntegrations/automatic_rollout_deployment_from_fast-jar_to_native_kit (52m47.04s)
✅ TestNativeIntegrations/unsupported_integration_source_language (860ms)
✅ TestNativeIntegrations/warm_up_before_native_build_testing (1m27.35s)

@tadayosi
Copy link
Member Author

It's already done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/continuous integration Related to CI and automated testing area/test
Projects
None yet
Development

No branches or pull requests

1 participant