CircleCI config overhaul, instrumentation tests & screenshot automation (EXPOSUREAPP-4480,DEV)

[d4rken]

@mtwalli, @AlexanderAlferov and me overhauled the CircleCI config, added instrumentation testing and automatic screenshots.

Highlights:

• Refactored the config (yes you can refactor YAML 😵) to make use more use of re-usable functions
• Use larger instances for faster test execution
• Adjusted gradle/JVM options to improve performance and use larger instances better
• Run instrumentation tests via macOS CircleCI executor
• Run automated screenshot routine via fastlane when a new release candidate is build
• Improved caching behavior
• Split works for better job restart and improved dependencies between tasks
• Renamed tasks and work flows to be easier to understand

We tried Firebase Testlab, but it was not working out, due to lack of control over the whole process (more of a black box) and inability to use fastlane with it. There were also issues with our mocking library which could fail on newer API emulators if not run on x86_64 architecture emulators, and firebase testlab doesn't allow us to enforce it.

Because we previously didn't run instrumentation tests via CI, a few tests fell into disarray and will need fixing. The current config setup is already a huge improvement over the previous config and I think should merge it despite the failing instrumentation tests. Until we enforce the check to pass for branches, there should be no issue. A follow up PR can then be made stress free to fix the failing tests.

This was initially targeted for 1.12, but as this will require adjusting the branch restrictions, it would make sense to merge this into 1.11, we share the same branch restrictions between all branches. Then the only problematic branch would be 1.10.x which will soon be superseeded by 1.11.x.

Once the config is merged I will change the restrictions and all open PRs will have to pull the new config to run the correct tests.

[d4rken]

Example screenshot execution from another run:

• https://app.circleci.com/pipelines/github/corona-warn-app/cwa-app-android/8528/workflows/88830ec8-83e5-494d-a03f-d7997e138065/jobs/43049
• https://43049-268027139-gh.circle-artifacts.com/0/screenshots/screenshots.html

[d4rken]

• The screenshot job takes quite long ~50-100 minutes, only executed on releases though.
• Instrumentation test take around 17min (once cached). Normal tests are around 10 min. So this adds 7 min to our test pipeline time, but I think that is just the price to pay and will be well worth it. Maybe someone has a good idea on how we can further speed this up? The alternative would be to either live with it, or make the test result non-blocking... 🤔

[kaddaSz]

Could you also Zip the ScreenShots? So I could just donwload the zip. The idea is, to add this zip later also to the release notes.
The same would make sence to the test results I guess.

[d4rken]

This would be merged after we have finished the statistics stuff so there is no delay due to pipeline issues.

[d4rken]

Could you also Zip the ScreenShots? So I could just donwload the zip. The idea is, to add this zip later also to the release notes.

I think so. Will check. We could also link the artifacts on CircleCI directly, though the link will probably break after a few weeks? CircleCI artifact storage times out at some time.

The same would make sense to the test results I guess.

What do you mean by that?

[kaddaSz]

The same would make sense to the test results I guess.

What do you mean by that?

https://app.circleci.com/pipelines/github/corona-warn-app/cwa-app-android/8541/workflows/2a8cd323-f037-4369-818a-eaf96ad45373/jobs/43183/artifacts

There are also a lot of artifacts, which you maybe could zip? I could use these results for our internal reporting, but it would be easier to just download a zip.

[d4rken]

Screenshots, Instrumentation tests (release), unit tests (release) are now uploaded as .zip too.

[AlexanderAlferov]

LGTM

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information