Goal: Make acceptance-tests run within 10 minutes

[rohitsakala]

I am working on the doc's story and evaluating if principles still stand or not.
One soft principle is to ensure our tests run in 10 minutes, which is not the case.

This would also boost our development.

Accpetance Criteria

• @manno proposed to make gitea optional which might reduce some time.
• Collect ideas or re-evaluate how to we run the acceptance tests.

If not, we might need to update our docs from 10 minutes to 20 or 30 minutes.

[kkaempf]

Do we have an overview how time is currently spent ? Is gitea really slowing tests down ?

[jimmykarily]

The alternative to gitea is to push with --git (we will have it after this is merged: #565). What Mario suggested in a call was to make gitea optional as a component, meaning we won't even install it when running tests. Given Epinio installs in less than 5 minutes and Gitea is a component installed in parallel with the rest, I doubt that skipping the gitea installation will save anything more than 1-2 minutes (at most).

On the other hand, gitea is one of the most resource hungry component we install. Skipping it may allow us to run more ginkgo node on the same hardware.Pulling from a remote git (e.g. github) may introduce other problems though, like network hickups and rate limiting.

[jimmykarily]

I see 2 ways to speed things up:

• Add more workers and distribute test efficiently
• Reduce the time it takes for each test to run

We can try both at the same time for better results. Some ideas:

Add more workers and distribute efficiently

• Split the test suite in more suites (Split test suite #563)
• Simplify and automate runner setup so we can better manage runners. This will
  allow us to easily start runners on any available hardware.

Reduce the time it takes for tests to run

• Use a sample application that deploys faster, in tests. We need to find the
  fastest buildpack out there.
• Use CACHE_IMAGE (or cache volume) (https://github.com/tektoncd/catalog/blob/main/task/buildpacks/0.3/README.md#parameters)
  to make re-pushing apps faster. Maybe that can make future pushes faster too?
• Use smaller builder image in tests. This will reduce the time to install Epinio ( Add builder image arg to push #623 )
• Create a "default" list of applications that are installed in "BeforeSuite". Those should be "read-only". In tests where we only
  check the output of a command (e..g epinio apps list) we can avoid pushing applications if we simply use the "default" ones. This will only save us time if we do that a lot, e.g. many tests for read-only commands (list, show, both testing the cli and the API).