Improve pipeline performance

[tseelbach]

The wall clock time to run a tekton pipeline in RHOCP is approx 7x over what it takes to run a basic Appsody Build on your laptop. This will not allow us to scale into production. Study performance for both wall clock and cpu times, and recommended tuning and changes necessary to scale functions up.

[marikaj123]

The following feature will contribute to the Performance Study with the goal to reduce Tekton pipeline time by 10x.

[jdmcclur]

My analysis looking at the java-openliberty stack build-push step.

There is no caching is done, so every build pulls two images and does a bunch of downloads from maven.

1. If it's possible to persist a container registry in the cluster somehow, we could eliminate the following time (after a first build)

Pulling images
53.5 seconds - Pulling docker image docker.io/kabanero/java-openliberty:0.2    53.5.  (docker.io/kabanero/java-openliberty:0.2)
30.2 seconds - STEP 16: FROM openliberty/open-liberty:19.0.0.12-kernel-java8-openj9-ubi    30.2 (openliberty/open-liberty:19.0.0.12-kernel-java8-openj9-ubi )

2. buildah does not cache things by default, (and running with --layers has not seemed to work well for me). With docker on my laptop the first two STEPS below get cached after the first build.
   If we could get buildah to do the same thing, it would help a lot also. We also need to look into if we really need all this stuff downloaded from maven or if we could persist a maven cache somewhere.

Maven Building/Downloading -> maven is slower on tekton than my laptop/docker
58.96 seconds - STEP 5: RUN cd /project && mvn -B ins2020-03-12T23:13:23.674657078Z tall dependency:go-offline -DskipTests    58.96
92.5 seconds  - STEP 7: RUN cd /project/user-app &&     mvn -B -f /project/user-app/preload-m2-pom.xml liberty:install-server dependency:go-offline    92.5
26.5 seconds - STEP 14 RUN cd /project/user-app &&     rm -f src/main/liberty/config/configDropins/defaults/quick-start-security.xml &&     mvn -Pappsody-build