Replace appsody extract build with specific appsody build

[stephenkinder]

The Kabanero build pipelines need to start using the appsody build command to build the application so that the appropriate container tags are added for kappnav.

[kvijai82]

• Use the appsody/appsody-buildah:0.5.0 image
• Update assemble-extract step to not call extract.sh; instead just cd /workspace/$gitsource
• Then call appsody build --buildah --buildah-options "--format=docker --tls-verify=false" (we should not need —tls-verify=false with the cert stuff we jammed in last week but leaving it in there for now if you have issues)
• Delete the build step
• Print out app-deploy.yaml after appsody build and in the beginning of the modify-app-deploy.yaml step in deploy task and make sure they look the same
• Once it works, update all the build tasks

[aadeshpa]

The update is as below

Blocker: We are blocked for nodejs and nodejs-express pipelines with the issue

appsody/appsody-buildah#10

that we created in appsody/appsody-buildah repository after discussion with @christik@us.ibm.com and Enrique.Lacal.Bereslawski@ibm.com.
The issue is in appsody-buildah:0.5.0 container image which we need for solving this current issue.

Forward path progress with java-microprofile collection pipeline which works with appsody-buildah:0.5.0 image without above blocker issue:
We followed further steps and below is the progress.

1. We tried the steps mentioned by @kvijai82 and found we could use appsody build --buildah command and build the project now which creates app-deploy.yaml file as a part of the build.
2. We saw build and push pipeline runs successfully with above point 1 change.
3. We are still figuring out how to share that generated app-deploy.yaml in the deploy task, since deploy task has new pod and we do not know yet how to share the files accross the pods. We have shared the files across the containers in same pod in existing pipelines code.

[brianxjx]

We were able to use a nfs PV to allow us to span Tasks but that doesn't seem like a good solution.

Tekton team pointed us to a new requirement for passing outputs to inputs for Tasks. This looks to be problematic from a types perspective (Task input/output resources are typed)

Question: Where is the generated 'app-deploy.yaml' ultimate home?

[marikaj123]

Please use the release Dec 2 rather than the label release-0.4

[aadeshpa]

For the forward path : we got suggestion to copy the app-deploy.yaml file from build task input to output and then use it in deploy task.

example given by Adams :
Use of From syntax
https://github.com/tektoncd/pipeline/blob/76e5d8b6678d510459c25eb49ba597bdec1499ac/examples/pipelineruns/output-pipelinerun.yaml

and below example showing how to copy the file from input to output

https://github.com/tektoncd/dashboard/blob/master/tekton/build.yml

We tried using the above two examples and we found that the file gets copied from one task build task input to output which copies the file to pvc volume, however the next task deploy task could not get it copied from the pvc volume back to the pod to access it. Posting the log outputs from the tasks.

[aadeshpa]

This is the build task where copy seems to happen correctly from the task pod to the pvc

======
copy-files-to-output-resource 
Input source path=/workspace/git-source
Output source path=/workspace/output/git-source
copy of app-deploy.yaml completed to output resource
printing content from  /workspace/output/git-source
giving 777 permision to /workspace/output/git-source/app-deploy.yaml
total 4
-rwxrwxrwx. 1 root root 2461 Nov 22 20:34 app-deploy.yaml
Getting image source signatures
Copying blob sha256:9a250b509a7d6475d0d11538b8f18a811e9cabf74d143f6991bbd75b3773d55f
Copying blob sha256:cee362048fe2af292115b96f6206da26562f2f57a9282adcb3f8d2fdc4c897cd
Copying blob sha256:481324a7ba6d9e3d7c1b6082f1868c5e43a8008a676cc150e9e1856edfceb39d
Copying blob sha256:ced05275a32ae89ba0b20370e75e28042e7fac7339752c9fc3fe5de3f23969ed
Copying blob sha256:26429bebe01965d18e655f2de221305885f89f42245ac67b10b1c4250227ac56
Copying blob sha256:e5d43876cf221560fc78191591b8c15b8707f393ef16418547956b0f16295c84
Copying blob sha256:e32c06ccdc29345a33a19807067d30a77f71197783def9154cc3ac255ba660f6
Copying blob sha256:02be5ac6a7d3c8bc39935103e2f3b027573457fc875c5a1842ee2f6cecb6fe7e
Copying blob sha256:e81c17a417fa3535262c836c06aec9c7abced051994169bf1a65149c20979488
Copying blob sha256:33202e24ae0d06dc19c82f970bbe051a71a34f1da03ffee3803007a8497179c0
Copying blob sha256:de634ec556cbf1e160d47ea6a1216435126d22534be61c7e6606f737180d2d03
Copying config sha256:3423c7dde4085033237ecab4e761a8d1911e9b6c6a71bfba60e489f6fc0e1531
Writing manifest to image destination
Storing signatures
Successfully pushed //aadeshpa/java-microprofile-image:latest@sha256:c24b56e7ce576a30be8869921d05b846ae3a1f3ad125d0176541fd5f357ad841
2019/11/22 20:34:16 ImageResource docker-image doesn't have an index.json file: stat /builder/home/image-outputs/docker-image/index.json: no such file or directory
2019/11/22 20:34:16 Image digest exporter output: [] 
{"level":"warn","ts":1574454857.2759147,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: \"KO_DATA_PATH\" does not exist or is empty"}
{"level":"info","ts":1574454857.2807634,"logger":"fallback-logger","caller":"bash/main.go:64","msg":"Successfully executed command \"sh -c mkdir -p /pvc/build-push-task/git-source\"; output "}
{"level":"warn","ts":1574454857.5559783,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: \"KO_DATA_PATH\" does not exist or is empty"}
{"level":"info","ts":1574454857.560734,"logger":"fallback-logger","caller":"bash/main.go:64","msg":"Successfully executed command \"sh -c cp -r /workspace/output/git-source/. /pvc/build-push-task/git-source\"; output "}
[root@peered-inf manual-pipeline-runs]# oc get pods

[aadeshpa]

This is the deploy task which was suppose to copy it and use that app-deploy.yaml file in its pod, however it gives Permission denied for some reason .

[root@peered-inf manual-pipeline-runs]# oc logs --max-log-requests=20 ashish-java-mp-build-push-deploy-manual-pipeline-run-depl-l75xd-pod-c5fe1d --all-containers
{"level":"warn","ts":1574454868.6284902,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: \"KO_DATA_PATH\" does not exist or is empty"}
{"level":"info","ts":1574454868.6299603,"logger":"fallback-logger","caller":"creds-init/main.go:40","msg":"Credentials initialized."}
{"level":"warn","ts":1574454878.484739,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: \"KO_DATA_PATH\" does not exist or is empty"}
{"level":"info","ts":1574454878.4896142,"logger":"fallback-logger","caller":"bash/main.go:64","msg":"Successfully executed command \"sh -c mkdir -p /workspace/git-source\"; output "}
{"level":"warn","ts":1574454878.934932,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: \"KO_DATA_PATH\" does not exist or is empty"}
{"level":"fatal","ts":1574454878.9406023,"logger":"fallback-logger","caller":"bash/main.go:62","msg":"Error executing command \"sh -c cp -r /pvc/build-push-task/git-source/. /workspace/git-source\" ; error exit status 1; cmd_output cp: cannot stat '/pvc/build-push-task/git-source/./app-deploy.yaml': Permission denied\n","stacktrace":"main.main\n\t/go/src/github.com/tektoncd/pipeline/cmd/bash/main.go:62\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:198"}
[root@peered-inf manual-pipeline-runs]# 

[aadeshpa]

I am getting above permission denied when I use the pv.yaml which is static persistent volume for the hostpath /var/lib/manual-pipeline-run-pvc . Somehow the build task is pushing it to the path, however when deploy task starts and it tries to copy the content from /pvc/build-push-task/git-source/. to /workspace/git-source\ we get permission denied.

Found some relevant issue which inclines some additional scc permissions to be given but not sure how
: openshift/origin#15850

@kvijai82 @brianxjx @smcclem

[aadeshpa]

For the blocker path with appsody-buildah, they have rollbacked the buildah 1.9.1 to 1.9.0 in the new image appsody/appsody-buildah:0.5.0-buildah1.9.0 and we tested with this image and it does not cause the issue in nodejs, nodejs-express and nodejs-loopback pipelineruns. So removing the blocker label from this issue.

[aadeshpa]

We were successful in moving the generated app-deploy.yaml file from build task to deploy task as a part of this issue.
The pull requested for the changes is created in the pipelines repository:

#148

cc: @brianxjx @kvijai82 @kyle @stephenkinder

[aadeshpa]

After discussion with @stephenkinder , we are not using the above pull request where we use two tasks for build and deploy, and we are not combining those two tasks as single task. So we do not need to move the app-deploy.yaml file between two pods. cc: @kvijai82 @smcclem

new pull request in pipelines repo is : #149

[aadeshpa]

Created the pull request for this in Collections repository
Pull request : kabanero-io/collections#204

[marikaj123]

As per Vijai, closing it