Update pipeline-service #1725

Roming22 · 2023-04-25T19:28:03Z

OSP-1.10
PaC deployed through OSP-1.10
allow for watcher logging level customization via gitops

gabemontero · 2023-04-25T20:16:14Z

I would think we would want to remove tekton-chains at https://github.com/openshift-pipelines/pipeline-service/tree/main/operator/gitops/argocd/pipeline-service/tekton-chains as well @Roming22 and validate that removal via pipeline service CI before updating staging

Admittedly though, I'm a little unclear about how we override openshift-pipelines over at https://github.com/openshift-pipelines/pipeline-service/tree/main/operator/gitops/argocd/pipeline-service/openshift-pipelines so as to install or not install tekton-chains

of course, maybe I'm wrong here .... can you elaborate ?

thanks

Roming22 · 2023-04-25T20:31:18Z

@gabemontero I'm not sure I understand why removing tekton-chains is a pre-requiste to upgrading to OSP-1.10.

We certainly want to remove it, sooner than later, but it's going to take some time before it's ready, and if possible I'd like to validate that OSP-1.10 is running smoothly as soon as possible to avoid last minute surprises before the freeze.

gabemontero · 2023-04-25T20:50:08Z

@gabemontero I'm not sure I understand why removing tekton-chains is a pre-requiste to upgrading to OSP-1.10.

We certainly want to remove it, sooner than later, but it's going to take some time before it's ready, and if possible I'd like to validate that OSP-1.10 is running smoothly as soon as possible to avoid last minute surprises before the freeze.

I thought PaC and tekton-chains were the same in that we had to install them from upstream to get required levels, but at 1.10 both the PaC and tekton-chains installed by openshift-pipelines were at sufficient levels

What motivated the removal of PaC via openshift-pipelines/pipeline-service#620 to be included, bu not chains ... I would think we would want to be consistent.

Please clarify when you get the chance, but I won't block merge, assuming e2e's pass (they did not on the first go)

/lgtm

gabemontero · 2023-04-25T20:53:20Z

and the e2e failed with a gitops hiccup wrt pipeline service:

 pipeline-service-in-cluster-local                 Unknown     Healthy
pipeline-service-in-cluster-local failed with:
[{"lastTransitionTime":"2023-04-25T20:26:09Z","message":"rpc error: code = Unknown desc = `kustomize build .components/pipeline-service/development` failed exit status 1: Error: accumulating resources: accumulation err='accumulating resources from 'git::https://github.com/openshift-pipelines/pipeline-service.git/developer/openshift/gitops/argocd/pipeline-service-storage?ref=14431236de0f1e3d3a9e27c145197d1a45942794': evalsymlink failure on '.components/pipeline-service/development/git::https:/github.com/openshift-pipelines/pipeline-service.git/developer/openshift/gitops/argocd/pipeline-service-storage?ref=14431236de0f1e3d3a9e27c145197d1a45942794' : lstat .components/pipeline-service/development/git::https:: no such file or directory': evalsymlink failure on '/tmp/kustomize-2974330201/developer/openshift/gitops/argocd/pipeline-service-storage' : lstat /tmp/kustomize-2974330201/developer/openshift/gitops/argocd/pipeline-service-storage: no such file or directory","type":"ComparisonError"}]

at first quick glance can't say for certain if it is related to the bump ...

/test appstudio-e2e-tests

Mo3m3n · 2023-04-25T21:05:17Z

@Roming22 can you update the commit message to add "using tls for rds connections" as result of this update ?

flacatus · 2023-04-25T21:52:48Z

/retest

gabemontero · 2023-04-25T22:58:37Z

@Roming22 - unfortunately the latest run https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/redhat-appstudio_infra-deployments/1725/pull-ci-redhat-appstudio-infra-deployments-main-appstudio-e2e-tests/1650981219980021760 failed in the same way as what I reported in #1725 (comment)

I think that points to gitops / kustomize is not happy with something in pipeline-service between 062d00d304d6fba316085784f3b948217fc99d12 and 14431236de0f1e3d3a9e27c145197d1a45942794

By rough count I see 21 or 22 commits between those 2 commits, so if I am correct there is a large set to choose from, unless the error message wrt pipeline-service-storage helps narrow it down

My best guess based on that clue is that it is somehow related to openshift-pipelines/pipeline-service@a07a5a0 from @Mo3m3n based on a comment in that file, but that is certainly guess that is more wild than educated, especially since that merged before 062d00d304d6fba316085784f3b948217fc99d12

adambkaplan · 2023-04-26T01:38:11Z

The external secrets for Pipelines as Code probably need to be updated. I think the PR as is will break staging because PaC will be deployed in the openshift-pipelines namespace.

components/pipeline-service/development/kustomization.yaml

gabemontero

/lgtm

Roming22 · 2023-04-26T15:15:34Z

@Mo3m3n @adambkaplan Comments processed. SHA updated, and there's no references to the pipelines-as-code namespace anymore.

> kustomize build components/pipeline-service/staging/base | grep -c "namespace: pipelines-as-code"
0

gabemontero · 2023-04-26T15:31:26Z

/lgtm

gabemontero · 2023-04-26T15:32:30Z

pr needs a rebase @Roming22

do you want to do it, or I can hit the "update with rebase" button if you like

components/pipeline-service/staging/base/kustomization.yaml

gabemontero · 2023-04-26T18:39:18Z

/lgtm

Mo3m3n · 2023-04-26T19:10:25Z

@Roming22 I have just found that you also need to update this line https://github.com/redhat-appstudio/infra-deployments/blob/main/hack/secret-creator/create-plnsvc-secrets.sh#L27 to match this change done here in pipeline-service openshift-pipelines/pipeline-service@b5f1dc2#diff-f394b1c9744cc285b6ac80384e967bd73210dd72a905e1ad7581b8ecab4e3773R201

Basically when switching to the helm based postgres deployment the name of the db service changed too (for dev mode). That's why I suggested in my comment to also add to the commit message that we are enabling tls with this update and using Helm for postgres deployment

gabemontero · 2023-04-28T13:30:47Z

@Mo3m3n - we minimally need #1737 to merge in order to pick up @mmorhun 's fix for with namespace PaC is installed in. It has not merged as I type this comment.

For what you noted with https://github.com/redhat-appstudio/infra-deployments/tree/main/components/authentication and https://github.com/redhat-appstudio/infra-deployments/tree/main/components/sprayproxy yes that is part of what @Michkov noted in #1725 (comment)

I would say @Roming22 each of those need to be addressed as part of this PR

Once all that happens, we can see about rebase/retest and try to merge

gabemontero · 2023-04-28T13:55:16Z

@Mo3m3n - we minimally need #1737 to merge in order to pick up @mmorhun 's fix for with namespace PaC is installed in. It has not merged as I type this comment.

@Roming22 just pulled ^^ into this PR :-)

For what you noted with https://github.com/redhat-appstudio/infra-deployments/tree/main/components/authentication and https://github.com/redhat-appstudio/infra-deployments/tree/main/components/sprayproxy yes that is part of what @Michkov noted in #1725 (comment)

I would say @Roming22 each of those need to be addressed as part of this PR

Once all that happens, we can see about rebase/retest and try to merge

Roming22 · 2023-04-28T15:01:08Z

@Michkov I believe I've updated all references to pipelines-as-code. I tried to make sure not to impact production by patching where necessary.

gabemontero · 2023-04-28T16:25:08Z

the last CI failure needs #1579

we just (possibly re-)triaged that mvp-demo failure in https://redhat-internal.slack.com/archives/C02FANRBZQD/p1682697746956539

gabemontero · 2023-04-28T19:44:49Z

hopefully konflux-ci/e2e-tests#488 fixes this

gabemontero · 2023-04-28T20:54:02Z

/lgtm

gabemontero · 2023-04-29T13:41:15Z

more new unrelated flakes being discussed in slack (I've asked QE to open blocking bugs and put skips in these failing tests until they are sorted out).

/retest

gabemontero · 2023-04-30T22:50:58Z

the latest failure is in some of the pipelines as code e2e code, so in theory could be related to this bump, as we are pulling in a new version of PaC, one which is installed by the openshift pipelines operator instead of upstream yaml

the pac logs, especially the controller, do not have a lot in them

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/redhat-appstudio_infra-deployments/1725/pull-ci-redhat-appstudio-infra-deployments-main-appstudio-e2e-tests/1652307069509832704/artifacts/appstudio-e2e-tests/redhat-appstudio-hypershift-gather/artifacts/pods/openshift-pipelines_pipelines-as-code-controller-5866597c4c-mjp5l_pac-controller.log
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/redhat-appstudio_infra-deployments/1725/pull-ci-redhat-appstudio-infra-deployments-main-appstudio-e2e-tests/1652307069509832704/artifacts/appstudio-e2e-tests/redhat-appstudio-hypershift-gather/artifacts/pods/openshift-pipelines_pipelines-as-code-watcher-7848847bf6-qxvr4_pac-watcher.log
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/redhat-appstudio_infra-deployments/1725/pull-ci-redhat-appstudio-infra-deployments-main-appstudio-e2e-tests/1652307069509832704/artifacts/appstudio-e2e-tests/redhat-appstudio-hypershift-gather/artifacts/pods/openshift-pipelines_pipelines-as-code-webhook-6b4f79686c-lgvcl_pac-webhook.log

maybe github throttling but is hard to say

gabemontero · 2023-04-30T22:51:56Z

/retest

gabemontero · 2023-05-01T12:27:24Z

I see the same timed out when waiting for init PaC PR to be created error @Roming22 .... with minimal activity in the PaC pods

By comparison, #1741 is showing green with the older version of PaC.

Might be time to try out this locally, work with @psturc and @flacatus to understand the details of this test, see where the breakdown is.

Michkov · 2023-05-02T05:15:17Z

Issue in build-service:

{"level":"error","ts":"2023-04-30T23:29:58.624Z","logger":"ComponentOnboarding.PaC-setup","caller":"controllers/component_build_controller_pac.go:415","msg":"failed to get Component PaC repository object","controller":"component","controllerGroup":"appstudio.redhat.com","controllerKind":"Component","Component":{"name":"test-mvp-component-kpta","namespace":"mvp-demo-dev-namespace-cbpg-tenant"},"namespace":"mvp-demo-dev-namespace-cbpg-tenant","name":"test-mvp-component-kpta","reconcileID":"9a13383c-095c-4cdc-91f8-a7a4898f6f74","action":"VIEW","error":"no matches for kind \"Repository\" in version \"pipelinesascode.tekton.dev/v1alpha1\"","stacktrace":"github.com/redhat-appstudio/build-service/controllers.(*ComponentBuildReconciler).ensurePaCRepository\n\t/opt/app-root/src/controllers/component_build_controller_pac.go:415\ngithub.com/redhat-appstudio/build-service/controllers.(*ComponentBuildReconciler).ProvisionPaCForComponent\n\t/opt/app-root/src/controllers/component_build_controller_pac.go:113\ngithub.com/redhat-appstudio/build-service/controllers.(*ComponentBuildReconciler).Reconcile\n\t/opt/app-root/src/controllers/component_build_controller.go:297\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235"}

no matches for kind \"Repository\" in version \"pipelinesascode.tekton.dev/v1alpha1\" -> trying to reproduce

openshift-ci · 2023-05-02T05:48:30Z

New changes are detected. LGTM label has been removed.

* OSP-1.10 * PaC deployed through OSP-1.10 * Allow for watcher logging level customization via gitops * Using TLS for RDS connections Signed-off-by: Romain Arnaud <rarnaud@redhat.com>

Michkov · 2023-05-02T18:57:24Z

{"severity":"error","timestamp":"2023-05-02T16:51:35.531Z","logger":"pipelines-as-code-webhook.ValidationWebhook","caller":"controller/controller.go:566","message":"Reconcile error","knative.dev/traceid":"78b06509-0c86-464b-8b97-034e745398df","knative.dev/key":"validation.pipelinesascode.tekton.dev","duration":0.000023802,"error":"secret \"pipelines-as-code-webhook-certs\" is missing \"ca-cert.pem\" key","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\t/go/src/github.com/openshift-pipelines/pipelines-as-code/vendor/knative.dev/pkg/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\t/go/src/github.com/openshift-pipelines/pipelines-as-code/vendor/knative.dev/pkg/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\t/go/src/github.com/openshift-pipelines/pipelines-as-code/vendor/knative.dev/pkg/controller/controller.go:491"}
-> "error":"secret "pipelines-as-code-webhook-certs" is missing "ca-cert.pem" key"

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/redhat-appstudio_infra-deployments/1725/pull-ci-redhat-appstudio-infra-deployments-main-appstudio-e2e-tests/1653436401850519552/artifacts/appstudio-e2e-tests/redhat-appstudio-hypershift-gather/artifacts/pods/openshift-pipelines_pipelines-as-code-webhook-6b4f79686c-bsvr2_pac-webhook.log

gabemontero · 2023-05-02T18:57:48Z

So still the same error in the last e2e run.

@Roming22 - so in my local repro of this, I killed the build-service pod, and when it restarted, it no longer got the no matches for kind \"Repository\" in version \"[pipelinesascode.tekton.dev/v1alpha1](http://pipelinesascode.tekton.dev/v1alpha1)errors and the Repository object was created in the test namespace (which I had left up) after it was missing during the test failure

I wonder if this is a startup order sort of thing, where the build-service initializes its client before the openshift-pipelines operator has "fully installed things", where as previously, PaC was getting created by pipeline-service directly

Here are the current sync-waves I see for the two:

gmontero ~/go/src/github.com/redhat-appstudio/infra-deployments  (main)$ find . -name "*.yaml" -exec grep -s -H sync-wave {} \;
./components/build-service/base/external-secrets/pipelines-as-code-secret.yaml:    argocd.argoproj.io/sync-wave: "-1"
./components/pipeline-service/base/external-secrets/pipelines-as-code/pipelines-as-code-secret.yaml:    argocd.argoproj.io/sync-wave: "-1"
./components/pipeline-service/base/external-secrets/tekton-results/tekton-results-database.yaml:    argocd.argoproj.io/sync-wave: "-1"
./components/pipeline-service/base/external-secrets/tekton-results/tekton-results-s3.yaml:    argocd.argoproj.io/sync-wave: "-1"

gabemontero · 2023-05-02T18:58:58Z

{"severity":"error","timestamp":"2023-05-02T16:51:35.531Z","logger":"pipelines-as-code-webhook.ValidationWebhook","caller":"controller/controller.go:566","message":"Reconcile error","knative.dev/traceid":"78b06509-0c86-464b-8b97-034e745398df","knative.dev/key":"validation.pipelinesascode.tekton.dev","duration":0.000023802,"error":"secret \"pipelines-as-code-webhook-certs\" is missing \"ca-cert.pem\" key","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\t/go/src/github.com/openshift-pipelines/pipelines-as-code/vendor/knative.dev/pkg/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\t/go/src/github.com/openshift-pipelines/pipelines-as-code/vendor/knative.dev/pkg/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\t/go/src/github.com/openshift-pipelines/pipelines-as-code/vendor/knative.dev/pkg/controller/controller.go:491"} -> "error":"secret "pipelines-as-code-webhook-certs" is missing "ca-cert.pem" key"

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/redhat-appstudio_infra-deployments/1725/pull-ci-redhat-appstudio-infra-deployments-main-appstudio-e2e-tests/1653436401850519552/artifacts/appstudio-e2e-tests/redhat-appstudio-hypershift-gather/artifacts/pods/openshift-pipelines_pipelines-as-code-webhook-6b4f79686c-bsvr2_pac-webhook.log

looks like we officially have multiple problems

openshift-ci · 2023-05-02T19:59:50Z

@Roming22: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/appstudio-e2e-tests	`ed9265b`	link	true	`/test appstudio-e2e-tests`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Roming22 · 2023-05-02T21:19:23Z

Closed in favor of #1769

We'll be taking baby steps:

Deploy OSP 1.10 without any changes to PaC
Update PaC and tekton-chains images to use images built by Red Hat
Deploy tekton-results through the operator
Deploy tekton-chains through the operator

openshift-ci bot requested review from adambkaplan and bnallapeta April 25, 2023 19:28

openshift-ci bot added the approved label Apr 25, 2023

Roming22 requested review from gabemontero and Mo3m3n April 25, 2023 19:28

openshift-ci bot assigned gabemontero Apr 25, 2023

openshift-ci bot added the lgtm label Apr 25, 2023

Mo3m3n reviewed Apr 26, 2023

View reviewed changes

components/pipeline-service/development/kustomization.yaml Outdated Show resolved Hide resolved

openshift-ci bot removed the lgtm label Apr 26, 2023

gabemontero approved these changes Apr 26, 2023

View reviewed changes

openshift-ci bot added lgtm and removed lgtm labels Apr 26, 2023

openshift-ci bot added the lgtm label Apr 26, 2023

Mo3m3n reviewed Apr 26, 2023

View reviewed changes

components/pipeline-service/staging/base/kustomization.yaml Show resolved Hide resolved

openshift-ci bot removed the lgtm label Apr 26, 2023

openshift-ci bot added the lgtm label Apr 26, 2023

openshift-ci bot removed the lgtm label Apr 26, 2023

gabemontero mentioned this pull request Apr 28, 2023

build-service update #1737

Merged

openshift-ci bot removed the lgtm label Apr 28, 2023

openshift-ci bot added the lgtm label Apr 28, 2023

gabemontero mentioned this pull request May 1, 2023

DO NOT MERGE! Work in progress #1741

Closed

openshift-ci bot removed the lgtm label May 2, 2023

Michkov mentioned this pull request May 2, 2023

bump pipelines-as-code to v0.17.3 konflux-ci/build-service#149

Merged

Update pipeline-service

be22e4d

* OSP-1.10 * PaC deployed through OSP-1.10 * Allow for watcher logging level customization via gitops * Using TLS for RDS connections Signed-off-by: Romain Arnaud <rarnaud@redhat.com>

gabemontero mentioned this pull request May 2, 2023

WIP: PLNSRVCE-1271: bump pipelines-as-code to v0.17.3 redhat-appstudio/application-service#311

Closed

3 tasks

Michkov and others added 3 commits May 2, 2023 20:47

Merge branch 'main' into update-pipeline-service

1f19068

Update kustomization.yaml

f7e87d8

Update kustomization.yaml

ed9265b

gabemontero mentioned this pull request May 2, 2023

Revert "Remove independent install of PaC as it is installed via the … openshift-pipelines/pipeline-service#630

Merged

Roming22 closed this May 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update pipeline-service #1725

Update pipeline-service #1725

Roming22 commented Apr 25, 2023

gabemontero commented Apr 25, 2023

Roming22 commented Apr 25, 2023

gabemontero commented Apr 25, 2023

gabemontero commented Apr 25, 2023

Mo3m3n commented Apr 25, 2023

flacatus commented Apr 25, 2023

gabemontero commented Apr 25, 2023 •

edited

Loading

adambkaplan commented Apr 26, 2023

gabemontero left a comment

Roming22 commented Apr 26, 2023

gabemontero commented Apr 26, 2023

gabemontero commented Apr 26, 2023

gabemontero commented Apr 26, 2023

Mo3m3n commented Apr 26, 2023

gabemontero commented Apr 28, 2023

gabemontero commented Apr 28, 2023

Roming22 commented Apr 28, 2023

gabemontero commented Apr 28, 2023

gabemontero commented Apr 28, 2023

gabemontero commented Apr 28, 2023

gabemontero commented Apr 29, 2023

gabemontero commented Apr 30, 2023

gabemontero commented Apr 30, 2023

gabemontero commented May 1, 2023

Michkov commented May 2, 2023 •

edited

Loading

openshift-ci bot commented May 2, 2023

Michkov commented May 2, 2023

gabemontero commented May 2, 2023 •

edited

Loading

gabemontero commented May 2, 2023

openshift-ci bot commented May 2, 2023

Roming22 commented May 2, 2023

Update pipeline-service #1725

Update pipeline-service #1725

Conversation

Roming22 commented Apr 25, 2023

gabemontero commented Apr 25, 2023

Roming22 commented Apr 25, 2023

gabemontero commented Apr 25, 2023

gabemontero commented Apr 25, 2023

Mo3m3n commented Apr 25, 2023

flacatus commented Apr 25, 2023

gabemontero commented Apr 25, 2023 • edited Loading

adambkaplan commented Apr 26, 2023

gabemontero left a comment

Choose a reason for hiding this comment

Roming22 commented Apr 26, 2023

gabemontero commented Apr 26, 2023

gabemontero commented Apr 26, 2023

gabemontero commented Apr 26, 2023

Mo3m3n commented Apr 26, 2023

gabemontero commented Apr 28, 2023

gabemontero commented Apr 28, 2023

Roming22 commented Apr 28, 2023

gabemontero commented Apr 28, 2023

gabemontero commented Apr 28, 2023

gabemontero commented Apr 28, 2023

gabemontero commented Apr 29, 2023

gabemontero commented Apr 30, 2023

gabemontero commented Apr 30, 2023

gabemontero commented May 1, 2023

Michkov commented May 2, 2023 • edited Loading

openshift-ci bot commented May 2, 2023

Michkov commented May 2, 2023

gabemontero commented May 2, 2023 • edited Loading

gabemontero commented May 2, 2023

openshift-ci bot commented May 2, 2023

Roming22 commented May 2, 2023

gabemontero commented Apr 25, 2023 •

edited

Loading

Michkov commented May 2, 2023 •

edited

Loading

gabemontero commented May 2, 2023 •

edited

Loading