Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

e2e test TestKamelCLIRun (Run with http dependency) consistently failing on OCP4 #3708

Closed
phantomjinx opened this issue Oct 1, 2022 · 10 comments
Labels
area/openshift Related to OpenShift status/stale

Comments

@phantomjinx
Copy link
Contributor

phantomjinx commented Oct 1, 2022

The test can be found at https://github.com/apache/camel-k/blob/main/e2e/namespace/install/cli/run_test.go#L132

The command invoked is kamel run --operator-id ${operatorID} -n ${ns} "../../../global/common/traits/files/jvm/Classpath.java" -d "https://github.com/apache/camel-k/raw/main/e2e/global/common/traits/files/jvm/sample-1.0.jar".

See example failure log here.

Running the test locally, I can repeat the failure and found it is the integration kit that generates an error. This is due to the github dependency not being found:

{"level":"error","ts":1664568016.4109485,"logger":"camel-k.maven.build","msg":"Failed to execute goal on project camel-k-integration: Could not resolve dependencies for project org.apache.camel.k.integration:camel-k-integration:jar:1.11.0-SNAPSHOT: org.apache.camel.example:sample:jar:1.0 was not found in https://repository.apache.org/content/repositories/snapshots during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of apache-snapshots has elapsed or updates are forced -> [Help 1]","stacktrace":"github.com/apache/camel-k/pkg/util/log.Logger.Errorf\n\tgithub.com/apache/camel-k/pkg/util/log/log.go:60\ngithub.com/apache/camel-k/pkg/util/maven.normalizeLog\n\tgithub.com/apache/camel-k/pkg/util/maven/maven_log.go:75\ngithub.com/apache/camel-k/pkg/util/maven.mavenLogHandler\n\tgithub.com/apache/camel-k/pkg/util/maven/maven_log.go:53\ngithub.com/apache/camel-k/pkg/util.RunAndLog.func1\n\tgithub.com/apache/camel-k/pkg/util/command.go:58\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/errgroup/errgroup.go:75"}

I can confirm using verbose logging and adding some print statements that the process of downloading the jar dependency and uploading it as an image to the OCP4 registry is executed:

Artifact http path  test-72890e77-18e5-4691-ade9-766d603dd3e0/maven_org_apache_camel_example_sample_1_0_sample-1_0_pom
spectrum - 2022/10/01 00:04:03 Pulling base image  (insecure=true)...
spectrum - 2022/10/01 00:04:03 Composing layers...
spectrum - 2022/10/01 00:04:03 Pushing image image-registry.openshift-image-registry.svc:5000/test-72890e77-18e5-4691-ade9-766d603dd3e0/maven_org_apache_camel_example_sample_1_0_sample-1_0_pom:1.0 (insecure=true)...
spectrum - 2022/10/01 00:04:03 pushed blob: sha256:ad1c053ef0b4e353c44cd805a8d4c25ecfd13f30ef669ed4aad66bf44d39f33b
spectrum - 2022/10/01 00:04:03 pushed blob: sha256:c600cb2a3f711bc95794a9f76b0eb8b25baa358d27a47fe389b65fbf121f6315
spectrum - 2022/10/01 00:04:03 image-registry.openshift-image-registry.svc:5000/test-72890e77-18e5-4691-ade9-766d603dd3e0/maven_org_apache_camel_example_sample_1_0_sample-1_0_pom:1.0: digest: sha256:28c31355187fbce117e23eb2c3d94487b4f48b210f051d564846fd6cf9205e5d size: 423
Uploaded: /tmp/camel-k1004906173/pom.xml to image-registry.openshift-image-registry.svc:5000/test-72890e77-18e5-4691-ade9-766d603dd3e0/maven_org_apache_camel_example_sample_1_0_sample-1_0_pom:1.0 
spectrum - 2022/10/01 00:04:03 Pulling base image  (insecure=true)...
spectrum - 2022/10/01 00:04:03 Composing layers...
spectrum - 2022/10/01 00:04:03 Pushing image image-registry.openshift-image-registry.svc:5000/test-72890e77-18e5-4691-ade9-766d603dd3e0/maven_org_apache_camel_example_sample_1_0_sample-1_0_pom_md5:1.0 (insecure=true)...
spectrum - 2022/10/01 00:04:03 pushed blob: sha256:2c6704a8e294cf1506837f859b4041d4ff6b6cd2b568c63fbb348eeac2cbb963
spectrum - 2022/10/01 00:04:03 pushed blob: sha256:8e80a3fca699e3536dfd74d1b3d109d5883a583038b46147c0991d8810315cba
spectrum - 2022/10/01 00:04:03 image-registry.openshift-image-registry.svc:5000/test-72890e77-18e5-4691-ade9-766d603dd3e0/maven_org_apache_camel_example_sample_1_0_sample-1_0_pom_md5:1.0: digest: sha256:a9eb77988bd626841a6bc032056bef147dc81f824b8c06115c18cda599d63e01 size: 423
spectrum - 2022/10/01 00:04:03 Pulling base image  (insecure=true)...
spectrum - 2022/10/01 00:04:03 Composing layers...
spectrum - 2022/10/01 00:04:03 Pushing image image-registry.openshift-image-registry.svc:5000/test-72890e77-18e5-4691-ade9-766d603dd3e0/maven_org_apache_camel_example_sample_1_0_sample-1_0_pom_sha1:1.0 (insecure=true)...
spectrum - 2022/10/01 00:04:03 pushed blob: sha256:488f58133fe9651f30b5e5079b014659caf28fdd3c7691d2ccc54382eb7abdf5
spectrum - 2022/10/01 00:04:03 pushed blob: sha256:1e0e60be433e1f2acbc9b0334453d19098cfc4be7d4398d652185fa2b5bbdb8c
spectrum - 2022/10/01 00:04:03 image-registry.openshift-image-registry.svc:5000/test-72890e77-18e5-4691-ade9-766d603dd3e0/maven_org_apache_camel_example_sample_1_0_sample-1_0_pom_sha1:1.0: digest: sha256:a8a66c149d0356a326660ce5631295214eeba47a5a60bed898ad72966710de77 size: 423
Added mvn:org.apache.camel.example:sample:jar:1.0 to the Integration's dependency list 
Added mvn:org.apache.camel.example:sample:jar:1.0 to the Integration's dependency list 
Artifact http path  test-72890e77-18e5-4691-ade9-766d603dd3e0/maven_org_apache_camel_example_sample_1_0_sample-1_0_jar
spectrum - 2022/10/01 00:04:03 Pulling base image  (insecure=true)...
spectrum - 2022/10/01 00:04:03 Composing layers...
spectrum - 2022/10/01 00:04:03 Pushing image image-registry.openshift-image-registry.svc:5000/test-72890e77-18e5-4691-ade9-766d603dd3e0/maven_org_apache_camel_example_sample_1_0_sample-1_0_jar:1.0 (insecure=true)...
spectrum - 2022/10/01 00:04:03 pushed blob: sha256:d55fb5a992a3f1fe245a210a2dbdcaac8a2799af3defee43772f40c63f1e3be9
spectrum - 2022/10/01 00:04:03 pushed blob: sha256:83270fb874bd7790c08692dccac22f6f8da9068f0f2822a7fa4949e78d44f377
spectrum - 2022/10/01 00:04:03 image-registry.openshift-image-registry.svc:5000/test-72890e77-18e5-4691-ade9-766d603dd3e0/maven_org_apache_camel_example_sample_1_0_sample-1_0_jar:1.0: digest: sha256:b4839e4e30fa1823e5813d88764c42fc2c433011bea74e100ed310c1ab5c84ec size: 424
Uploaded: /tmp/832819540.sample-1.0.jar to image-registry.openshift-image-registry.svc:5000/test-72890e77-18e5-4691-ade9-766d603dd3e0/maven_org_apache_camel_example_sample_1_0_sample-1_0_jar:1.0 
spectrum - 2022/10/01 00:04:03 Pulling base image  (insecure=true)...
spectrum - 2022/10/01 00:04:03 Composing layers...
spectrum - 2022/10/01 00:04:03 Pushing image image-registry.openshift-image-registry.svc:5000/test-72890e77-18e5-4691-ade9-766d603dd3e0/maven_org_apache_camel_example_sample_1_0_sample-1_0_jar_md5:1.0 (insecure=true)...
spectrum - 2022/10/01 00:04:03 pushed blob: sha256:f630d69619d32d17ab4fd9a0f0c59e61d6d10eb65798d6d1cb1983da116aee7f
spectrum - 2022/10/01 00:04:03 pushed blob: sha256:9ccd1eb5bac15371c39ba63c140fc372fb262910b2f1d60fc849ccf773dd7b8d
spectrum - 2022/10/01 00:04:03 image-registry.openshift-image-registry.svc:5000/test-72890e77-18e5-4691-ade9-766d603dd3e0/maven_org_apache_camel_example_sample_1_0_sample-1_0_jar_md5:1.0: digest: sha256:280be2df8f59721e3c3400c684719d86a431780df15a3f83246f7fc4ef60c24c size: 423
spectrum - 2022/10/01 00:04:03 Pulling base image  (insecure=true)...
spectrum - 2022/10/01 00:04:03 Composing layers...
spectrum - 2022/10/01 00:04:03 Pushing image image-registry.openshift-image-registry.svc:5000/test-72890e77-18e5-4691-ade9-766d603dd3e0/maven_org_apache_camel_example_sample_1_0_sample-1_0_jar_sha1:1.0 (insecure=true)...
spectrum - 2022/10/01 00:04:03 pushed blob: sha256:0397347ccf687166570ae10c129035f4234259e42e377330a099c26c17c82cf7
spectrum - 2022/10/01 00:04:03 pushed blob: sha256:9a9ee632789b493c99ac5063559e4331b26189cf24bb706ca4f533f0aa60be12
spectrum - 2022/10/01 00:04:03 image-registry.openshift-image-registry.svc:5000/test-72890e77-18e5-4691-ade9-766d603dd3e0/maven_org_apache_camel_example_sample_1_0_sample-1_0_jar_sha1:1.0: digest: sha256:269e5ebf5ca0d6d420ee06f6ca7d1c426a073c533ea100d2a8a271e17d0915af size: 423
Integration "classpath" created

This test succeeds on kind, although the type of registry being used is different (external v internal). Can we confirm that this logging looks correct? Any paths or hostnames that may be incorrect?

Any thoughts, please comment. Thanks.

@tadayosi
Copy link
Member

tadayosi commented Oct 3, 2022

The default publishing strategy is S2I because it's on OpenShift but this feature uses Spectrum. I wonder if this combination is tested well. @johnpoth Do you know if it should work or not?

@tadayosi
Copy link
Member

tadayosi commented Oct 3, 2022

OK, so we've been avoiding running the tests that upload jars to the registry on OpenShift, which means this feature is not (yet) supported on OpenShift, right?
https://github.com/apache/camel-k/blob/main/e2e/global/registry/registry_maven_wagon_test.go#L45

@tadayosi tadayosi added the area/openshift Related to OpenShift label Oct 3, 2022
@johnpoth
Copy link
Member

johnpoth commented Oct 3, 2022

Hi @tadayosi ! Yes it works on OpenShift but you do have to extract the secret and trust it client side, so something like:

oc extract secret/router-ca --keys=tls.crt -n openshift-ingress-operator export SSL_CERT_FILE=$pwd/tls.crt

If we want to be fancy we can do it automatically in the Kamel CLI #3176

Thanks !

@johnpoth
Copy link
Member

johnpoth commented Oct 3, 2022

From @phantomjinx's log it actually fails from the Operator side so it might be that the Image registry's secret is not correctly configured during Operator Install.

To confirm we could look into the full Maven build log from the Operator

@phantomjinx
Copy link
Contributor Author

Here is the gist of the operator log. Line 2630 is the error executing the maven build.

@johnpoth
Copy link
Member

johnpoth commented Oct 3, 2022

Thanks @phantomjinx ! I think this outputs only the last Maven repository used. To see the full list we'd have to run the Maven build on debug with the -X option

@phantomjinx
Copy link
Contributor Author

Ran the test with -X. The log is 58M so have uploaded it to dropbox
The repositories that are checked in the resolution are:

{"level":"debug","ts":1664803054.9874434,"logger":"camel-k.maven.build","msg":"Resolving artifact org.apache.camel.example:sample:jar:1.0 from [apache-snapshots (https://repository.apache.org/content/repositories/snapshots, default, releases+snapshots), image-registry (docker://image-registry.openshift-image-registry.svc:5000/test-54e17631-e029-48df-8957-ed909389142c, default, releases+snapshots), central (https://repo.maven.apache.org/maven2, default, releases)]"}

It appears that no attempt is made to download the maven dependency from the image-registry.

@johnpoth
Copy link
Member

johnpoth commented Oct 3, 2022

Thanks @phantomjinx ! I thought there would be more info on debug regarding the connection to the Image Repository but it doesn't seem to be the case. Everything seems to be setup correctly so I suspect it's a cert issue but I'll have to reproduce locally to be sure. We can disable the test on OpenShift in the meantime.

phantomjinx added a commit to phantomjinx/camel-k that referenced this issue Oct 3, 2022
* Rather than trying to delete resources at the end of each sub-test, it
  is simpler and more reliable to generate a new namespace for each and
  have it deleted

* The sampleJar URL is changed for the http dependency tests to avoid the
  request have to do a redirect. This improves the reliability of its
  retrieval

* Sets the http dependency tests to problematic since on OCP4, the
  repositories are not being detected by the maven build causing test
  failures. See apache#3708.
@phantomjinx
Copy link
Contributor Author

Am wrapping up this and other changes in a set of commits that I'll post in a PR shortly.

phantomjinx added a commit to phantomjinx/camel-k that referenced this issue Oct 4, 2022
* Rather than trying to delete resources at the end of each sub-test, it
  is simpler and more reliable to generate a new namespace for each and
  have it deleted

* The sampleJar URL is changed for the http dependency tests to avoid the
  request have to do a redirect. This improves the reliability of its
  retrieval

* Sets the http dependency tests to problematic since on OCP4, the
  repositories are not being detected by the maven build causing test
  failures. See apache#3708.
phantomjinx added a commit to phantomjinx/camel-k that referenced this issue Oct 4, 2022
* Rather than trying to delete resources at the end of each sub-test, it
  is simpler and more reliable to generate a new namespace for each and
  have it deleted

* The sampleJar URL is changed for the http dependency tests to avoid the
  request have to do a redirect. This improves the reliability of its
  retrieval

* Sets the http dependency tests to problematic since on OCP4, the
  repositories are not being detected by the maven build causing test
  failures. See apache#3708.
phantomjinx added a commit to phantomjinx/camel-k that referenced this issue Oct 4, 2022
* Rather than trying to delete resources at the end of each sub-test, it
  is simpler and more reliable to generate a new namespace for each and
  have it deleted

* The sampleJar URL is changed for the http dependency tests to avoid the
  request have to do a redirect. This improves the reliability of its
  retrieval

* Sets the http dependency tests to problematic since on OCP4, the
  repositories are not being detected by the maven build causing test
  failures. See apache#3708.
phantomjinx added a commit to phantomjinx/camel-k that referenced this issue Oct 5, 2022
* Rather than trying to delete resources at the end of each sub-test, it
  is simpler and more reliable to generate a new namespace for each and
  have it deleted

* The sampleJar URL is changed for the http dependency tests to avoid the
  request have to do a redirect. This improves the reliability of its
  retrieval

* Sets the http dependency tests to problematic since on OCP4, the
  repositories are not being detected by the maven build causing test
  failures. See apache#3708.
phantomjinx added a commit to phantomjinx/camel-k that referenced this issue Oct 5, 2022
* Rather than trying to delete resources at the end of each sub-test, it
  is simpler and more reliable to generate a new namespace for each and
  have it deleted

* The sampleJar URL is changed for the http dependency tests to avoid the
  request have to do a redirect. This improves the reliability of its
  retrieval

* Sets the http dependency tests to problematic since on OCP4, the
  repositories are not being detected by the maven build causing test
  failures. See apache#3708.
tadayosi pushed a commit that referenced this issue Oct 7, 2022
* Rather than trying to delete resources at the end of each sub-test, it
  is simpler and more reliable to generate a new namespace for each and
  have it deleted

* The sampleJar URL is changed for the http dependency tests to avoid the
  request have to do a redirect. This improves the reliability of its
  retrieval

* Sets the http dependency tests to problematic since on OCP4, the
  repositories are not being detected by the maven build causing test
  failures. See #3708.
@github-actions
Copy link
Contributor

github-actions bot commented Jan 2, 2023

This issue has been automatically marked as stale due to 90 days of inactivity.
It will be closed if no further activity occurs within 15 days.
If you think that’s incorrect or the issue should never stale, please simply write any comment.
Thanks for your contributions!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/openshift Related to OpenShift status/stale
Projects
None yet
Development

No branches or pull requests

3 participants