Switch to using OCI Artifacts via build-trusted-artifacts rather than Jib #1824

rnc · 2024-06-13T07:42:58Z

As https://github.com/GoogleContainerTools/jib doesn't preserve symbolic links this switches to https://github.com/konflux-ci/build-trusted-artifacts/ which is based upon https://oras.land/

Requires #1842 merged first.

openshift-ci · 2024-06-13T07:43:02Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

codecov · 2024-06-13T07:45:51Z

Codecov Report

Attention: Patch coverage is 77.98165% with 48 lines in your changes missing coverage. Please review.

Project coverage is 39.84%. Comparing base (c81fb44) to head (5974af2).
Report is 4 commits behind head on main.

Files	Patch %	Lines
pkg/reconciler/dependencybuild/buildrecipeyaml.go	84.61%	20 Missing and 8 partials ⚠️
...s/container/deploy/DeployPreBuildImageCommand.java	0.00%	10 Missing ⚠️
...edhat/hacbs/container/deploy/TagDeployCommand.java	36.36%	6 Missing and 1 partial ⚠️
...hat/hacbs/container/deploy/BuildVerifyCommand.java	0.00%	2 Missing ⚠️
...cbs/common/images/ociclient/OCIRegistryClient.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1824      +/-   ##
============================================
- Coverage     40.37%   39.84%   -0.54%     
+ Complexity      813      786      -27     
============================================
  Files           308      308              
  Lines         14225    14237      +12     
  Branches       1451     1445       -6     
============================================
- Hits           5744     5673      -71     
- Misses         7817     7918     +101     
+ Partials        664      646      -18

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

stuartwdouglas · 2024-06-18T06:51:32Z

pkg/reconciler/dependencybuild/scripts/Dockerfile.build-trusted-artifacts

@@ -0,0 +1,3 @@
+#This file is used to enable renovate to update the digests. It is updated by renovate then substituted into the golang code.
+
+FROM quay.io/redhat-user-workloads/rhtap-build-tenant/trusted-artifacts/trusted-artifacts@sha256:9e2ffee0cb28f8a0ed7895a357c73f006005b26ef143f00df067f090282e8cbd


This should really be provided by config, rather than being baked into the binary. This is fine for now, but we should look at this later, as we will need a general solution for Konflux tasks we are referencing.

Sure. I did consider adding to e.g. system-config.yaml. I'm not actually renovate will catch this (but equally I know we have to update the builder images manually which is not ideal). I don't know how Konflux keeps the references up to date.

I think we can configure renovate to handle this. Konflux runs renovate manually to update references, its run by the build service as a scheduled task.

stuartwdouglas · 2024-06-20T05:33:27Z

The minikube tests are failing with:

Creating pre-build-image archive
jq: error: Could not open file /root/.docker/config.json: No such file or directory
jq: error (at <stdin>:1): null (null) and null (null) cannot be multiplied

rnc · 2024-06-21T10:19:38Z

...s/build-request-processor/src/main/java/com/redhat/hacbs/container/deploy/DeployCommand.java

 import io.quarkus.logging.Log;
 import picocli.CommandLine;

-@SuppressWarnings("OptionalUsedAsFieldOrParameterType")
 @CommandLine.Command(name = "deploy")
 public class DeployCommand implements Runnable {



I renamed MavenDeployCommand to TagDeployCommand and the original DeployCommand to BuildVerifyCommand.

rnc · 2024-06-21T10:22:51Z

pkg/reconciler/dependencybuild/buildrecipeyaml.go

@@ -338,22 +356,26 @@ func createPipelineSpec(log logr.Logger, tool string, commitTime int64, jbsConfi
 			{Name: PipelineResultImageDigest},
 			{Name: PipelineResultPassedVerification},
 			{Name: PipelineResultVerificationResult},
-			{Name: PipelineResultGitArchive},
+			// TODO: ### DeployPreBuildSource and Deploy push to git. Currently the former is used for GitArchive results.
+			//			{Name: PipelineResultGitArchive},


Previously PipelineResultGitArchive was storing the result of Deploy's git push while DeployPreBuildSource git push was 'lost'. Currently PipelineResultGitArchive is now storing the result of DeployPreBuildSource. I haven't (yet) added another pipeline result for the Deploy pipeline to also store the result of its git archiving - do we want to add that (or perhaps in a subsequent PR) ?

I think this can come later. It is not 100% clear if we will be pushing to git at all, or if that will come from repour/PNC.

pkg/reconciler/dependencybuild/buildrecipeyaml.go

rnc · 2024-06-21T16:03:10Z

Current status - there is an issue with rebuilding contaminants ; @stuartwdouglas I suspect moving the Deploy around may have affected it. Perhaps we can sync in order check how its meant to work?

rnc · 2024-06-24T17:10:22Z

...cache/src/test/java/com/redhat/hacbs/artifactcache/ContainerRegistryTestResourceManager.java

+        // HACK : OCIRepositoryClient assumes that the artifacts are in a directory and it then places them
+        //        within 'artifacts/...'. This is due to build-trusted-artifact changes as its storage stores
+        //        the contents of a directory not including the directory itself.
+        return List.of(layer1Path, layer2Path, layer3Path.resolve("com"));


Not overly fond of this, but the current test doesn't match how build-trusted-artifacts/oras stores (it stores the contents of a directory versus the directory itself) : https://github.com/konflux-ci/build-trusted-artifacts/blob/main/create-oci.sh#L84

rnc · 2024-06-24T17:12:11Z

pkg/reconciler/dependencybuild/buildrecipeyaml.go

+	// using AUTHFILE to override. Setting ORAS_OPTIONS to ensure the archive is compatible with jib (for OCIRepositoryClient).
+	preBuildImageArgs := fmt.Sprintf(`echo "Creating pre-build-image archive"
+echo $REGISTRY_TOKEN > ~/config.json
+export ORAS_OPTIONS="%s --image-spec=v1.0 --artifact-type application/vnd.oci.image.config.v1+json"


Ensures that the OCIRegistry can read the archives.

…REGISTRY_TOKEN and port

…gistry_token. Fix prependTagToImage.

…t. Fix OCI handling to add directory. Fix e2e tests. Add prependTag to GAV list.

openshift-ci bot added the do-not-merge/work-in-progress label Jun 13, 2024

rnc mentioned this pull request Jun 13, 2024

Add oras and build-trusted-artifact scripts redhat-appstudio/jvm-build-service-builder-images#78

Closed

rnc force-pushed the OCI branch 2 times, most recently from 8b9dd39 to a358181 Compare June 14, 2024 14:38

rnc requested a review from stuartwdouglas June 14, 2024 16:02

rnc changed the title ~~Initial attempt to use OCI Artifacts~~ Switch to using OCI Artifacts via build-trusted-artifacts rather than Jib Jun 14, 2024

rnc force-pushed the OCI branch 2 times, most recently from 43f3c94 to 6c44b93 Compare June 18, 2024 06:20

stuartwdouglas reviewed Jun 18, 2024

View reviewed changes

rnc force-pushed the OCI branch 5 times, most recently from 05c383f to fe857dd Compare June 19, 2024 16:01

rnc force-pushed the OCI branch 7 times, most recently from 32ffbb9 to 86ba642 Compare June 21, 2024 10:00

rnc marked this pull request as ready for review June 21, 2024 10:00

openshift-ci bot removed the do-not-merge/work-in-progress label Jun 21, 2024

openshift-ci bot requested a review from stuartwdouglas June 21, 2024 10:01

rnc commented Jun 21, 2024

View reviewed changes

pkg/reconciler/dependencybuild/buildrecipeyaml.go Outdated Show resolved Hide resolved

rnc force-pushed the OCI branch 2 times, most recently from 5fb7ec4 to 2ba5dab Compare June 21, 2024 15:15

rnc force-pushed the OCI branch 3 times, most recently from f0aa740 to 6027568 Compare June 24, 2024 16:02

rnc commented Jun 24, 2024

View reviewed changes

rnc force-pushed the OCI branch 3 times, most recently from e27be9a to 5720c25 Compare June 25, 2024 09:35

rnc added 14 commits June 25, 2024 12:01

Initial attempt to use OCI Artifacts

fe17d56

Use oras for deploy pipeline. More code refactoring

b3ed7c1

Remove more code and fix test

8ecc254

Update deploy pipeline tag step to use oras

446a6d8

Update build-trusted-artifacts to use new AUTHFILE variable

f5703b9

Fix pullPolicy

9212e69

Add oci artifact support for hermetic builds

3948a36

Restore source. Rename imageRegistryCommands

f2d487a

Move final git push to deploy pipeline

8c94f70

Fixes and rename classes.

ef3504d

Don't merge REGISTRY_TOKEN with preexisting configuration. Fix empty …

d68d6fc

…REGISTRY_TOKEN and port

Add ORAS_OPTIONS. Update to latest trusted-artifacts. Handle empty re…

a8d8802

…gistry_token. Fix prependTagToImage.

Fix artifact storage order. Change oras push output to retrieve diges…

2a0b88e

…t. Fix OCI handling to add directory. Fix e2e tests. Add prependTag to GAV list.

Remove debug

5974af2

rnc force-pushed the OCI branch from 5720c25 to 5974af2 Compare June 25, 2024 11:01

stuartwdouglas merged commit 56caca8 into redhat-appstudio:main Jun 26, 2024
24 checks passed

rh-tap-build-team bot mentioned this pull request Jun 26, 2024

jvm-build-service update redhat-appstudio/infra-deployments#3948

Closed

rnc deleted the OCI branch June 26, 2024 07:55

rnc mentioned this pull request Jun 27, 2024

Add jvm-build-image-secrets to pipeline to avoid using REGISTRY_TOKEN #1892

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to using OCI Artifacts via build-trusted-artifacts rather than Jib #1824

Switch to using OCI Artifacts via build-trusted-artifacts rather than Jib #1824

rnc commented Jun 13, 2024 •

edited

Loading

openshift-ci bot commented Jun 13, 2024

codecov bot commented Jun 13, 2024 •

edited

Loading

stuartwdouglas Jun 18, 2024

rnc Jun 18, 2024

stuartwdouglas Jun 18, 2024

stuartwdouglas commented Jun 20, 2024

rnc Jun 21, 2024 •

edited

Loading

rnc Jun 21, 2024 •

edited

Loading

stuartwdouglas Jun 25, 2024

rnc commented Jun 21, 2024 •

edited

Loading

rnc Jun 24, 2024 •

edited

Loading

rnc Jun 24, 2024

		@@ -0,0 +1,3 @@
		#This file is used to enable renovate to update the digests. It is updated by renovate then substituted into the golang code.

		FROM quay.io/redhat-user-workloads/rhtap-build-tenant/trusted-artifacts/trusted-artifacts@sha256:9e2ffee0cb28f8a0ed7895a357c73f006005b26ef143f00df067f090282e8cbd

Switch to using OCI Artifacts via build-trusted-artifacts rather than Jib #1824

Switch to using OCI Artifacts via build-trusted-artifacts rather than Jib #1824

Conversation

rnc commented Jun 13, 2024 • edited Loading

openshift-ci bot commented Jun 13, 2024

codecov bot commented Jun 13, 2024 • edited Loading

Codecov Report

stuartwdouglas Jun 18, 2024

Choose a reason for hiding this comment

rnc Jun 18, 2024

Choose a reason for hiding this comment

stuartwdouglas Jun 18, 2024

Choose a reason for hiding this comment

stuartwdouglas commented Jun 20, 2024

rnc Jun 21, 2024 • edited Loading

Choose a reason for hiding this comment

rnc Jun 21, 2024 • edited Loading

Choose a reason for hiding this comment

stuartwdouglas Jun 25, 2024

Choose a reason for hiding this comment

rnc commented Jun 21, 2024 • edited Loading

rnc Jun 24, 2024 • edited Loading

Choose a reason for hiding this comment

rnc Jun 24, 2024

Choose a reason for hiding this comment

rnc commented Jun 13, 2024 •

edited

Loading

codecov bot commented Jun 13, 2024 •

edited

Loading

rnc Jun 21, 2024 •

edited

Loading

rnc Jun 21, 2024 •

edited

Loading

rnc commented Jun 21, 2024 •

edited

Loading

rnc Jun 24, 2024 •

edited

Loading