Copy Artifact Reference Graph

[SteveLasker]

Goal

Support copying a graph of artifacts from a source to a destination registry.

Scope

oras copy ...

• source and destination registry
• source and destination authentication
• filter to a set of artifactTypes
• filter out a set of artifactTypes
• All references are copied before the tag is applied to support tag update scenarios.
• If the digest exists on the destination, the cp should continue to assure all references are copied as on may be added to the source, not yet on the destination.

While the oras cli would support the above commands, the functionality would be provided in oras-go, enabling other CLIs to benefit from the capabilities. The oras cli is simply a wrapper of the underlying library capabilities.

Promoting Artifacts

DevOps workflows account for content promotion between environments. Content may be promoted from dev through staging to production. Or, it may be promoted from a public source into a controlled environment.

Consider the guidance for Consuming Public Content, where content required for internal use is imported from a public registry. How is this different from promoting content from dev through production?

What Gets Promoted

ORAS Artifacts enables a graph of content, including signatures, SBoMs, security scan results and other content that hasn't been thought of yet.

A key value of separable artifacts is the ability to validate content, independent from where it may originate from. Before importing a container image, a signature and/or SBoM may be validated. Only after the validation is the image imported, or promoted.

When import is promoted, the entire graph of artifacts would likely be imported as the next stage of the promotion should be able to validate independent of any previous steps.

At the same time, not all referenced artifacts may be desired. A category of artifactTypes or specific artifacts within a category may need to be filtered out, or filtered in.

1. Import the nginx image from docker hub, including the associated SBoMs and signatures. However, the released notes, gpl source or license info may not be wanted.
2. Promote the nginx image from staging through production. The SBoM, Scan Results and upstream signatures have all been pre-validated. The only referenced artifact required in production is the ACME Rockets signature.

Promoting Content Walkthrough

• Copy with all references

    oras login registry.acme-rockets.io -u $USER -p $PASSWORD
    oras cp registry.wabbit-networks.io/net-monitor:v1 \
      registry.acme-rockets.io/library/net-monitor:v1 \
      -r

• Copy with specific references, included

  oras cp registry.wabbit-networks.io/net-monitor:v1 \
    registry.acme-rockets.io/library/net-monitor:v1 \
    -r \
    -m application/vnd.cncf.notary.v2 \
    -m sbom/example

• Copy with specific references, excluded

  oras cp registry.wabbit-networks.io/net-monitor:v1 \
    --destination registry.acme-rockets.io/library/net-monitor:v1 \
    -r \
    -x sbom/example

• Copy with specific references, included, filtered by annotation (inclusion)

  oras cp registry.wabbit-networks.io/net-monitor:v1 \
    -r \
    -m application/vnd.cncf.notary.v2 \
    --match-annotation "org.cncf.notary.v2.signature.subject=acme-rockets.io"

• Copy with specific references, included, filtered by annotation (exclusion)

  oras cp registry.wabbit-networks.io/net-monitor:v1 \
    -r \
    -m application/vnd.cncf.notary.v2 \
    --exclude-annotation "org.cncf.notary.v2.signature.subject=wabbit-networks.io"

Options

# login to the destination registry
oras login [destination] -u $USER -p $PASSWORD

# copy just the image 
oras cp source destination

# copy all references
oras cp -r source destination  

# image and sboms
oras cp -r --match sbom/example source destination  

# image and sboms
oras cp -r -m *sbom* -x *non-distributable* source destination  

-f, --force                  Overwrite the tag incase the tag
                            exists in the destination

[SteveLasker]

Note: I added a comment around tag update scenarios

[junczhu]

I will help with this

[shizhMSFT]

@CharingJC You may start with basic functionalities similar to #391

[shizhMSFT]

Advanced scenarios depend on oras-project/oras-go#201 and oras-project/oras-go#203.

[sajayantony]

I would recommend limiting the number of references of that get copied.
for e.g. --max-referrs-per-artifact 10
Also filter by artifacts types --include-referrer-type $TYPE1, $TYPE2

If we are deep copying referrers then we need to make sure that we also have --max-depth 2 which could be the default.

[SteveLasker]

If we compared this to a file system folder copy, would a user expect only the first n files, or only x folders deep to be copied?

What if we made the copy cancellable, just like a folder copy?
As oras cp is copying artifacts, it outputs

oras cp registry.wabbit-networks.io/net-monitor:v1 registry.acme-rockets.io/net-monitor:v1
copying tag: [tag], digest: abc123, artifactType: subject: def456
on [x] of [y] artifacts

If the user cancells the operation, we simply stop where it left off.
The question would be:

• Do we need to calculate the number of manifests before starting to copy? This could be time intensive, slowing the beginning of the copy
• If the user runs again, and the manifests and blobs already exist, does cp just proceed? Is there a reason to ask the user to copy what we know hasn't changed?

For filtering, we should tease out the scenarios to a phase 2. Would a user really want to filter, just by artifactType? I could see wanting to copy the last n versions of a scan result, based on the created date. Or, possibly the first. Or, specific notary signatures, like acme-rockects, filtering out wabbit-networks signatures. But, this detail is likely in the annotations. Which means filtering would need to include the artifactType and annotations. For annotations like created, we'd need to provide asc and desc.
So, just suggesting we start with a cancellable copy, then add more as we have more usage feedback.

[FeynmanZhou]

If we compared this to a file system folder copy, would a user expect only the first n files, or only x folders deep to be copied?

What if we made the copy cancellable, just like a folder copy? As oras cp is copying artifacts, it outputs

oras cp registry.wabbit-networks.io/net-monitor:v1 registry.acme-rockets.io/net-monitor:v1
copying tag: [tag], digest: abc123, artifactType: subject: def456
on [x] of [y] artifacts

If the user cancells the operation, we simply stop where it left off. The question would be:

• Do we need to calculate the number of manifests before starting to copy? This could be time intensive, slowing the beginning of the copy
• If the user runs again, and the manifests and blobs already exist, does cp just proceed? Is there a reason to ask the user to copy what we know hasn't changed?

For filtering, we should tease out the scenarios to a phase 2. Would a user really want to filter, just by artifactType? I could see wanting to copy the last n versions of a scan result, based on the created date. Or, possibly the first. Or, specific notary signatures, like acme-rockects, filtering out wabbit-networks signatures. But, this detail is likely in the annotations. Which means filtering would need to include the artifactType and annotations. For annotations like created, we'd need to provide asc and desc. So, just suggesting we start with a cancellable copy, then add more as we have more usage feedback.

@SteveLasker To support cancellable copy of artifacts like file system mechanism sounds useful. I think we can turn it into a GitHub discussion and listen to more users' feedback. This feature might be nice to have but not very urgent for basic scenarios.

[SteveLasker]

This feature might be nice to have but not very urgent for basic scenarios.

Agreed. What I'd suggest is recognizing this is more likely the direction we'd go, that we don't overly constrain the depth or quantity, which could actually create instability as users won't understand why some references were truncated when copying.

To this end, I'd suggest we might limit to 50 total references.

What is the current behavior of the prototype? If a user starts a copy, and hits cancel, what happens?