Designing ORAS

[shizhMSFT]

Introduction

ORAS is a tool for working with OCI artifacts, especially ones in the OCI registries. It provides user-friendly interfaces to push and pull artifacts to / from registries as well as options for advanced scenarios.

Unified Experience

The objective of ORAS is simple as transferring artifacts from one place to another.

In the conventional client-server model, the operation of downloading artifacts from the remote registries is referred to as pull, and the operation of uploading artifacts from the remote registry is referred to as push.

This model can be generalized by abstracting the client and the server as targets so that pull and push can be viewed as copying from one target to another (see #8). For instances,

• Copy from memory to a remote registry.
• Copy from a remote registry to a local file folder.
• Copy from a remote registry to another remote registry.
• Copy from memory to a local file folder.

Targets

Generally, a target is a content-addressable storage (CAS) with tags. All blobs in a CAS are addressed by their descriptors.

To retrieve a blob,

1. Get a descriptor. Optionally, it can be resolved by a tag.
2. Query the blob with a descriptor.

To store a blob,

1. Store the blob directly in the CAS. A descriptor will be returned.
2. Optionally, associate the returned descriptor with a tag.

It is worth noting that a target is not equal to a registry.

• Blobs can be tagged in a target but not in a registry.
• Tag list is available in a registry but not always available in a target.

Graphs

Besides plain blobs, it is natural to store directed acyclic graphs (DAGs) in a CAS. Precisely, all blobs are leaf nodes and most manifests are non-leaf nodes.

An artifact is a rooted DAG where its root node is an OCI manifest. Additionally, artifacts can be grouped by an OCI index, which is also a rooted DAG.

Given a node of a DAG in a CAS, it is efficient to find out all its children. Since CASs are usually not enumerable or indexed, it is not possible to find the parent nodes of an arbitrary node. Nevertheless, some CASs choose to implement or partially implement the functionality of parent node finding. For instances, registries with Manifest Referrers API support are CASs with partially implementation where parent node finding is only available for manifest nodes.

Extended Copy

With the concepts above, we can formally define that

• Copy is a function to replicate a rooted DAG from one CAS to another.
• Extended Copy is a function to replicate a DAG from one CAS to another.

It is worth noting that extended copy is possible only if the source CAS supports parent node finding. Based on the scenarios, extended copy can have many options such as opting to copy a sub-DAG rooted by a certain node and all its parent nodes of a certain depth with / without their children.

Optionally, node filters or even node modifiers can be attached to a copy process for advanced scenarios.

Related issues:

• Support copy of images and associated references #29
• Copy Artifact Reference Graph oras#307

Hint: A polytree is a DAG.

Challenges

We have lots of challenges in developing and maintaining ORAS:

1. Where should this documentation be stored?
   • oras-www looks like a good place but what is the exact path?
2. The oras-go library should have as less dependencies as possible.
   • At least, we need to decouple containerd as we just need a little bit of that project.
   • We can define brand new interfaces suitable for CASs with tags.
   • Then we can provide adapters for containerd related implementation in the examples folder.
3. How file systems can be mapped into a CAS.
   • Currently, files are mapped as blobs with metadata stripped and directories are compressed into gzipped tarballs. Then a rooted DAG is generated based on above blobs.
   • There is an on-going discussion Revisit the Design of ORAS oras#304 about improving the file system mapping while backward compatibility is preserved.

[shizhMSFT]

I just noticed that I can put this doc into the Wiki page of the oras-go repo.

[sajayantony]

I wanted to make sure were have an API spec for this - Do we need to remove ProviderWrapper types since they will have to be non-breaking if we release them is my understanding

/cc @deitch

func Copy(ctx context.Context, from target.Target, fromRef string, to target.Target, toRef string, opts ...CopyOpt) (ocispec.Descriptor, error)
func ValidateNameAsPath(desc ocispec.Descriptor) error
func WithPullByBFS(o *copyOpts) error
type CopyOpt
    func WithAdditionalCachedMediaTypes(cachedMediaTypes ...string) CopyOpt
    func WithAllowedMediaType(allowedMediaTypes ...string) CopyOpt
    func WithAllowedMediaTypes(allowedMediaTypes []string) CopyOpt
    func WithContentStore(store orascontent.Store) CopyOpt
    func WithLayerDescriptors(save func([]ocispec.Descriptor)) CopyOpt
    func WithNameValidation(validate func(desc ocispec.Descriptor) error) CopyOpt
    func WithPullBaseHandler(handlers ...images.Handler) CopyOpt
    func WithPullCallbackHandler(handlers ...images.Handler) CopyOpt
    func WithPullEmptyNameAllowed() CopyOpt
    func WithPullStatusTrack(writer io.Writer) CopyOpt
    func WithRootManifest(save func(b []byte)) CopyOpt
    func WithUserAgent(agent string) CopyOpt
type ProviderWrapper
    func (p *ProviderWrapper) ReaderAt(ctx context.Context, desc ocispec.Descriptor) (content.ReaderAt, error)

[sajayantony]

Looking through this more and checkout out the types that we are exposing. What all should we trim out?

I am not sure if this is my lack of awareness but the content package seems like it exposing a large number of types that might just be better internal.