Support OCI images in API

[vmarmol]

Today the API assumes Docker images, but we'd like to be able to support other images in the future. This issue specifically targets the ACI standard (which is used by Rkt)

Arguably having the flexibility to have a different image type in the API is for v1.0, although actually supporting a different image type is not.

/cc @dchen1107 @bgrant0607 @yifan-gu @jonboulle

[thockin]

We have to decide whether the API should reflect the choice of runtime at
all. Ideally the API is agnostic to all that. Container image is just a
string that we pass down. I'm not sure we can do that, though. I really
dislike the idea of the end user having to change their behavior based on
the admin's choice of runtime.

It's a kubernetic ideal that the dame pod spec should run on a docker
cluster and on a rocket cluster.

On Wed, Apr 22, 2015 at 5:20 PM, Victor Marmol notifications@github.com
wrote:

Today the API assumes Docker images, but we'd like to be able to support
other images in the future. This issue specifically targets the ACI
https://github.com/appc/spec/blob/master/SPEC.md#app-container-image
standard (which is used by Rkt https://github.com/coreos/rkt)

Arguably having the flexibility to have a different image type in the API
is for v1.0, although actually supporting a different image type is not.

/cc @dchen1107 https://github.com/dchen1107 @bgrant0607
https://github.com/bgrant0607 @yifan-gu https://github.com/yifan-gu
@jonboulle https://github.com/jonboulle

—
Reply to this email directly or view it on GitHub
#7203.

[bgrant0607]

Other example runtimes / formats:

• Fabric8 Java
• LXC + some image format (targz?)
• Google internal
• Non-Linux platforms: BSD, Solaris, Windows, ...

Eventually we'll need some kind of plugin API in the pod spec.

Ideally, when running in a cloud, the system would eventually be able to instantiate VMs capable of running whatever image/spec was provided, as necessary.

I agree that differences in semantics should not be implicit based on implementation choices, and the pod spec should not need to be changed in the case that the same semantics are desired on different runtimes.

[thockin]

By way of example, I would be very against a solution that required

image: nginx

on docker runtimes but required

image: docker://nginx

on rkt

On Wed, Apr 22, 2015 at 10:03 PM, Brian Grant notifications@github.com
wrote:

Other example runtimes / formats:

• Fabric8 Java
• LXC + some image format (targz?)
• Google internal
• Non-Linux platforms: BSD, Solaris, Windows, ...

Eventually we'll need some kind of plugin API in the pod spec.

Ideally, when running in a cloud, the system would eventually be able to
instantiate VMs capable of running whatever image/spec was provided, as
necessary.

I agree that differences in semantics should not be implicit based on
implementation choices, and the pod spec should not need to be changed in
the case that the same semantics are desired on different runtimes.

—
Reply to this email directly or view it on GitHub
#7203 (comment)
.

[bgrant0607]

Agree, though requiring a prefix for all images would be a breaking change. We could do that in v1 and add a format field in the older APIs.

However, with respect to images, there are multiple issues:

• name/address format
• data format
• image identity, both mutable (tags) and immutable (content-based id)
• location to fetch from

I haven't had time to think about which we want to explicitly represent. In enterprise environments, admins generally want control over the latter 2.

[vmarmol]

In the short term, ACI-based images do natively support Docker images so no changes would need to be made to express Docker images. For others we would have to identify and know where to schedule them. Initially it'd be an explicit choice on the user's part I imagine.

If we did have to schedule only on nodes that supported format X, parsing the image string is probably not the best way.

[jonboulle]

@vmarmol:

ACI-based images do natively support Docker images

"ACI-based images" -> "ACI-based runtimes"? Even so, not strictly correct; rkt natively supports Docker images (by transparently converting them to ACIs using docker2aci), but this isn't a provision of the spec (i.e. other implementations might not have this support).

[jonboulle]

A little more detail on the ACI side:

It is definitely a different beast to Docker in that there's no shorthand way to deterministically [1] reference remote images (in part since there's no registry API). Instead, the appc spec currently defines two processes (strategies) for discovering image locations [2], where "location" is an HTTP endpoint, torrent, HDFS URI, etc; but these strategies are not necessarily deterministic, depending on their implementation (essentially this is the "mutable" identity bgrant0607 refers to above). The only canonical image reference is instead the image ID (the immutable, content-based identity).

Taking rkt as an example again, it currently supports three image reference types:
a) a discovery reference
b) a direct URI (either HTTP/S or a file on local disk)
c) an image ID (at the moment this must be already locally cached in rkt's CAS, but you can imagine an implementation where it is available in a well-known remote CAS as well)

[1] For some definition of deterministic; obviously Docker repositories can still be mutated behind the scenes. But there is a general assumption in the ecosystem today that "nginx:14.04" or "foo.com/bar:baz" refer to a consistent image over time. (I know Docker have just added content-based tags to provide this)
[2] Two ordered processes, but soon to be reframed as a pluggable set of strategies

[philips]

I think where we need to end up on this is having the API be very explicit about the users intended image. This is something like how the volume API is explicit about the type of volume and its arguments:

  manifest:
    containers:
      - image: kubernetes/pause
        name: testpd
        volumeMounts:
          - mountPath: "/testpd"
            name: "testpd"
    id: testpd
    version: v1beta1
    volumes:
      - name: testpd
        source:
          persistentDisk:
            # This GCE PD must already exist.
            pdName: test
            fsType: ext4

So if we take this same approach and change the image we get something like:

docker images

  manifest:
    containers:
      - name: testpd
        image:
            docker:
                repository: kubernetes/pause
                tag: latest
                digest: sha256:cbbf2...
        volumeMounts:
          - mountPath: "/testpd"
            name: "testpd"

ACIs

  manifest:
    containers:
      - name: testpd
        image:
            aci:
                name: kubernetes.io/pause
                labels:
                   version: 2.0.9
                id: sha512-a83b3f...
        volumeMounts:
          - mountPath: "/testpd"
            name: "testpd"

Possible problems and points of research:

• Docker has other things like os and arch which need to be accounted for https://github.com/docker/docker/blob/master/image/spec/v1.md#image-json-field-descriptions
• appc implementations often have default labels for runtimes like arch and os, do we let those be implied?

[philips]

On @bgrant0607 point of letting the user control where things are fetched from this feels like a configuration problem that is orthogonal to this problem. If the user wants to configure their container runtime to fetch from an NFS share, local store, etc we need to have a cluster-wide or namespace wide API for this sort of thing.