Allow exposing status.containerStatuses[*].imageID through Downward API

[mitar]

What would you like to be added:

It would be great if status.containerStatuses[0].imageID and status.containerStatuses[0].image could be exposed through Downward API and passed as an environment variable. That would allow the container to obtain information about which exactly digest of the image it is running, which can then be useful for any logging from that container.

Why is this needed:

It is useful for the container to know which exactly version of the image it is running. By allowing passing the image and digest/ID to the container, this is achievable.

[mitar]

/sig node

[BenTheElder]

@kubernetes/sig-node-feature-requests

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[mitar]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[mitar]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[mitar]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[mitar]

/remove-lifecycle stale

[BenTheElder]

/lifecycle frozen
Not actually speaking for SIG Node but: This might require raising to them via some other means / someone stepping up to do the work. I don't think they have enough active contributor bandwidth to do feature requests these days. (Most successful FR also seem to be KEPs now, for better or worse ...)

[mitar]

Thanks. So by "stepping up to do the work" you mean making a KEP or implementing this? So is the idea is to just go with this and make a PR with code or first KEP? So I was hoping that somebody would officially approve this (it seems pretty simple change) and then community could provide a PR.

[BenTheElder]

Implementing.

Though to get it approved might? require a KEP.
probably at least interacting with the SIG directly (slack or mailinglist at minimum. more likely getting it on their meeting agenda and discussing).

I don't really work in SIG node, but that's what I'd expect to have to get a feature through myself, unfortunately.
I'm pretty sure I dropped a link to this before in slack, but at the time they were pretty under-staffed.

[mitar]

Thanks. I added it to the meeting agenda for 9/22/2020.

[alejandrox1]

Overall question for this one: could you enumerate why this info may be useful? What kind of workloads are you running or wanting to run?

[sjenning]

I am not against this. It can be easily done today by injecting the image as an envvar.

spec:
  containers:
  - name: envar-demo-container
    image: ${value}
    env:
    - name: IMAGE
      value: ${value}

https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/#capabilities-of-the-downward-api

I would be against pulling the value from the status and not the spec. There could be race here if downwardapi volumes/envvars are populated before the pod status.

I understand that the status will have the image digest vs the spec which might pull by tag, but if you want the digest, you should specify the image pull path by digest.

[mitar]

Thank you for comments and questions.

Overall question for this one: could you enumerate why this info may be useful?

For me it is really about container introspection. Container being able to know some information about itself. This is what downward API provides anyway. And I just seek to expand it a bit to provide some other extra information which is otherwise hard to obtain: what image exactly is it being run.

I see/have few use cases:

• Being able to report to the end-user a digest of the image of the service they just hit in the case of the error, or alternatively log that. That would then allow me to be able to re-run the exact same Docker image and try to reproduce the issue easier.
• To help with scientific reproducibility. I have a 10 GB Docker images with many many dependencies which are build daily. Users run ML jobs using them. I want to be able to record exactly which version of the image was used so that if there is a need to retrain/verify/reproduce results, it is easy to fetch exactly that version of the Docker image. Digest of the Docker image is simply the easiest way to describe all the dependencies: system packages, Python packages, etc. (There is also hardware which can influence the results, but that we collect differently.)

this proposal I think will need to account for this issue of mismatch of tags

Not really. This proposal would be exposing the hash/digest version of the image ID, so confusion over tags for that image ID available on the host do not apply.

It can be easily done today by injecting

Same hold for all downward API, no? I could also copy in the same way the resource limits to ENV section. But this approach means that you have to use some additional templating language around pod files. And can be error prone if templating language is not being used but somebody forgets to update it in one place but not another. Moreover, it is not possible in all cases, see below.

I understand that the status will have the image digest vs the spec which might pull by tag, but if you want the digest, you should specify the image pull path by digest.

This is exactly the use case I want. I want to know the digest while keeping the non-digest image tag. So non-digest image tag includes something like registry.gitlab.com/datadrivendiscovery/images/primitives:ubuntu-bionic-python36-stable and then there might be different versions of this image/tag through time. I want to know which digest of that image ended up running. Users might not care about which exactly (latest) version of the image they used in the job they queued, but the container itself should know it so that it can log that information into our metalearning database, so if somebody wants to reproduce the job, they have information of exactly the Docker image used available.

This is also the reason why env variable workaround does not work. That works only if you specify the digest in advance. But in decoupled system where somebody is building an image, and you have imagePullPolicy: Always there is no way to know what is really running, except if the Kubernetes itself provides that.

There could be race here if downwardapi volumes/envvars are populated before the pod status.

Exactly. This is why this race should be addressed and made sure this does not happen. This can be done by making sure Kubernetes first fetch the image given the tag (imagePullPolicy: Always), do docker inspect or equivalent on the image to obtain image ID, and then use the image ID to run the container, not the original tag, and optionally pass it in as environment variable (if configured like that using downward API).

but if you want the digest, you should specify the image pull path by digest.

That prevents the use case where user (job submitter) does not care about particular Docker image version (wants the latest), but for debugging/logging/reproducibility purposes you do care what was picked up at the end. So the pod spec contains non-digest version of the image, with imagePullPolicy: Always set, but the container should then be able to introspect/obtain this information.

[mitar]

@timhowes, @mpoqq, @Robpol86, @surenraju, @alexeyzimarev, @alonrbar, @lukinko, @Skonx, @jtackaberry, @yhpark, @CharlyF, @adwin5, @seamusabshere, @hicolour, @daniel-pp, @haizaar, @lopf, @shiwano, you all upvotes this issue. I am working on KEP for it. Would you share your use cases which made you upvote the issue? I could then try to include them in the KEP.

[mitar]

I started working on the KEP here: kubernetes/enhancements#2013

[shiwano]

We are using Spinnaker for deploying containers, and injecting the image tag to env by using Spinnaker Pipeline Expression like below.

env:
- name: CONTAINER_IMAGE_VERSION
  value: '${ trigger.artifacts.?[type == "docker/image"][0].version }'

For example, we get the image tag like 1600784656-master-8abc17e5f6cd by the config above, and output running container version to Stackdriver logging.

I think it is great if Kubernetes will support this, because we will not need to depends on Spinnaker’s difficult feature.

[mqsoh]

I think it should be supported in the downward API, but if you're using Kustomize, this will work.

kustomization.yml

vars:
- name: DOCKER_IMAGE
  objref:
    apiVersion: apps/v1
    kind: Deployment
    name: mydeployment
  fieldref:
    fieldpath: spec.template.spec.containers[0].image

mydeployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: mydeployment
  name: mydeployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mydeployment
  template:
    metadata:
      labels:
        app: mydeployment
    spec:
      containers:
      - image: foobar
        name: foobar
        env:
        - name: DOCKER_IMAGE
          value: $(DOCKER_IMAGE)

[mitar]

It is not possible to work to get digest of the image as it was pulled by Kubernetes.

[justicezyx]

My use case:

A job launcher replicate itself with different roles, needs to know the image digest to ensure the same version of image being used.

Tag would not work because potential stale image on the nodes.

Workarounds include:

• User inject digest in envvar
• User inject unique tag in envvar

As already mentioned above these workaround fails in typical production environment, because the consumer and builder of the image are often separate teams, in separate system, and subject to complicated multitenant cluster, etc.

[cfiderer]

Similar use case: an operator that needs to run its own image as a job for longer-running maintenance tasks.

[lpgc]

just having the image id and container id is essential for logging and tracing... this should be specified at the r/t level such that any OCI conformant environment expressed this information in a std fashion ... irrespective of the container mgmt environment in which it is executing...

[swatisehgal]

/triage accepted
/priority important-longterm

[n-branigan]

Has any other way to access container image/imageid from within a container come up or is this still an open issue?

[lpgc]

Has any other way to access container image/imageid from within a container come up or is this still an open issue?

not to the best of my knowledge, I have been looking at perhaps a workaround whereby using 'events' determining the process id of the container and then correlating that with the metadata available in /proc ... but this is to enable an external process to introspect on a running container, and not for a process within a running container to obtain either its image or container ids ...

[cfiderer]

My hackaround:

• Read the pod name and namespace from Downward API, then get the pod object from the K8s Client API.
• Extract further information from the pod status.
• Requires get;list;watch access to v1/pods.

Not nice, but it works... 😒

[n-branigan]

I'm actually trying to do something in a multi container pod so just knowing the pod name doesn't seem like enough to ascertain which container I'm in (without knowing something about what the images have in them)

[lpgc]

IMO this needs to be solved not only for k8s but for any OCI compliant r/t, not sure why its such a challenge and has languished as per: https://github.com/opencontainers/runtime-spec/issues/1105

[larry-cable]

What would you like to be added:

It would be great if status.containerStatuses[0].imageID and status.containerStatuses[0].image could be exposed through Downward API and passed as an environment variable. That would allow the container to obtain information about which exactly digest of the image it is running, which can then be useful for any logging from that container.

Why is this needed:

It is useful for the container to know which exactly version of the image it is running. By allowing passing the image and digest/ID to the container, this is achievable.

while its great that there is movement on this, I personally do not think that a kubernetes only solution is satisfactory., there are other container orchestration engines that are responsible for container l/c these engines should also expose
image and container id to their managed instances in a portable fashion.

[notcool11]

I'd like this expanded to provide everything for the OpenTelemetry container spec. Each otel vendor does data correlation differently, so best to have all the data to be certain it works.

[larry-cable]

I'd like this expanded to provide everything for the OpenTelemetry container spec. Each otel vendor does data correlation differently, so best to have all the data to be certain it works.

+1

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted