[Feature Request]: ability to configure image pull secrets and linking of them to dynamically-created serviceaccounts

[shalberd]

Feature description

Goal: allowing authenticated image pulling from an enterprise docker registry. Context on-premise, airgapped, enterprise environments hosting mirrored ODH docker images in general including the notebook images (the ones used in the imagestreams).

Currently, image-pulling using dynamic, notebook-specific serviceaccounts

a) in jupyter tile, user name based

b) in data science projects, workbench name based

linked to notebooks is not possible when using an image registry source that requires authenticated access.

@andrewballantyne @VaishnaviHire @LaVLaS some thoughts on how to achieve this

a) dockerconfigjson secret creation would need to be done after data science project creation.

Necesssary infos for that (repopath, username, userpassword/token) could come from OdhDashboardConfig or an already-existing Openshift secret in the main open data hub namespace. I am not sure how secure OdhDashboardConfig would be for putting authentication-relevant fields in there. But, on the other hand, an Openshift secret once generated is not exactly secure either. All admins of a project can decode and see its base64 contents. Edit type users can see secrets, but not decode the content, though I guess they could do a manual base64 decode.

b) linking up the image pull secret by name with the notebook-specific dynamic serviceaccount. Currently, odh notebook controller handles the creation of dynamic new notebok serviceaccounts. We could probably pass the image pull secret name to odh-notebook-controller as well, similar to workbench / notebook name.

https://github.com/opendatahub-io/kubeflow/blob/v1.6-branch/components/odh-notebook-controller/controllers/notebook_oauth.go#L46

That would require a change in assembleNotebook, though in odh dashboard. Either a label or annotation for image pull secret name.

related issue in odh manifests, where for static serviceaccounts,

• creating the image pull secret in the main opendatahub namespace
• the linking-up of image pull secret by name to a serviceaccount can be done in the manifests themselves with parameter-passing (image pull secret name) and patching of the static application service accounts

https://github.com/opendatahub-io/odh-manifests/issues/833

Describe alternatives you've considered

No response

Anything else?

No response

[andrewballantyne]

@shalberd Once the image stream is created, do you still need to authenticate to use it in pulling at the notebook start? Or is this more of the admin getting it "added" to the cluster (eg. on the admin's custom notebook images page)

[shalberd]

@andrewballantyne the imagestream name and tag are the basis for assembly of the image-field in the notebook, leading to a statefulset, leading to a pod. The moment you need the image pull secret is when the notebook / statefulset pods are started with the image-download/pull. The image pull secret is not needed for the imagestream itself, but for pulling referenced images in an imagestream.

But only when either the from.name repo in an imagestream tag e.g. quay.io/opendatahub/workbench/datascience:v2023.1 requires authentication (only ever likely with dockerhub) or when in airgapped clusters, the original location, e.g quay.io/opendatahub/workbench/datascience:v2023.1 points to a private repo (kinda like a softlink) with authentication indirectly via ImageContentSourcePolicy. At least for the case of airgapped, though, mirroring of images requires setting up a so-called cluster-level global pull secret anyways, to push up to the internal repo during mirroring.

@Jooho @LaVLaS So I am holding off for now, it could be the secret for that corporate docker repo, e.g. Artifactory or VMWare Harbor, is already set up as a global pull secret by cluster-admins anyways. The reason we do not have it yet: operator is not yet airgapped / image mirroring compatible. And my very manual way of copying images over to a private docker repo while also changing all the image references in odh manifests is kind of a hack.