Integrate DLF into ODH JupyterHub

[vpavlin]

In opendatahub.io we use https://github.com/opendatahub-io/jupyterhub-singleuser-profiles to customize Jupyter instances for users.

I'd like to come up with a good integration between DLF and the profiles library.

As discussed with @YiannisGkoufas we would need to be able to mount volumes read-only too avoid users overwritting datasets for others

Then we could add a simple way to automatically label pods and add mounts to pods for users from a specific group

[YiannisGkoufas]

Thanks for moving this forward @vpavlin !
For the read-only, a simple solution would be to extend the CRD to something like this:

apiVersion: com.ie.ibm.hpsys/v1alpha1
kind: Dataset
metadata:
  name: your-dataset
spec:
  local:
    type: "COS"
    accessKeyID: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
    secretAccessKey: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
    endpoint: "https://s3.eu.cloud-object-storage.appdomain.cloud"
    bucket: "kubeflow-yiannis"
    region: "" #it can be empty
    readonly: true

Regarding the labeling etc, the current way to mount in /mnt/datasets/* is as follows:

  labels:
    dataset.0.id: "your-dataset"
    dataset.0.useas: "mount"

Do you need a different functionality about the pods labeling convention?

[vpavlin]

@YiannisGkoufas The labels should be fine, I would suggest to move the readonly up to spec.readonly instead of spec.local.readonly - I believe the readonly option applies to all the backends so it makes more sense to make it more "global" on dataset level. WDYT?

[YiannisGkoufas]

Oh yes the local bit is going away in any case. Will work on this and update the current issue. Thanks @vpavlin

[vpavlin]

Is there any progress on readonly access?

[YiannisGkoufas]

Hey @vpavlin yes yes, not documented yet, but if you are feeling brave, try the following images for the dataset operator and csi-s3.

yiannisgkoufas/dataset-operator:readonly-amd64
yiannisgkoufas/csi-s3:readonly-amd64

Example readonly dataset (only working on COS)

apiVersion: com.ie.ibm.hpsys/v1alpha1
kind: Dataset
metadata:
  name: your-dataset
spec:
  local:
    type: "COS"
    accessKeyID: "xxxx"
    secretAccessKey: "xxxxxx"
    endpoint: "http://s3.eu.cloud-object-storage.appdomain.cloud"
    bucket: "kubeflow-yiannis"
    readonly: "true"
    region: "" #it can be empty

edit: removing the local bit is part of a bigger effort to restructure the CRD, but its on the pipeline

[YiannisGkoufas]

Hi @vpavlin can you check the branch https://github.com/IBM/dataset-lifecycle-framework/tree/readonly it support the readonly functionality in a Dataset as shown in the example

[vpavlin]

Hi @YiannisGkoufas we are pushing a release of ODH and trying to finish some changes in JupyterHub, which is preventing me from looking at this. Just wanted to let you know this is still on the list for me!

[YiannisGkoufas]

No worries @vpavlin ! thanks for letting me know. I have actually gathered some questions/ideas on ODH integration so would love discuss more when you have time

[srikumar003]

Closing this issue as this is stale