Include elyra package in all data science workbenches #58

harshad16 · 2023-03-23T14:53:12Z

Include elyra package in all data science workbenches

Description

This PR includes elyra package as whole to data science workbench images.
It would bring in support for elyra pipeline, extensions and plugins.

Followed the similar pattern from contrib: https://github.com/opendatahub-io-contrib/workbench-images/tree/main/jupyter

Related-to: #55
opendatahub-io/s2i-lab-elyra#61

How Has This Been Tested?

To be tested from PR images.

Merge criteria:

The commits are squashed in a cohesive manner and have meaningful messages.
Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
The developer has manually tested the changes and verified that the changes work

atheo89 · 2023-03-24T13:43:05Z

Awesome! I have just one observation to make. Would it be possible to hide the /utils/setup-elyra.sh from the /opt/app-root/src workdir? Perhaps you could adopt the approach used in the minimal dockerfile.

harshad16 · 2023-03-24T13:59:32Z

Thanks @atheo89 , i will fix that 👍

harshad16 · 2023-03-28T13:26:19Z

Ready for Review:
Include elyra plugins:
pipeline-editor, python-editor and code-snippet
reference: #55 (comment)

The plugin are working in the image:

harshad16 · 2023-03-28T21:40:02Z

/test notebooks-e2e-tests

atheo89 · 2023-03-29T10:10:41Z

/test notebooks-e2e-tests

atheo89 · 2023-03-29T12:05:51Z

jupyter/datascience/ubi8-python-3.8/Dockerfile

 # Install Python packages and Jupyterlab extensions from Pipfile.lock
 COPY Pipfile.lock ./

+# Copy Elyra setup to utils so that it's sourced at startup
+COPY setup-elyra.sh ./utils/
+
 RUN echo "Installing softwares and packages" && micropipenv install && rm -f ./Pipfile.lock



It seems that the Python version displayed on the launcher is not inherited from minimal.
Could you please add the following command on the data-science dockerfiles 3.8/3.9?

# Replace Notebook's launcher, "(ipykernel)" with Python's version 3.x RUN sed -i -e "s/Python.*/$(python --version | awk '{print "Python", $2}' | cut -d'.' -f-2)\",/" /opt/app-root/share/jupyter/kernels/python3/kernel.json

Moreover, update the corresponding command into the minimal dockerfile, and remove it from the TrustyAi dockerfile, because it will be inherited by the datascience dockerfile.

This cmd displays the python version without the patch version, so eventually, we could close also this issue: #49

Seems like this cmd is needed explicitly in each image build.
As the jupyter setup is getting overridden in build-time, the base image change of the kernel version is not carried over.
Tested on different images.
Included it in another images as well.

This one is not implemented yet.
(Maybe we can leave it for another time, in an enhancement pr)

i didnt understand the comment,
would like me to remove the change made and move it separate pr for later ?

Sure, let's leave it for later on, to unblock this pr

atheo89 · 2023-03-29T12:26:05Z

Moreover, while I was browsing into the files, I observed that the pyTorch 3.9 notebook, has wrong references on the Lebels cmd: https://github.com/harshad16/odh-notebooks/blob/setup-elyra/jupyter/pytorch/ubi9-python-3.9/Dockerfile#L4-L12 Could you update the those to 3.8 to 3.9?
Additionally, to that, I realized the io.openshift.build.image label in all the dockefiles points to the old registry, but should points to quay.io/opendatahub/workbench-images:${NOTEBOOK_IMAGE} 😆 .
I know that these updates don't relate to that particular pr, but if it is easy for you could you update them, otherwise we could open another PR later for it.

harshad16 · 2023-03-30T15:27:48Z

/test notebooks-e2e-tests

harshad16 · 2023-04-03T13:16:38Z

/test notebook-cuda-ubi8-python-3-8-pr-image-mirror

harshad16 · 2023-04-03T14:06:22Z

/test notebook-cuda-ubi8-python-3-8-pr-image-mirror

harshad16 · 2023-04-03T16:25:40Z

/retest

harshad16 · 2023-04-03T16:33:44Z

/retest

harshad16 · 2023-04-03T18:21:55Z

/test images

harshad16 · 2023-04-03T18:35:56Z

/test notebooks-e2e-tests

jupyter/datascience/ubi9-python-3.9/Dockerfile

atheo89 · 2023-04-05T13:18:29Z

jupyter/datascience/ubi8-python-3.8/Dockerfile

 # Install Python packages and Jupyterlab extensions from Pipfile.lock
 COPY Pipfile.lock ./

+# Copy Elyra setup to utils so that it's sourced at startup
+COPY setup-elyra.sh ./utils/
+
 RUN echo "Installing softwares and packages" && micropipenv install && rm -f ./Pipfile.lock



This one is not implemented yet.
(Maybe we can leave it for another time, in an enhancement pr)

jupyter/datascience/ubi8-python-3.8/Dockerfile

- Filter out Pipeline plugin only for local and kfp - Pre-set the kfp-tekton for kfp pipeline support Signed-off-by: Harshad Reddy Nalla <hnalla@redhat.com>

Signed-off-by: Harshad Reddy Nalla <hnalla@redhat.com>

atheo89 · 2023-04-07T14:43:41Z

/lgtm

VaishnaviHire · 2023-04-07T15:03:01Z

/lgtm

Tested this by running an example pipeline

openshift-ci · 2023-04-07T15:03:26Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: VaishnaviHire

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [VaishnaviHire]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

harshad16 · 2023-04-07T16:12:35Z

/test notebooks-e2e-tests

) * Include elyra package in all data science workbenches - Filter out Pipeline plugin only for local and kfp - Pre-set the kfp-tekton for kfp pipeline support Signed-off-by: Harshad Reddy Nalla <hnalla@redhat.com> * Point all images label at opendatahub/workbench-images repo Signed-off-by: Harshad Reddy Nalla <hnalla@redhat.com> --------- Signed-off-by: Harshad Reddy Nalla <hnalla@redhat.com>

* Include as a reviewer for workbench images (#52) Signed-off-by: Harshad Reddy Nalla <hnalla@redhat.com> * Include elyra package in all data science workbenches (#58) * Include elyra package in all data science workbenches - Filter out Pipeline plugin only for local and kfp - Pre-set the kfp-tekton for kfp pipeline support Signed-off-by: Harshad Reddy Nalla <hnalla@redhat.com> * Point all images label at opendatahub/workbench-images repo Signed-off-by: Harshad Reddy Nalla <hnalla@redhat.com> --------- Signed-off-by: Harshad Reddy Nalla <hnalla@redhat.com> * Add documentation for notebooks image update process (#61) * updated trustyai to 0.2.10 * Update Owners file (#66) * updated trustyai to 0.2.11 * Include runtime images into workbench images Signed-off-by: Harshad Reddy Nalla <hnalla@redhat.com> * Included the validation test for runtime images Signed-off-by: Harshad Reddy Nalla <hnalla@redhat.com> * updated trustyai to 0.2.12 * Wait for the runtime pod readiness for e2e tests (#76) Signed-off-by: Harshad Reddy Nalla <hnalla@redhat.com> * Remove the default Pipeline runtime config (#70) Signed-off-by: Harshad Reddy Nalla <hnalla@redhat.com> * Provide runtimes image in workbench images (#64) Signed-off-by: Harshad Reddy Nalla <hnalla@redhat.com> * Allow elyra pipeline config mount on ~/runtimes dir (#65) Signed-off-by: Harshad Reddy Nalla <hnalla@redhat.com> * Set default env vars for ssl_sa_certs and sa_token path (#77) Signed-off-by: Harshad Reddy Nalla <hnalla@redhat.com> --------- Signed-off-by: Harshad Reddy Nalla <hnalla@redhat.com> Co-authored-by: aTheo <atheodorak@outlook.com> Co-authored-by: Tommaso Teofili <tteofili@redhat.com> Co-authored-by: Vaishnavi Hire <vhire@redhat.com>

shalberd · 2023-05-25T06:57:27Z

Elyra has been taken over by RedHat, and RedHat in turn is a part of IBM, so Redhat will now also take part of discussions at elyra-ai.slack.com?

Regarding this PR: That is very cool! @harshad16 So there is no more need for an explicit Elyra notebook image via overlay? https://github.com/opendatahub-io/odh-manifests/tree/master/notebook-images/overlays/additional

harshad16 · 2023-05-25T10:53:30Z

Elyra has been taken over by RedHat, and RedHat in turn is a part of IBM, so Redhat will now also take part of discussions at elyra-ai.slack.com?

we will take part in the slack.

Regarding this PR: That is very cool! @harshad16 So there is no more need for an explicit Elyra notebook image via overlay? https://github.com/opendatahub-io/odh-manifests/tree/master/notebook-images/overlays/additional

yes , the eventual goal is to remove the overlay
our plan is to wait for one release and then once it announce and user have time to switch to new images
we would remove the overlay.

openshift-ci bot requested review from atheo89 and VaishnaviHire March 23, 2023 14:53

harshad16 changed the title ~~Include elyra package in all data science workbenches~~ [wip] Include elyra package in all data science workbenches Mar 24, 2023

openshift-ci bot added the do-not-merge/work-in-progress label Mar 24, 2023

harshad16 force-pushed the setup-elyra branch 4 times, most recently from b1d564f to a3a283a Compare March 28, 2023 13:08

harshad16 changed the title ~~[wip] Include elyra package in all data science workbenches~~ Include elyra package in all data science workbenches Mar 28, 2023

openshift-ci bot removed the do-not-merge/work-in-progress label Mar 28, 2023

atheo89 reviewed Mar 29, 2023

View reviewed changes

harshad16 force-pushed the setup-elyra branch 2 times, most recently from e50bd97 to 82e4402 Compare March 30, 2023 13:07

harshad16 force-pushed the setup-elyra branch from 82e4402 to b898c0b Compare March 30, 2023 20:34

harshad16 force-pushed the setup-elyra branch from 9c2dd68 to 5692f1d Compare April 3, 2023 16:15

harshad16 changed the title ~~Include elyra package in all data science workbenches~~ [wip] Include elyra package in all data science workbenches Apr 3, 2023

openshift-ci bot added the do-not-merge/work-in-progress label Apr 3, 2023

harshad16 changed the title ~~[wip] Include elyra package in all data science workbenches~~ Include elyra package in all data science workbenches Apr 3, 2023

openshift-ci bot removed the do-not-merge/work-in-progress label Apr 3, 2023

This was referenced Apr 7, 2023

Include runtime images into workbench images #63

Merged

Allow elyra pipeline config mount on ~/runtimes dir #65

Merged

atheo89 reviewed Apr 7, 2023

View reviewed changes

harshad16 added 2 commits April 7, 2023 10:27

Include elyra package in all data science workbenches

6c7ecce

- Filter out Pipeline plugin only for local and kfp - Pre-set the kfp-tekton for kfp pipeline support Signed-off-by: Harshad Reddy Nalla <hnalla@redhat.com>

Point all images label at opendatahub/workbench-images repo

a76bfd4

Signed-off-by: Harshad Reddy Nalla <hnalla@redhat.com>

harshad16 force-pushed the setup-elyra branch from 5692f1d to a76bfd4 Compare April 7, 2023 14:28

openshift-ci bot assigned atheo89 Apr 7, 2023

openshift-ci bot added the lgtm label Apr 7, 2023

openshift-ci bot assigned VaishnaviHire Apr 7, 2023

VaishnaviHire approved these changes Apr 7, 2023

View reviewed changes

openshift-ci bot added the approved label Apr 7, 2023

openshift-merge-robot merged commit e95157b into opendatahub-io:main Apr 7, 2023

harshad16 deleted the setup-elyra branch May 11, 2023 10:03

shalberd mentioned this pull request May 26, 2023

Elyra now part of ODH, but Airflow optional support needs to be there opendatahub-io/opendatahub-community#45

Open

This was referenced Oct 4, 2023

Pin kfp-tekton dependency to match installed Tekton version opendatahub-io/s2i-lab-elyra#59

Closed

Hide runtimes other than Kubeflow Pipelines opendatahub-io/s2i-lab-elyra#60

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Include elyra package in all data science workbenches #58

Include elyra package in all data science workbenches #58

harshad16 commented Mar 23, 2023 •

edited

Loading

atheo89 commented Mar 24, 2023

harshad16 commented Mar 24, 2023

harshad16 commented Mar 28, 2023

harshad16 commented Mar 28, 2023

atheo89 commented Mar 29, 2023

atheo89 Mar 29, 2023 •

edited

Loading

harshad16 Mar 29, 2023

atheo89 Apr 5, 2023

harshad16 Apr 7, 2023

atheo89 Apr 7, 2023

atheo89 commented Mar 29, 2023

harshad16 commented Mar 30, 2023

harshad16 commented Apr 3, 2023

harshad16 commented Apr 3, 2023

harshad16 commented Apr 3, 2023

harshad16 commented Apr 3, 2023

harshad16 commented Apr 3, 2023

harshad16 commented Apr 3, 2023

atheo89 Apr 5, 2023

atheo89 commented Apr 7, 2023

VaishnaviHire commented Apr 7, 2023

openshift-ci bot commented Apr 7, 2023

harshad16 commented Apr 7, 2023

shalberd commented May 25, 2023

harshad16 commented May 25, 2023

Include elyra package in all data science workbenches #58

Include elyra package in all data science workbenches #58

Conversation

harshad16 commented Mar 23, 2023 • edited Loading

Description

How Has This Been Tested?

Merge criteria:

atheo89 commented Mar 24, 2023

harshad16 commented Mar 24, 2023

harshad16 commented Mar 28, 2023

harshad16 commented Mar 28, 2023

atheo89 commented Mar 29, 2023

atheo89 Mar 29, 2023 • edited Loading

Choose a reason for hiding this comment

harshad16 Mar 29, 2023

Choose a reason for hiding this comment

atheo89 Apr 5, 2023

Choose a reason for hiding this comment

harshad16 Apr 7, 2023

Choose a reason for hiding this comment

atheo89 Apr 7, 2023

Choose a reason for hiding this comment

atheo89 commented Mar 29, 2023

harshad16 commented Mar 30, 2023

harshad16 commented Apr 3, 2023

harshad16 commented Apr 3, 2023

harshad16 commented Apr 3, 2023

harshad16 commented Apr 3, 2023

harshad16 commented Apr 3, 2023

harshad16 commented Apr 3, 2023

atheo89 Apr 5, 2023

Choose a reason for hiding this comment

atheo89 commented Apr 7, 2023

VaishnaviHire commented Apr 7, 2023

openshift-ci bot commented Apr 7, 2023

harshad16 commented Apr 7, 2023

shalberd commented May 25, 2023

harshad16 commented May 25, 2023

harshad16 commented Mar 23, 2023 •

edited

Loading

atheo89 Mar 29, 2023 •

edited

Loading