Update Elyra image to v0.0.3

[vpavlin]

FYI @akchinSTC

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vpavlin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [vpavlin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vpavlin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [vpavlin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[vpavlin]

/test all

[vpavlin]

/test odh-manifests-e2e

[nakfour]

I have been testing this new image, and at this moment cannot get it to work. Here are some issues I found

1. The need to set /read/write to bucket for the download of .tgz file from bucket to work in pod
2. !pip install package in the notebook is not executing in the pods, so you end up with "module not found"

[vpavlin]

Can you share the error messages related to these issues?

[nakfour]

1. Bucket read issue

269
Traceback (most recent call last):
270
  File "bootstrapper.py", line 353, in <module>
271
    main()
272
  File "bootstrapper.py", line 345, in main
273
    file_op.process_dependencies()
274
  File "bootstrapper.py", line 81, in process_dependencies
275
    self.get_file_from_object_storage(archive_file)
276
  File "bootstrapper.py", line 129, in get_file_from_object_storage
277
    self.cos_client.fget_object(bucket_name=self.cos_bucket,
278
  File "/mnt/jupyter-work-dir/python3.6/minio/api.py", line 719, in fget_object
279
    stat = self.stat_object(bucket_name, object_name, sse)
280
  File "/mnt/jupyter-work-dir/python3.6/minio/api.py", line 1138, in stat_object
281
    response = self._url_open('HEAD', bucket_name=bucket_name,
282
  File "/mnt/jupyter-work-dir/python3.6/minio/api.py", line 2017, in _url_open
283
    raise ResponseError(response,
284
minio.error.AccessDenied: AccessDenied: message: Access Denied

2. !pip install issue

Package Installation Complete.....
279
Get file tf-mnist-677b8c2d-7ef5-49ab-96c6-046880197c06.tar.gz from bucket nakfour
280
Processing dependencies........
281
TAR Archive pulled from Object Storage.
282
Unpacking........
283
Unpacking Complete.
284
Executing notebook through Papermill: tf-mnist.ipynb ==> tf-mnist-output.ipynb
285
Unexpected error: <class 'papermill.exceptions.PapermillExecutionError'>
286
Converting from ipynb to html....
287
Uploading errored Notebook back to Object Storage
288
Uploading file tf-mnist-output.ipynb as tf-mnist.ipynb to bucket nakfour
289
Uploading file tf-mnist.html as tf-mnist.html to bucket nakfour
290
Traceback (most recent call last):
291
File "bootstrapper.py", line 353, in <module>
292
main()
293
File "bootstrapper.py", line 347, in main
294
file_op.execute()
295
File "bootstrapper.py", line 205, in execute
296
raise ex
297
File "bootstrapper.py", line 184, in execute
298
papermill.execute_notebook(
299
File "/mnt/jupyter-work-dir/python3.6/papermill/execute.py", line 110, in execute_notebook
300
raise_for_execution_errors(nb, output_path)
301
File "/mnt/jupyter-work-dir/python3.6/papermill/execute.py", line 222, in raise_for_execution_errors
302
raise error
303
papermill.exceptions.PapermillExecutionError:
304
---------------------------------------------------------------------------
305
Exception encountered at "In [3]":
306
---------------------------------------------------------------------------
307
ModuleNotFoundError Traceback (most recent call last)
308
<ipython-input-3-603629bd586d> in <module>
309
1 from __future__ import absolute_import, division, print_function
310
2
311
----> 3 import tensorflow as tf
312
4 from tensorflow.keras import Model, layers
313
5 import numpy as np
314
315
ModuleNotFoundError: No module named 'tensorflow'
316

[vpavlin]

Sorry, where are these errors coming from? I am not sure how I'd replicate these issues - can you provide more details?

[akchinSTC]

@vpavlin - The errors are coming from a very helpful sample notebook that @nakfour sent over to test and debug this issues with the elyra image. The notebook contains sensitive information so I wont post it here.
The problem stems from installing packages from within the notebook itself using pip. It will attempt to write to the base layer which is not permitted and so the install and import will fail. My workaround for elyra is to move the volume we create to /opt/app-root/src in the runtime pipeline images to align with the volume in place for spawned elyra container and update the pip globals installation target path to this location as well.
As for the bucket permissions issue, I have yet to experience the issue and am still actively trying to replicate.

[nakfour]

@akchinSTC I also ran the example listed here: https://github.com/elyra-ai/examples/tree/master/pipelines/hello_world_kubeflow_pipelines using Pandas 1.1.1 image, the only available Pandas image.
And the pipeline complained about a missing module in "Part 2 - Data Analysis.ipynb"
See error below

Converting from ipynb to html....
Uploading errored Notebook back to Object Storage
Uploading file Part 2 - Data Analysis-output.ipynb as Part 2 - Data Analysis.ipynb to bucket nakfour
Uploading file Part 2 - Data Analysis.html as Part 2 - Data Analysis.html to bucket nakfour
Traceback (most recent call last):
  File "bootstrapper.py", line 353, in <module>
    main()
  File "bootstrapper.py", line 347, in main
    file_op.execute()
  File "bootstrapper.py", line 205, in execute
    raise ex
  File "bootstrapper.py", line 184, in execute
    papermill.execute_notebook(
  File "/mnt/jupyter-work-dir/python3.6/papermill/execute.py", line 110, in execute_notebook
    raise_for_execution_errors(nb, output_path)
  File "/mnt/jupyter-work-dir/python3.6/papermill/execute.py", line 222, in raise_for_execution_errors
    raise error
papermill.exceptions.PapermillExecutionError: 
---------------------------------------------------------------------------
Exception encountered at "In [2]":
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-2-db2393c7a124> in <module>
      2 import numpy as np
      3 import pandas as pd
----> 4 import seaborn as sns
      5 from pandas import DataFrame as df
      6 from matplotlib import pyplot as plt
ModuleNotFoundError: No module named 'seaborn'

Part 3 also had a missing model sklearn

[nakfour]

@vpavlin just add a "!pip install" in a notebook, import the lib and run the pipeline.

[nakfour]

Will Updating to 0.0.4 in PR #207 close this PR?

[vpavlin]

/hold

[vpavlin]

The #207 will make this PR obsolete, but I'll keep it open for now to not lose the issues you mentioned

[akchinSTC]

The #207 will make this PR obsolete, but I'll keep it open for now to not lose the issues you mentioned

^^ thx. yes I wanted to keep this around as well for information sake. Once #207 is merged and closed, I will close this issue and document/create new ones for anything outstanding.