Jupyter notebooks to another repository?

[cmgrote]

At the moment our Jupyter notebooks for the various labs are all embedded in this core repository -- as with other examples (Helm charts, samples, etc) this means that we take a very long hit before we can merge anything while we wait for Java code to build (which the notebooks don't touch at all)...

I vaguely recall we'd previously discussing moving these out to e.g. the egeria-samples repository (or perhaps some other repository), but that we were waiting until some of the other dependent material was moved (like the Helm charts) (?)

Now that the Helm charts are in their own dedicated repository and no longer embedded here, would now be a logical time to move these lab notebooks out as well -- or was there something else we were dependent on / waiting on (?)

[planetf1]

Yes we have discussed, but indeed it was partly impacted by where the charts where, though that was never a hard block since the charts just depend on a docker image, more figuring out the logical split. We also still have the 'docker-compose' as part of core egeria (which uses the notebook charts). I'd be tempted to deprecate them, leave as is, and no develop further, but that needs discussion. An alternative is to move them outside, perhaps even their own repo.

So I don't see a notable impediment to moving the charts. This should include the creation of the jupyter docker image (or an alternative approach that delivers the right content), the main question would be where to
egeria-samples - have discussed for placing a variety of samples
a new dedicated repo more targetted, just for coco
a new dedicated repo specifically for charts

One thing to note is that new developments in the base are sometimes picked up in these notebooks quickly, so we have to be careful in terms of rebuilding etc.

Finally I won't be able to review or work on this during the remainder of the 3.0 release cycle, but am happy to talk through what is needed, testing requirements, prep for release pipelines etc if needed in the next few days.

[cmgrote]

Challenges with moving at the moment:

• Coco Pharma security setup is in the code itself at the moment (e.g. CocoPharmaServerSecurityConnector, etc)

[planetf1]

The notebooks could still be moved out for packaging, even if the security connector is in the base. It's arguably logically incomplete since the coco pharma enviroment is to be seen as a 'whole' but there's certainly no blocking issue to the move of the notebooks if we feel that is important and needed. - as per previous comment.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[planetf1]

I don't think we're waiting for anything on this -- but it could be good to either move them before the dojo, or leave them as-is longer term.

If moved, I'm neutral to where exactly - egeria samples, or a new repo . I would suggest moving the notebooks themselves, along with the docker build of jupyter including the notebooks as they belong closely together

In future other assets that belong in the jupyter environment should go here - I would keep them seperate from changes to the server environment as they are packaged differently, for a different target. (but it's still ok to be in a subdirectory of a larger repo)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[planetf1]

I would propose we make this change 'soon' (and I am happy to do it) (after 3.6 release)

I think this is needed so that

• We can encourage more notebooks into the labs environment (even if within a 'contrib' or 'unsupported' directory so that's clear what they are) - for example I have been building one for a 1:1 demo
• We can more rapidly iterate notebooks when needed, independent of release
• We continue stripping down egeria repo to the real 'core'.
• slims down the egeria container (albeit small!)

• Agree target repository/path
• Copy notebooks to new location
• move any outstanding issues across
• Create a distribution archive (gzip etc) via CI/CD (and versioned)
• update egeria coco charts to use new archive & test
• update any links in docs/code to new location
• identify/merge across any additional changes made to charts since copy
• delete original notebooks

The hardest aspect is probably agreeing the name/location. The options include

• A new repository ie 'egeria-demo-notebooks' or 'egeria-coco'
• Using 'egeria-samples'
• ...
  I would probably opt for the latter, under 'jupyter-notebooks'. I think the former is too restrictive. demos may require new containers, data files, sql imports etc.. we may have technical notebooks unrelated to coco (so if we go with coco we may need a second).

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[planetf1]

this move is mostly done -- the development versions of the charts now use this repository as the source of the notebooks.

[planetf1]

this move is complete