Publish helm charts to helm chart repository

[planetf1]

The egeria-palisade project needs to make use of our 'lab' helm chart
To enable this easily, the egeria lab chart needs to be served via a helm repo as per https://helm.sh/docs/chart_repository/

Ultimately this needs to be done as part of the build process, to ensure these charts are always current. However for expediency in integration the generated file is being checked in to enable this reuse quickly and will be refined in time.

For now only the 'lab' chart is exposed.

Part 1 - create repo (can only be tested after merge due to the way github pages works....)

[planetf1]

The first step of this will publish to github pages

Depending on how usable that is, I may then investigate alternatives - be it hosted at the odpi as an output from the build pipeline(s) and/or in future public repos (but not yet)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[planetf1]

See https://github.com/kiwigrid/helm-maven-plugin

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[planetf1]

Keeping open - we should determine where our Helm charts are published to

[planetf1]

Update June 2021

When testing the postgres connector I wanted to 'package' up the deployment I had created for future re-use, and to act as an interesting sample. This is in addition to the 'odpi-egeria-lab' and 'egeria-base' charts we have, and the 'vdc' chart we used to use. We also have a test chart for postgres testing (not currently used)

I figured the 'egeria-base' (or lab) could be a good starting point, to which I just wanted to

• use a custom image so I also got the connector (other approaches like wget are feasible)
• add in a postgres deployment (helm chart) configured to use a sample data source
• extend the configuration with additional startup scripting to setup the data manager & integration daemon

I did this manually by installing the egeria-base chart, a postgres chart, then issueing the right REST requests

It would be possible to document these steps, or indeed explain how to pull the source for the original chart, make changes & then deploy to k8s, but this loses the attraction of codifying the example for reproducibility (and potentially use for testing). Too many steps, too much scope for error.

I can't simply write a new chart for postgres which depends on one of our other charts since at this point:

• Helm does not support installing charts directly from git, only from a local filesystem, or packaged in a repository (though there has been a PR being worked on for a few years which is getting closer Continue to add support for git urls as dependency repositories helm/helm#9482 )
• There is no longer a public chart repository
• charts need packaging and indexing as part of a build process
• I need somewhere to push them too

This means I have quite a specific workflow specific to helm charts. Just the same as base egeria is primarily java (with a little docker), and the k8s operator is in go - those sit best in different repositories so the CI/CD process can be customized to their needs.

There are some evolving examples of automation of this form for helm ie https://github.com/marketplace/actions/helm-chart-releaser . This would allow us to publish charts to github pages for direct installation via URL but I note

• releases are tied to chart versions (which are likely more frequent than egeria releases)
• github pages is already heavily in use for egeria documentation
• the ci/cd process is quite unique to helm

We are likely to increase the charts we have in future. WHilst work is also underway on an operator, a chart remains a good way to customize further & aggregate - and being able to reuse existing charts make this even easier. Other contributors can add their own examples

The options for where to manage the charts might include
- egeria : main repo
- many: whichever repo best aligns with the content - hadoop, database-connector etc - this has an attraction too in that the content is related - but it often is an aggregation from many places, and the kinds of assets we are dealing with (ie charts vs java) mean a very different process
- egeria-samples: a location for disparate resources associated with demos
- egeria-charts: a new proposal
- egeria-k8s: put all the k8s together (operator, charts etc) -- but I think this is less preferable due to different lifecycle management/cicd etc

I would propose we create a new repo 'egeria-charts' which will allow for automated publishing. This also removes a step from the user installing a chart (git clone(s)). This would ONLY include the charts. Not images, sample data etc. They are really just yaml definitions and perhaps a few other script files. Most code is in the actual docker images. They would stay where they are - though independently can also be looked at to see where they better belong

Comments welcome @mandy-chessell @cmgrote @davidradl @wbittles @lpalashevski & anyone else

[planetf1]

@mstrelchuk please also add as discussion item for next zoom meeting

[planetf1]

I am prototyping this in a new repo https://github.com/planetf1/egeria-charts (including git migration) as I need to validate the actions & publishing
If we go ahead and create a new repo I would propose to transfer this repo to odpi (rather than recreate afresh)

[planetf1]

• Moving content to another repo whilst preserving history -> https://medium.com/@ayushya/move-directory-from-one-repository-to-another-preserving-git-history-d210fa049d4b
• Automating helm publish process -> https://github.com/marketplace/actions/helm-chart-releaser
• Creating an empty branch -> https://gist.github.com/blattmann/4bc90557c2bc2907e715f45ec35674a2

[planetf1]

This action has now produced 2 releases. These can be found at https://github.com/planetf1/egeria-charts/releases and are versioned as per the content in the chart definition. So for each publishable change this needs to be manually incremented currently.

The charts are published to a compliant helm repository at https://github.com/planetf1/egeria-charts/blob/gh-pages/index.yaml

The resulting chart can therefore now be directly installed by adding the repository first:

$ helm repo add egeria https://planetf1.github.io/egeria-charts/                                                         [11:00:47]
"egeria" has been added to your repositories

And updating charts (this is standard practice):

$ helm repo update                                                                                                       [11:05:14]
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "egeria" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈

At this point I expect the chart to be installable - however :

 jonesn:charts/ (main) $ helm install lab odpi-egeria-lab                                                                                          [11:06:57]
Error: found in Chart.yaml, but missing in charts/ directory: kafka
jonesn:charts/ (main) $ helm dep update                                                                                                           [11:07:07]
Error: Chart.yaml file is missing

So it's possible the dependency may need retrieving before the chart is packaged ....

Ultimately though this should mean no need to clone repo, easier versioning, and a more independent lifecycle for the charts

[cmgrote]

Yes please -- I have loads of charts in my connector repos that I'd like to add into this new repository as well 😁

[planetf1]

Agreed in developer call 2021-06-22 that we can go ahead with this
I will request a repo migration via @jmertic (expect a request incoming via github)
I will not remove/update docs on the charts until after the immiment release, and will test the charts for 2.11 there.

[planetf1]

Waiting for a transfer of this report . Will update when complete and then other charts can be added and should get published automatically

[planetf1]

Mostly done -- but need to update documentation appropriately

[planetf1]

This is now in place & docs updated

For example chart can now be installed via

helm repo add egeria https://odpi.github.io/egeria-charts
helm repo update
helm install lab egeria/odpi-egeria-lab

Also can use

helm install --devel lab egeria/odpi-egeria-lab 

to get any development charts