Helm charts refactor proposal

[Tomcli]

As many users complaint about the usability of FfDL (a lots of them are regrading the deployment). We might want to deprecate the makefile approach because it's very scripted. Instead, I refactor the helm charts and convert some scripts such as creating the configmaps as part of the helm deployment. I also break down the current 2 helm charts into the below 3

• docs/helm-charts/storage-plugin (for installing s3 driver)
• docs/helm-charts/ffdl-helper (Installing the monitoring and infrastructure services (etcd/mongo/etc..) along with any local persistent volume and storage configmaps, previously done by scripts. This way we can avoid race condition from the ffdl-core services because DLaaS also have these services/pods deployed first).
• docs/helm-charts/ffdl-core (All the ffdl-core services (trainer/lcm/etc..))

With this approach, we can deprecate the make deploy/quickstart-deploy to using 3 helm commands (Assuming Helm charts are hosted on our GitHub Pages)

export NAMESPACE=default
export SHARED_VOLUME_STORAGE_CLASS="ibmc-file-gold";

helm installibmcloud-object-storage-plugin --name ibmcloud-object-storage-plugin --repo https://ibm.github.io/FfDL/helm-charts --set namespace=$NAMESPACE # Configure s3 driver on the cluster
helm install ffdl-helper --name ffdl-helper --repo https://ibm.github.io/FfDL/helm-charts --set namespace=$NAMESPACE,shared_volume_storage_class=$SHARED_VOLUME_STORAGE_CLASS --wait # Deploy all the helper micro-services for ffdl
helm install ffdl-core --name ffdl-core --repo https://ibm.github.io/FfDL/helm-charts --set namespace=$NAMESPACE,lcm.shared_volume_storage_class=$SHARED_VOLUME_STORAGE_CLASS --wait # Deploy all the core ffdl services.

Having multiple helm charts allow users to delete/rollback their core ffdl charts without touching many of the infrastructure to help the development/testing process faster. This might also help with DLaaS because we can make the ffdl-core helm charts more concise.

Note this is a proposal PR with the helm charts. If we agree to work on this, we need to to deprecate the make deploy and publish the helm charts using GitHub-Pages (Will be published once merged to master branch).

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

[Tomcli]

Update: I rewritten the instructions and travis CI to use the helm charts only for deployment. Since the GitHub page won't be available until we merge it to master, anyone who wants to test it should use the below commands that use the local helm charts.

export NAMESPACE=default
export SHARED_VOLUME_STORAGE_CLASS="ibmc-file-gold";

helm install docs/helm-charts/ibmcloud-object-storage-plugin-0.1.tgz --set namespace=$NAMESPACE
helm install docs/helm-charts/ffdl-helper-0.1.1.tgz --set namespace=$NAMESPACE,shared_volume_storage_class=$SHARED_VOLUME_STORAGE_CLASS --wait
helm install docs/helm-charts/ffdl-core-0.1.1.tgz --set namespace=$NAMESPACE,lcm.shared_volume_storage_class=$SHARED_VOLUME_STORAGE_CLASS --wait