Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Yan 893 - Multinode deployments of helm charts (in-progress) #116

Merged
merged 40 commits into from
May 4, 2022
Merged

Yan 893 - Multinode deployments of helm charts (in-progress) #116

merged 40 commits into from
May 4, 2022

Conversation

pritchardn
Copy link
Collaborator

An in-progress pull that permits single-node helm deployments from the translator interface (assuming the machine running the lg-web has access to kubectl).

@pritchardn
Adds multiple minikube nodes instructions.
991ab3c
@pritchardn
Adds anti-affinity rule to minikube deployment.
7023a41 
Means there will only be a single daliuge-daemon pod deployed on each node.
This minimises namespace conflicts.
@pritchardn
Adds simple function to find k8s node IPs
71627ab
@pritchardn
Adds node option to start_helm_cluster and experiments with scaling
894f95b
@pritchardn
Adds replica scaling to helm_client.
90732a1 
Now, all requests to the exposed service hangs. This needs debugging.
@pritchardn
Broken dreams attempting to run multiple daliuge instances.
6ee93a5
@pritchardn
A much closer attempt at a k8s deployment.
9687487
@pritchardn
Reverts back to a scaling stateful set approach.
f4afd27
@pritchardn
Re-implements manually scaled deployment options.
d869c17 
Makes ip finding deterministic.
@pritchardn
Working towards multi-node deployments with simple single-node deploy…
5cdaae2 
…ments.
@pritchardn
Updates start_helm function to force single node deployments.
cde344c 
Removes needless printing from helm_client.
@pritchardn
Merge branch 'master' into yan-893
2339a58
@pritchardn
Changes HELM Deploy button to HELM Deploy (Experimental)
a59a5d6
@pritchardn
Cleans up submission of PGT to k8s cluster.
45d1d71
@pritchardn
adds co-host argument (which further restricts helm deployment to a s…
cd17764 
…ingle pod)
@pritchardn
Debugging helm client tests.
fccd427
@pritchardn
More debugging helm client tests.
81d63e8
@coveralls
Copy link

coveralls commented Feb 23, 2022 •
edited
Loading

Coverage Status

Coverage increased (+0.2%) to 66.905% when pulling 9e7d82c on yan-893 into 03cabaf on master.

@pritchardn
Adds kubectl check code. Ensures helm client has access to kubectl be…
043e29a 
…fore continuing.
@pritchardn
Rest interface now accepts text/plain requests
f8740aa
@pritchardn
Changes HELM deploy button to run JS function in main.js
d9f2b20
@pritchardn
Provides a dummy implementation of helm_deploy which uses a local RES…
0dbb554 
…T end-point.

This code should not have to change much, although what it points to, will.
@pritchardn
Removes gen_pg_helm from lg_web.py
7e889cb
@pritchardn
Fixes helm_deploy append_graph request to avoid errors.
d45fe30
@pritchardn
Updates imports in slurm_client.py
ca2878e
@pritchardn
Aliases localhost to 127.0.0.1 when performing graph translation.
29768d6
@pritchardn
Formatting adjustments
a280d86
@pritchardn
Refactors submit_job into submit_pgt and submit_pg respectively.
9065b47
@pritchardn
Removes unnecessary print
a4d9b4b
@pritchardn
Removes redirect after helm deployment.
a753e03
@pritchardn
Adds logging to start_helm_cluster.py, helm_client.py and reated depl…
f83c775 
…oyment_utils.py
@pritchardn
Successfully implements multi-node helm charts.
e2a5055
@pritchardn
Adds port specification to submit_pg
07a25f0
@pritchardn
Adds monitoring to helm_client and start_helm_cluster.py
ca86fc8
@pritchardn
Removes co-host from helm client and changes k8s check to return fals…
9ad82ac 
…e rather than fail on exception
@pritchardn
Adds helm tests back with k8s check skip
39da4c4
@pritchardn
Conditionally imports helm client only if k8s deployment is available…
1b717e1 
… (requires both translator and engine).
@pritchardn
Moves check_k8s_env to deployment_utils to avoid CI import failure.
3fb9f0e
@pritchardn
Adds docs for k8s deployment
2493ab6
@pritchardn
Merge branch 'master' into yan-893
700a7b9 
# Conflicts:
#	daliuge-translator/dlg/dropmake/web/main.js
#	daliuge-translator/dlg/dropmake/web/pg_viewer.html
@pritchardn
Adds co-hosting back to helm_client.
9e7d82c
@pritchardn
Copy link
Collaborator Author

pritchardn commented Mar 14, 2022 •
edited
Loading

This feature is now relatively stable, supports multiple node (pod) deployments and is integrated with the latest master.
However, it is not perfect, below are some limitations:

  • Log files will collide when running multiple k8s pods on the same physical node
  • Multi-pod (node) deployments on a single physical machine will take up to 3 minutes to start. This is due to a namespacing conflict with zeroconf - it does work but takes longer than one would like
  • All pods get their own service endpoints - while useful for external control, this could be removed to save resources
  • There is no user-side indication that a helm deployment has been successful or even begun.
  • Launching multiple helm charts with the same name simultaneously is obviously problematic, but there is no workaround or detection of such issues yet.

I argue that the implementation of the base feature is 'complete', however, there is still more that could be done.

@pritchardn
Copy link
Collaborator Author

To replicate the entire stack locally:

Setup K8s:
Add multiple nodes if interested with minikube node add
minikube start
minikube tunnel --cleanup
minikube mount ~/dlg:/dlg -v 3

Setup dlg-ood (3bff037)
python passenger_wsgi.py helm

Setup lgweb (9e7d82c)
dlg lgweb -d ~/dlg -t ~/dlg_temp/

One should be able to launch jobs through the helm deploy button in the translator view.

@pritchardn pritchardn merged commit 09cb668 into master May 4, 2022
@pritchardn pritchardn deleted the yan-893 branch May 4, 2022 03:16
awicenec pushed a commit that referenced this pull request May 19, 2022
@pritchardn
Merge pull request #116 from ICRAR/yan-893
6a5e92e 
Yan 893 - Multinode deployments of helm charts
pritchardn added a commit that referenced this pull request May 20, 2022
@pritchardn
Merge pull request #116 from ICRAR/yan-893
e7925ec 
Yan 893 - Multinode deployments of helm charts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@awicenec awicenec Awaiting requested review from awicenec

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@pritchardn @coveralls