Add OCluster to development docker #213
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is preliminary integration with ocluster: This PR only adds an ocluster to the development docker, but it doesn't actually change the pipeline to use workers. It can be helpful to play with workers though, and more integration will follow soon!
Some things to note:
capnp-secrets
contains the permissions to talk with the ocluster. Theadmin.cap
is used by the pipeline inentrypoint.sh
to generatesubmission.cap
which can then be used to submit new jobs to the cluster.pool-linux.cap
to receive jobs from the cluster (for that pool, you can add more, thelinux
name is arbitrary). By default, this file hascluster:9000
as the host... so you may need to edit it to a real IP orlocalhost:9000
or addcluster
to your hosts.local-test-repo
should be specified in thedevelopment.env
as an absolute path: Otherwise when the pipeline will submit jobs to the cluster, the workers will not be able to find the git repository (since they are running outside of the pipeline docker, the external path should be the same and not/app/local-test-repo
).See the OCluster readme for some commands to experiment with workers!