Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Various improvements to AWS Batch support #67

Merged
merged 6 commits into from Sep 25, 2023
Merged

Conversation

gillins
Copy link
Member

@gillins gillins commented Sep 25, 2023

This PR adds the following:

  • Allow setting of the AWS Region used for creation of resources
  • Insert the region in the bucket name so a user can have buckets in different regions without name clash
  • A new ECR repo for the "main" script that actually contains the user's script. Building and pushing to this (along with the submission to Batch) is left as a task for the user. It seems more likely that users will want to run their "main" script as a Batch job rather than running locally.
  • Permissions to allow submission of jobs so that the user's "main" script as a Batch job can then fire off worker jobs.
  • Implement tags for users

In future I suspect it may be necessary to ensure that the "main" script has access to more local storage as you'd only be using this if you had very large files and by default Batch jobs don't have much local storage. This can be done by either adding a LaunchTemplate to the ComputeEnvironment that increases the size of the root volume, or my directly creating and mounting an EBS Volume. RIOS should probably do something here to make it easy, but I'll make that a separate PR.

@neilflood
Copy link
Member

That seems all very good (as if I understood it....). My only quibble is to request that you change the name of the region environment variable. Currently you have it as RIOS_BATCH_REGION, but I would prefer RIOS_AWSBATCH_REGION, just to be completely clear that this relates only to AWS Batch.

Sometimes my nit-picking amazes even me........

@gillins
Copy link
Member Author

gillins commented Sep 25, 2023

Good thought!

@neilflood
Copy link
Member

I just bothered to check, and it appears you already had a RIOS_BATCH_STACK variable as well. Sorry, I had not noticed that. Anyway they should be consistent, so I guess that means change that to RIOS_AWSBATCH_STACK. Does it???

Sorry, I am not being a very good reviewer.

@gillins
Copy link
Member Author

gillins commented Sep 25, 2023

I'd forgotten about that - good spotting. PR's are great aren't they.

@gillins gillins merged commit fb648e8 into ubarsc:master Sep 25, 2023
2 checks passed
@gillins gillins deleted the aws_region branch September 25, 2023 05:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants