Create robert-allaway-project

[allaway]

Adding a new project for personal testing and pilot research projects.

[BrunoGrandePhD]

@allaway: I hesitate to merge this because AWS imposes a hard limit of 50 AWS Batch compute environments (CEs) per account. This is something we discovered relatively recently. Since each Tower project creates 3 CEs and we already have 10 active Tower projects in nextflow-prod, we're getting close to our limit, so I want to ask some follow-up questions.

• Would one of your existing Tower projects make sense for the testing you plan on carrying out?
• If not, do you plan on processing PHI or controlled-access data? (Asking to see if you need a production-grade account)
• How urgent is this request? (Asking to see if we can spin up a new AWS account, which will likely take a few weeks)

[BrunoGrandePhD]

I forgot to post my previous comment as a review. Doing this now to silence GitHub reminders.

[allaway]

Hi Bruno,

Thanks for the info! I did not realize that otherwise I would have been more parsimonious with our accounts.

Would one of your existing Tower projects make sense for the testing you plan on carrying out?

Unfortunately, this is for an unfunded pilot project, so I don't want these costs to be billed against/recorded as being part of the other projects, which all have distinct & different program codes.

If not, do you plan on processing PHI or controlled-access data? (Asking to see if you need a production-grade account)

Yes, the data are from dbGaP and EGA :)

How urgent is this request? (Asking to see if we can spin up a new AWS account, which will likely take a few weeks)

It would be very helpful to be able to process the data sooner so that we can send the counts to some external collaborators who are working on it. If I am not able to use Tower, I would probably resort to setting up a beefy EC2 instance with service catalog and running everything there, but I think that is probably less ideal.

I'm curious, let's say I process everything within a week or two and then can deposit the reads on Synapse and "tear down" the project. Does this allow you to recover the CEs?

Cheers,
Robert

[BrunoGrandePhD]

Yes, there is a way to decommission a Tower project (i.e. delete the CEs) while maintaining read-access to the workflow runs. Since this is intended as a short-lived effort, I'll deploy the project. You don't need to rush anything, but I might ping you if we run into our CE limit (which is unlikely in the coming months).

[allaway]

That sounds good. Thank you Bruno!