Timeout on /containers when creating a container with cellpose

[odagayev]

Here is the build log: https://wave.seqera.io/view/builds/dded52ad814da816_1

I'm trying to build something with this package: https://github.com/MouseLand/cellpose

It appears that this container times out. I have a feeling that this takes a long time because of the CUDA dependencies that this package has. I'd imagine that many ML focused packages right now are running into a similar issue.

I would love to be able to address this issue for the ML community.

[pditommaso]

Currently all build have an hard limits to 15 minutes. It could looked into how to have different build profiles having a longer expiration time.

[odagayev]

What are the implications of doubling this build time? Is the 15 arbitrary or is there a reason for the hard limit?

[pditommaso]

One side the longer the build the easier for a malicious user to saturate our build cluster.

Also a too long build would may cause the invoking client (e.g. a nextflow pipeline job to hold for all build time doing anything and burning money)

[ewels]

How about having extended build times only for authenticated users? That might mitigate the risk of malicious users a little.

Another option is to make it an opt-in flag, so that longer build times are only used if specifically requested. This is what GitHub Actions does for example.

[ewels]

Suggestion from @pditommaso in weekly meeting: Quick patch to double timeout to 30 mins for when user is authed + freeze is set. This will apply to all builds in Seqera Containers @odagayev.

Can investigate more complex solutions at a later date if we continue to see problems.

[odagayev]

I'm afraid this won't work. This is for seqera.io/containers which is mostly unauthed users. Let me know if this is possible.

[pditommaso]

Yes, we are using an account for it

[odagayev]

Thank you guys. 🙏

CC: @FloWuenne

[pditommaso]

Solved by #600

[pditommaso]

This has been released. Build submitted by auth users and having freeze enable will have 25 minutes (instead of 15 min) build time.