Curious about CI costs

[afiaka87]

I've been thinking about adding CI to clip-guided-diffusion - I have a few integration tests that just run a few forward passes on CPU/GPU. Saw your note about pricing being an issue and it's essentially the main thing preventing me from doing that on a GPU setup in the cloud. For now, I'm fortunate the most of the checkpoints run inference fine on my RTX 2070 that I own.

Assuming you don't have any issue with revealing such info; what are the costs like for CI on the project? Do you have any tips for people trying to do CI with machine learning?

Thanks in advance for any info

[duskvirkus]

@afiaka87

Currently using google cloud platform. You can check pricing on their site. They'll give you $300 dollars in free credit if you join. You have to add a credit card to get access to a gpu and write a request. (Also if you request a quota of more than one you'll have to get in touch with a sales rep)

You can see the specs for the gpu template I have. (It's slightly better compute than a colab instance but only a t4 gpu). It costs about $0.11 for each ~12 minute ci run. Which doesn't sound like way to much but it also starts to add up.

If you benchmark the tests you run you can get a decent estimate. But also add 5 to 6 minutes for installing nvidia drivers. Trying to get rid of that overhead but I need a higher gpu quota which I haven't gotten yet and then you also have to pay for persistent storage.

Between all the debugging of ci and rerunning tests (and forgetting to turn off an instance for 8ish hours) I'm up to about $100 for this project so far (about 2 months or so). Given I've almost run 100 ci tests on this repo so far you could estimate a practical cost of about $1 per ci run. Of course that includes TPU tests which each one of those 10 minute test costs about $0.28.

So if you're just doing a gpu with a similar run time and config I'd predict you end up spending $0.20-$0.35 per ci run when all is said and done.

[duskvirkus]

Closing but feel free to reopen if you have more questions.