Financing arm64 runners

[mathbunnyru]

My Google Cloud trial is over, Tau T2A VMs are available for a free trial until March 31, 2024.

And, there is already some charge for networking - our building process consumes lots of networking (uploading/downloading images as artifacts). Also some daily fees for SSD ($0.34) and static IP addresses ($0.19).
I'm glad I introduced zstd 3 months ago - it definitely saved me some money 😆
My costs for August and September were $24.28 and $26.42 respectively.

I'm okay to pay for these runners out of my pocket (for now), but it would be nice to hear from Jupyter community if there is something we can do 😄

[mathbunnyru]

One day after I complained, Apple silicon powered M1 macOS Github runners were released as a public beta.
https://github.blog/2023-10-02-introducing-the-new-apple-silicon-powered-m1-macos-larger-runner-for-github-actions/

I think I will create a PR trying to use them.
It's gonna be just a little bit weird, that we build Linux images on Mac, but actually it's nice - these images should build just fine on Mac (I've always developed these images using my local Mac and sometimes Linux VM).

[yuvipanda]

If you want to continue using GCP with custom runners, I can try ask internally at 2i2c if we can help cover some of the cost.

[yuvipanda]

Alternatively we can use QEMU in the regular runners! It's a bit slower, but costs no money and doesn't require us to maintain more infra. We use it through the jupyterhub project (https://github.com/jupyterhub/jupyterhub/blob/70717dc9ab163b568c51f70369374d35b91ba56f/.github/workflows/release.yml#L111, https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/78bec62d591b7a97560c37edd79c7a15186fcfe9/.github/workflows/publish.yml#L71, etc). Would you be open to trying that out here?

[mathbunnyru]

If you want to continue using GCP with custom runners, I can try ask internally at 2i2c if we can help cover some of the cost.

That would be nice, thank you!

[mathbunnyru]

Alternatively we can use QEMU in the regular runners! It's a bit slower, but costs no money and doesn't require us to maintain more infra. We use it through the jupyterhub project (https://github.com/jupyterhub/jupyterhub/blob/70717dc9ab163b568c51f70369374d35b91ba56f/.github/workflows/release.yml#L111, https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/78bec62d591b7a97560c37edd79c7a15186fcfe9/.github/workflows/publish.yml#L71, etc). Would you be open to trying that out here?

We had QEMU before and this is gonna be going backwards, not forward.
#1703

At least one image doesn't build using it, maintaining qemu is more difficult than maintaining self-hosted runners, we also had qemu related hard-to-fix bug, the build time is much slower - our CI takes 1h30min, if I remember correctly it's gonna take at least ~2.5 times longer with QEMU.

[yuvipanda]

Ah TIL #1703.

I'll reach out internally and try figure something out!

[minrk]

FWIW, I've used CircleCI a few places for free-tier native ARM builders, pyzmq for wheels, and some docker images for work. I believe Travis CI also has free-tier aarch46 builders.

[mathbunnyru]

FWIW, I've used CircleCI a few places for free-tier native ARM builders, pyzmq for wheels, and some docker images for work. I believe Travis CI also has free-tier aarch46 builders.

Thanks, I'll take a look. I kind of like having everything in GitHub, and it works well.
Also, a build system is relatively complicated, so I'll have to find some alternatives in other CI systems (and hope that there will be no other limitations).

[minrk]

Yeah, it's all trade-offs. Spreading builds across CIs definitely has a cost, if not a financial one. There isn't a single great answer. Paying some $$$ to keep things simple is definitely sensible, if we can find a way to do it.

[consideRatio]

Paying some $$$ to keep things simple is definitely sensible, if we can find a way to do it.

I fully agree, i think 2i2c can fund this, otherwise my "sundell open source consulting" will!

This is the most advanced use of githubs CI system i've seen, so maintaining it in two systems isn't an option i think.

[yuvipanda]

@mathbunnyru I'm working internally in making this happen. Do you have a sense of what total cost would be? I am currently asking for Google Cloud support, capped at $100 a month for the next year (and check-in at that point). Do you think that'll cover it?

[mathbunnyru]

I think it should be $150 a month.

Note: Tau T2A VMs are available for a free trial until March 31, 2024, that's why currently my monthly bills are $20-$30 (it depends on the number of builds).

This is how it looks like without discounts (and this is what I'll be paying from Aprial 1, 2024).

[yuvipanda]

@mathbunnyru good news! I've $2000 approved for use over the next 12 months at least (and then check-in)! We'd like to be credited with a simple line in the docs and README as sponsoring the CI for ARM images. If this sounds ok to you, I can work with you on setting this up. If you already have a google cloud project going, perhaps the easiest is for us to work together on putting that on a different payment method that comes from us? Then we can set budget alerts to monitor.

Thank you for all the amazing tireless work you do <3

[mathbunnyru]

@mathbunnyru good news! I've $2000 approved for use over the next 12 months at least (and then check-in)! We'd like to be credited with a simple line in the docs and README as sponsoring the CI for ARM images. If this sounds ok to you, I can work with you on setting this up. If you already have a google cloud project going, perhaps the easiest is for us to work together on putting that on a different payment method that comes from us? Then we can set budget alerts to monitor.

Thank you for all the amazing tireless work you do <3

That's great!

We'd like to be credited with a simple line in the docs and README as sponsoring the CI for ARM images. If this sounds ok to you, I can work with you on setting this up.

Sure, I think it's fair. Send a PR 🙂

If you already have a google cloud project going, perhaps the easiest is for us to work together on putting that on a different payment method that comes from us? Then we can set budget alerts to monitor.

Well, to be honest, I'm using my personal and main Google account currently.
I don't feel comfortable sharing the credentials of this account, as you understand 🙂

I think the best approach would be if you or I created a separate Google (and the Google Cloud) account, shared permissions, and you tied your payment method to this account.
I think it will only take around an hour for me to set up all github runners on a new account.
This way you won't have to share the credit card number with me, and I won't have to share my Google Account with you.

[yuvipanda]

@mathbunnyru oh yeah that makes total sense! Here's the actions I will take:

1. I'll create a Google Cloud Project
2. Tie our billing into that (I have requested a $2000 virtual card, should have it in a day or two)
3. Invite you to that project with full administrative privileges

Does that sound good? I'll do this once the card gets approved for my use.

[mathbunnyru]

@yuvipanda sounds wonderful! 👍

[yuvipanda]

@mathbunnyru this is set up! Can you share your email address with me (privately if you prefer - yuvipanda@gmail.com) and I'll give you full permissions?

[mathbunnyru]

@mathbunnyru this is set up! Can you share your email address with me (privately if you prefer - yuvipanda@gmail.com) and I'll give you full permissions?

Done

[mathbunnyru]

@yuvipanda I recreated the setup in the new project.
It seems to work fine: https://github.com/jupyter/docker-stacks/actions/runs/6705303694/job/18219446578

[mathbunnyru]

@yuvipanda I also created a PR mentioning 2i2c project. Please, take a look, and then I'll be able to completely close this issue.

[yuvipanda]

yay, thanks for getting this all set up @mathbunnyru - and thanks for all the work you do here :)

[mathbunnyru]

We somehow got rid of curl errors after we re-created aarch64 runners 🎉
Google might have fixed something, or maybe our new runners are in another network.

We tried to fix it here, but it didn't actually work and I was still getting some failures during downloads in aarch64 builds (and had to rerun failed builds from Web UI).
And now everything works fine - in the past 3 weeks I committed a lot, updated PRs, and so on - not a single unexplainable failure, everything works as it should.
Great news (mostly for me) 😆