Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TravisCI price model change #38

Closed
icemac opened this issue Nov 2, 2020 · 41 comments
Closed

TravisCI price model change #38

icemac opened this issue Nov 2, 2020 · 41 comments
Labels
question Further information is requested

Comments

@icemac
Copy link
Member

icemac commented Nov 2, 2020

TravisCI is introducing a new price model today.
If I understand the new model correctly we get 10,000 credits per month and a single linux build minute costs 10 credits. So we have 1,000 build minutes per month for the whole zopefoundation organization (?).

I fear this is not enough.

A simple calculation:
We have 392 public repositories minus 62 archived ones = 330 repositories which might run on TravisCI.
Let's assume 300 repositories have a monthly cron job running 5 jobs (aka 5 different Python versions).
If each job takes only 1 minute we only need 1,500 minutes for the cron jobs (!).
(I know many repositories are run weekly an only a few jobs can run in 1 minute.)

It is possible to apply for more credits which is decided by the TravisCI support on a per request basis.
Maybe we need a new solution to run our CI jobs –

  • or is my calculation simply wrong
  • or do we have another chance to reduce the needed build minutes so 10,000 credits could be enough
  • or is my understanding wrong and each contributor or repository has 10,000 credits?

Source: https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing#building-on-a-public-repositories-only

@icemac icemac added the question Further information is requested label Nov 2, 2020
@icemac
Copy link
Member Author

icemac commented Nov 2, 2020

We switched to Travis-CI.com so we have to add up org an com.

https://travis-ci.org/github/zopefoundation?tab=insights: about 4,700 build minutes.
https://travis-ci.com/github/zopefoundation?tab=insights: about 4,300 build minutes.

This are about 9,000 minutes aka at least 90,000 credits.

This includes two script runs changing many repositories via PRs, so they were run thrice (2 times for the PR and one time for the merge) but we still seem to use way more build time than are given from now on.

@The-Compiler
Copy link

I've already mentioned this on Twitter, but let me elaborate here a bit:

If I understand the new model correctly we get 10,000 credits per month

I don't see anything in the announcement saying you get them per month - on the contrary, I'm pretty sure you get them once:

For those of you still kicking the tires on the Travis CI product, we are switching the plan from 100 builds on private repositories to our trial (free) plan with a 10K credit allotment. This will allow you to build on whichever environment you’d like to test out. Trial plans expire when you use all of our credits or after a year.

and

For those of you who have been building on public repositories (on travis-ci.com, with no paid subscription), we will upgrade you to our trial (free) plan with a 10K credit allotment (which allows around 1000 minutes in a Linux environment).

as well as:

When your credit allotment runs out - we’d love for you to consider which of our plans will meet your needs.

So, unless I'm missing something, you'd need to haggle with Travis CI staff to get more build-minutes - again, nothing in there says you'll get them every month, quite the contrary: should your run out of credits again you can repeat the process to request more

From my experience with the Travis CI support so far (though, granted, a while back) and given that this will affect tens if not hundreds of thousands of projects, I don't think this will work out well.

@icemac
Copy link
Member Author

icemac commented Nov 3, 2020

I looked at https://travis-ci.com/organizations/zopefoundation/plan there is a little question mark next to the free plan its hover text reads as: „Free Plan is trial plan. The credits will be not replenished. Switch to Free Plan if you wish to cancel your subscription.“

It seems that @The-Compiler is right: we seem to get the 10,000 credits only once.
Either we have to

  • contact TravisCI support to ask for more each time we run out of credits,
  • we have to pay for the service (The „5 Concurrent plan“ looks like what we currently use, it costs $249/month!)
  • or we have to switch to another CI system (Github actions provides 2,000 minutes/month in the free plan and it seems that public repos are free, see https://github.com/pricing)

@dataflake
Copy link
Member

Wringing hands and making assumptions is not really helpful. I have followed the procedure outlined on their blog article announcing the change and lodged a support request asking how to proceed.

@dataflake
Copy link
Member

Quick update: I have received a reply that sounded like boilerplate when a single project applies for an ongoing free subscription. They list several requirements which we all meet. I wrote back a lengthy email explaining that I am speaking for an entire GitHub organization and provided information about the history of Zope, the ecosystem and the foundations. Let's see what happens.

@mgedmin
Copy link
Member

mgedmin commented Nov 13, 2020

Let's have a plan B. Is anyone using GitHub actions already in a way they like?

I think currently https://github.com/zopefoundation/z3c.jbot/blob/master/.github/workflows/test.yml is the only project in the zopefoundation namespace that uses GitHub actions. I'm not in love with the excessive boilerplate, but I think we can learn to live with it.

Does anyone have better examples to imitate?

@dataflake
Copy link
Member

I'm all for using GH actions, just never found enough time to get into it. Maybe there are repos in the plone GH organizations that have better usage examples?

@jamadden
Copy link
Member

AppVeyor may be another option. We already use it when we need to test on Windows; it also supports macOS and Linux builds.

A positive is that they seem to add new Python versions relatively quickly; a negative is that they only do so with full release versions (not betas or RCs, at least as far as I've seen on Windows).

Another positive is that they don't limit the build minutes in a given month, as GitHub actions does. However, they do restrict the free plan to one concurrent build, which could add substantial delays. I don't know what levels of concurrency GitHub actions supports?

@dataflake
Copy link
Member

I cannot stand Appveyor. It is so slow that it's basically unusable.

@hugovk
Copy link

hugovk commented Nov 13, 2020

I have stats on how quickly CIs added Python 3.8 and 3.9!

From release to support:

2019: Python 3.8

🥇 21 hours: Travis CI
🥈 20 days: GitHub Actions
🥉 24 days: Azure Pipelines
4️⃣ 25 days: AppVeyor

2020: Python 3.9

🥇 17 hours: GitHub Actions ⬆️
🥈 20 days: Travis CI ⬇️
🥉 21 days: AppVeyor ⬆️
4️⃣ 22 days: Azure Pipelines ⬇️

@jugmac00
Copy link
Member

Lol, Hugo :-D I just searched for your tweet and was about pasting :-D you are so damn fast!

@mauritsvanrees
Copy link
Member

@jensens added GitHub Actions on plone.recipe.zope2instance.

@gotcha added them in buildout.

@The-Compiler
Copy link

I'm not in love with the excessive boilerplate, but I think we can learn to live with it.

FWIW you could probably share it between different projects by creating a custom action and using that in the projects instead.

Another positive is that they don't limit the build minutes in a given month, as GitHub actions does.

Note that's only the case for private repositories. Build minutes for public repos are unlimited.

I don't know what levels of concurrency GitHub actions supports?

The 20 jobs I have in my repository all run in parallel.

@icemac
Copy link
Member Author

icemac commented Nov 13, 2020

We are using GH actions in the private projects of our company let me try to apply this knowledge to a zopefoundation project.

@gotcha
Copy link
Member

gotcha commented Nov 13, 2020

@gotcha added them in buildout.

I did and I am pretty happy.

The web UI could be better but the integration worked out well.

@gotcha
Copy link
Member

gotcha commented Nov 13, 2020

@dataflake In case Travis answer comes back positive, would you mind sharing your mails in order for me to request for the buildout organization ?

@jugmac00
Copy link
Member

@hynek has a good blog post https://hynek.me/articles/python-github-actions/ after which I applied GH actions to some of my personal projects.

@icemac
Copy link
Member Author

icemac commented Nov 13, 2020

See zopefoundation/zope.browser#7 for my take to run the tests on GHA like they were run on TravisCI. (Sorry, the diff in the PR is not very much readable because I startet with zopefoundation/zope.browser@815752c on master to activate GHA for this repos.

@hugovk
Copy link

hugovk commented Nov 13, 2020

Quick update: I have received a reply that sounded like boilerplate when a single project applies for an ongoing free subscription. They list several requirements which we all meet.

Sounds like the reply I got: https://twitter.com/hugovk/status/1326935425903185920. I've had no reply to my last mail, but it's only been one day and it took them 10 days to reply to the first.

@hugovk
Copy link

hugovk commented Nov 14, 2020

The Raku org were given 25k credits per month on 5th November, here's the reply from Travis:

https://twitter.com/jjmerelo/status/1324272244349173760

https://github.com/Raku

@dataflake
Copy link
Member

One thing I wrote the Travis people is that with the number of repositories we have and (as far as I know) no obvious way to check how many "minutes" we use I cannot even tell them what we need.

@hugovk
Copy link

hugovk commented Nov 14, 2020

There's an "insights" tab:

image

https://travis-ci.org/github/zopefoundation?tab=insights

image

https://travis-ci.com/github/zopefoundation?tab=insights

So 1,408 mins on .org + 8,300 mins on .com = 9,708 mins total for the last month.

@dataflake
Copy link
Member

@hugovk Thanks for pointing that out. Learned something new 😄

@icemac
Copy link
Member Author

icemac commented Nov 17, 2020

I migrated zope.browser to GHA, see the result in https://github.com/zopefoundation/zope.browser/blob/master/.github/workflows/tests.yml.
I'd like to update meta/config to GHA so we can deploy it more easily.

@icemac
Copy link
Member Author

icemac commented Nov 18, 2020

I integrated GHA support here in meta/config.

@mgedmin
Copy link
Member

mgedmin commented Nov 20, 2020

check-python-versions 0.17 can now check (and update) .github/workflows/tests.yml based on the meta/config template.

@icemac
Copy link
Member Author

icemac commented Dec 2, 2020

I disabled all the Travis-CI cron jobs, so we do not burn valuable credits there.
Even though https://travis-ci.com/organizations/zopefoundation/plan says: "You have used 6480 of 10000 credits".
@dataflake Did you get an answer from the TravisCI support regarding your request for more credits?

@dataflake
Copy link
Member

I did not get an answer yet, no. I asked for an update several days ago.

@hugovk
Copy link

hugovk commented Dec 2, 2020

I put my last request in on 16 November.

On 20 Nov, Travis said:

Thanks so much for your reply and very sorry for the delay here as we've had a higher than average backlog. Your request for the OSS plan for your account Hugovk ​is waiting for managerial approval. More on this to come!

Yesterday they said:

Thanks for your patience while we worked on your request.

We are awaiting further approval for your account and will promptly apply credits as directed to your account.

In the meantime, however, we have added 50,000 credits to your account to enable you continue using the service. I hope this helps.

Thanks and we will follow up soon.

@mauritsvanrees
Copy link
Member

Was there any communication from Travis yet?

When I go to https://travis-ci.com/organizations/zopefoundation/plan I see:

Builds have been temporarily disabled for private and public repositories due to a negative credit balance.

and:

-660 available credits (purchase date: January 08, 2021)
You have used 10660 of 10000 credits

Screenshot 2021-01-08 at 12 32 10
Screenshot 2021-01-08 at 12 31 54

@dataflake
Copy link
Member

Zero communication on the original issue in their ticketing system, and zero communication on a second issue I created to flag up this total lack of communication. Travis CI appears to be a lost cause. I won't follow up on it anymore unless I hear back from them. If anyone else wants to contact them, please do so.

@dataflake
Copy link
Member

I have left another comment in the issue I had created at Travis CI and told them they are free to close it and that we are moving to GitHub Actions instead. I also told them that I fully understand when business models change, but this silent treatment for 4 months is extremely unprofessional.

From my Travis CI dashboard I also looked at some other GitHub organizations that I am a member of. Both Plone and the Collective suffer from the same issue, they're still on the limited free plan and exhausted their minutes a long time ago.

@icemac I suggest uninstalling the Travis CI "application" at the zopefoundation organization root level hoping that it will become more obvious for everyone that Travis CI won't work anymore.

@jensens
Copy link
Member

jensens commented Mar 5, 2021

Well, yes, we need to move all collective repos to GHA - what a waste of time.

I like the idea of removing TravisCI on orga root level, it is dead. I will bring this up for Plone and Collective as well.

@jugmac00
Copy link
Member

jugmac00 commented Mar 5, 2021

A couple of days ago, I created a PR for the cpython project, and I was extremely surprised as they still use Travis (partly).

This was the first open source repository I saw with a working Travis CI in months.

Maybe you need a big lobby...

Anyway, I am glad it is over.

@dataflake
Copy link
Member

This was the first open source repository I saw with a working Travis CI in months.

Maybe you need a big lobby

I think you hit it on the head. We're small fish, they really don't give a damn. In my comment I also expressed my surprise that they think this unprofessional treatment is OK, as if prospective or current paying clients won't notice any of it and draw their own conclusion. If I were a business owner on one of their paid plans I'd immediately think "business risk - unreliable/unprofessional vendor".

@hugovk
Copy link

hugovk commented Mar 5, 2021

Likewise I've not had any replies for months for multiple requests for my account and several orgs, and have also moved everything important over to GitHub Actions and disabled all the rest (they'll be moved if needed).

If I see an open source repo using Travis CI, I'm much less likely to contribute because I can't test changes on my fork before creating a PR. 🤷

@mauritsvanrees
Copy link
Member

Note that repositories that use travis-ci.ORG actually still work. For example this week I worked on fixing tox and Travis for plone.api and that goes fine. But of course this is only a matter of time, as on top it says:

Please be aware travis-ci.org will be shutting down in several weeks, with all accounts migrating to travis-ci.com. Please stay tuned here for more information.

It is March now and I am surprised it is still running.
Moving to GHA is definitely the plan.

@icemac
Copy link
Member Author

icemac commented Mar 5, 2021

@dataflake Let's wait with the removal of the TravisCI GitHub app until we have fully migrated to GHA. I do no know what will happen on TravisCI's site if we remove the app. It would be bad if they would delete the history of the builds. It could be useful for some packages to be able to see if the tests were previously successful over there.

@dataflake
Copy link
Member

It won't really matter for us anymore, but today I received a reply to my original trouble ticket at Travis CI telling me they have added 25k credits. They are asking to contact support again if credits are running low, so it seems to be a one-time credit.

@icemac
Copy link
Member Author

icemac commented Apr 12, 2021

Nice to hear, there are still some dozens of packages to be migrated to GHA, so it could help in between.

@icemac
Copy link
Member Author

icemac commented Jun 21, 2021

We settled on GHA. So I am closing this issue.

@icemac icemac closed this as completed Jun 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

10 participants