-
Notifications
You must be signed in to change notification settings - Fork 659
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consider migrating off Travis CI #1770
Comments
"Capitalism Considered Harmful" Can we just do what Ansible core does and get the same guarantees and workflows? It uses Shippable? Can Red Hat pay for a CI account for this project so we don't get limited? Financial support considered useful :) EDIT: Thanks for raising this issue! Very important, it seems. |
There's a pretty good awesome list here: https://github.com/ligurio/awesome-ci |
@decentral1se Ansible already spends a lot of funds on Shippable CI (and we already create an enormous load for them with this) for Ansible Core Engine and I'm not aware of any funds dedicated to extending it. OTOH I'm not the right person to comment on this. And yet I think it's typical for businesses to pay for things only if they see business need/opportunities in such things. Additionally, if we had a wider community we could think of deploying http://buildbot.net/. I know it's used by CPython project and AFAIK it's based on various community members providing their machines to be nodes and centralized orchestration system puts jobs on those. Based on this I'd like to discover freely available resources first. Another point to this would be that we will probably benefit from distributing jobs across various CIs regardless of Travis CI going down just because we have a lot of heavy jobs in our test suite and there's more coming. |
This comment has been minimized.
This comment has been minimized.
I would like to see us add Zuul to the list, we have access to zuul resources in the ansible-network org and believe we can discuss opening it up to other teams. In fact we already have awx using it, would be great to have another project. |
Last week, I had some time to hack on this a little. If you are interested in the results of zuul running molecule tests, you can follow along at: |
The fact that Molecule is now ansible/molecule tells me this is already the case. |
@decentral1se not really, funding process is still separated. Molecule is not being sold as a supported product, at this point, it's mostly community-based. |
Some good discussion here, I've closed #1874 so we can just have one place to discuss Molecule CI. I thought a summary of where we are up to would be good:
[1] It's much cheaper by node to add resources in Shippable than Travis. Also as resources are added at the GitHub Org level (ie github.com/ansible) those paid resources are available to Ansible, ansible-lint and Molecule (the three projects that used paid Shippable) so it benefits multiple projects by giving us a larger common CI resource pool. |
@gundalow To me it seems that for short term we should try to piggyback Ansibe Shipably and medium term to have Zuul running. |
@ssbarnea Yup, that's my thinking as well. We can discuss this during Wednesday's meeting.
|
@gundalow sounds good. I'd like to improve publishing flow tho. By exploiting GitHub Apps maybe (not sure about Actions, yet) |
Regarding zuul, realistically, we are likely looking at 1st of May for live date for new zuul.ansible.com. I am hoping sooner, but we first need to migrate ansible-network, then awx / ansible-runner. |
OK, update to my comments in #1770 (comment) Given Zuul is closer than I thought (yay) I'll setup a Travis =-CI subscription just for Molecule to use for the next few months till Zuul is here. Given out non-trivial |
Great news then! |
Question: How many concurrent jobs do we want to be able to run? The Travis plans seem to be Bootstrap Startup Small Business Premium |
Assuming that doubling the number of concurrent jobs would divide by 2 the duration of a build (I know this assumption is borderline).
4 jobs per day is very close to the current workload, it does not let room for much workload increase. Considering this, I would vote for an increase to 10 jobs per day. |
We now have a concurrent job count of 10 on Travis-CI. We can monitor this. Feedback welcome via this issue. |
@gundalow is that for repo or for org? |
I've cancelled a bunch of things and now it all just looks stuck |
@webknjaz https://www.traviscistatus.com/ shows there have been issues with Linux node not booting 1450-1809UTC, though they are still cleaning up thing. 10 nodes are for the org, though I don't see much other traffic on the other projects |
ansible-network/sandbox#31 shows an example of how molecule can be tested on https://dashboard.zuul.ansible.com. For the example, I just used the existing tox testing for python36 on a fedora-29 node. |
@pabelanger I see that uses old Check Status API. How can it be transitioned to Checks API? |
If I remember well @pabelanger said at some pint that porting to the check status api was on its way, the only reason being lack of time. |
Currently, it's exceptionally hard to navigate to the proper log pages so I'd say that proper reporting is critical here. |
@webknjaz Is happening but I would not say that zuul logs are easier to read, in fact the opposite. While I faces the issue already in my projects I ended up using pytest and pytest-html as a wrapper for molecule calls in order to get the output easier to read. Look at https://review.rdoproject.org/r/#/c/20240/ - click py27 job and see the last pytest, which is mainly an execution of molecule. As you can imagine this output scales well with multiple executions. |
@ssbarnea show me plz where I can see that log for molecule tests from this example: #1770 (comment) I'm having a hard time finding the view for that :( |
@webknjaz This is just the default log collection done by our tox based jobs, if you point me which specific logs you'd like collected, this can be handled in a post-run playbook. Also, check status api is a work in progress, when I lasted checked we were waiting for github3.py to add support, and somebody to write the patch. |
It's not Check Status API. This one is used currently. I'm talking about Checks API. I just want the output of whatever was run during the job execution and can't find it anywhere. |
Is this an example of the output you are looking for? https://logs.zuul.ansible.com/31/31/af348c98d67a9f59028159c2a9656049428f1692/check/molecule-tox-py37-ansible27-unit/9379045/job-output.html#l437 Please keep in mind, I have zero experience on how molecule developers are doing testing or what expected output should be seen. This was my 1 hour attempt to show how zuul.ansible.com could run a tox based job. Any sort of log collection, reporting, failing tests all can be fixed, I'm happy to work with people to do that. It looks like the test runner is pytest, by default the jobs today are setup to use testr / stestr and subunit, which is why some of the additionally logging / reporting is missing. |
@pabelanger yes, thanks! It's just that I don't know how to navigate to that URL by myself. And it's a blocker as most folks will probably have the same problem. So this must be solved in order to have a successful transition. And it's nice that you can refer to lines in logs. I like that. Though, it'd be nice if you could link a range of lines like in other CIs. Travis/GitHub and a lot of others use a URI fragment like Our entrypoint for everything is tox. So the main objective would be setting up the mapping with tox envs. I'm curious whether it's possible to generate job matrixes in a compact way in Zuul. Regarding logs, I think the main thing I want to see is just raw line-addressable logs. And additionally, I'd be nice to see the representation from xunit reports which pytest is able to generate out of the box. |
The good news here, is we can propose any changes to make Zuul better with GitHub integration, we can likely drive a lot of that design. What we'd need to do, is come up with what the spec would look like and propose it upstream to Zuul mailing list. Then we can start work on contributing that work. For logs, we have full control over how they are generated and presented, today this is just the default. We can totally support multi-line selects, today we use an ansible role to generate them (htmlify-logs)[1] before we upload them into swift. We just need to determine what the javascript looks like and update the role to include it. I can poke around on online and see if I can find an example. As for job matrix in zuul, I would say yes. Zuul jobs them self do not support templating of variables or parameters, however we do have the ability to do job inheritance. We can create a base molecule tox job that applies to all tox env, then create child jobs as we need more defined variables. If you can point me to an example of the xunit reports you have for molecule today, I can write up a post-run playbook to collect them properly. [1] http://git.zuul-ci.org/cgit/zuul-jobs/tree/roles/htmlify-logs |
We don't have xunit reports yet. I have used them in other projects, though. It's an easy to use built in feature of pytest. |
Today I saw some disturbing news on the Internet:
TL;DR Idera (new Travis' overlord) has unexpectedly fired an enormous amount of engineers. There are rumors/suggestion that Travis CI platform itself may be trashed.
With that in mind, there are several candidates:
The text was updated successfully, but these errors were encountered: