-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add CI configs #51
Comments
Any thoughts on Travis vs Circle here being our default CI provider? Travis has better support for testing matrices (Circle can do it with a bit extra config boilerplate) but it doesn't have free artifact storage. We'd have to rely on GitHub releases/tags which doesn't seem like a real problem. Circle also has a number of features I haven't found in Travis, e.g. workspaces and workflows, as well as more out-of-the-box support for Docker which means the install dependencies step for language/pair packages could just not exist (just a reference to the Apertium dev docker tag). That means faster builds and smaller configs. |
What is it we want from CI for apertium-init, apart from running tests (and getting pretty badges)? I've never used circle, but have struggled a bit with travis configs – OTOH, it's typically fairly rare you have to change configs if you just want tests. |
@unhammer we have CI for apertium-init already. I think I was unclear: I'm talking about a default CI config for language pairs or language modules. |
Travis: 5 (https://twitter.com/travisci/status/651856122559774722) That's for the entire org. |
Either will do, but neither will matter when none of the languages or pairs have usable regression test suites. |
🤷♂️ Can't fix that one; at least it's a step in the right direction. IIRC, you found the build step useful when looking at PRs for ltoolbox. This would at least catch malformed XML and simple things like that in PRs. |
Oh right, it also builds branches and PRs. Yeah, that would be worth it. |
as travis-ci user I've few things noted:
I don't know if Circle has similar issues. For travis if you needed artifacts, you can easily do scp in post_success, I do that for continuous deployment. For xmllintigng PRs hound-ci should eventually be sufficient. In summary though I don't have any preference for either solution, just few things to not.e |
Alas, I don't think either solution has 100% uptime and I think it's futile to compare the number of 9s. For a Circle setup though, there would not be any installation step from Tino's packages. The CI config would directly pull down the Docker image (https://github.com/apertium/docker) with apertium-all-dev and related friends already installed and run the tests on it. The only failure point is Docker hub download.
There isn't exactly an equivalent for Circle but you can do the same thing, just with different words since it provides a more generic framework. I do it in e.g. https://github.com/sushain97/web2fs-notepad/blob/b4b48eb335370b70e3e85223c83202beeccb49c3/.circleci/config.yml#L69-L77 and https://github.com/apertium/apertium-html-tools/blob/b258f26c116cecbc03479ea0d085841349b4a923/.circleci/config.yml#L33-L34. Circle offers free artifact storage which is served with the correct content type. Travis requires you to set up your own AWS S3 bucket and serve stuff yourself as well.
I like Hound except that sometimes it just comments way too much and makes actually reviewing the code painful. |
FWIW, Travis had some very serious stability issues in the last couple of weeks. I think it's correlated to them being purchased by some random IB and a large part of their eng team quitting (see https://news.ycombinator.com/item?id=19218036). |
Thanks for sharing this thread with me @sushain97 ! Indeed, we should collaborate on this and add at least a bare minimum of automated tests by default to any (linguistic) package. I've been rambling about this automated testing stuff for quite a while now. Time to gather up the notes (and scripts) and integrate it into something standard and useful I guess. I'll try to put some reference text (say, UDHR and Bible translations, to begin with) under [1] or somewhere close and then make a PR with tests in [2] (but in a way which should work for all languages, of course. There is some ad-hoc stuff in there. Also, I believe that
is absolute mimimum which must run after each commit to a monolingual repo. Travis and CircleWhat bothers me about Circle
What bothers me about Travis
What bothers me about both
From engineering perspective, I believe that a proper way to test linguistic data repos is to have a machine which has core tools installed and which, upon receiving a hook/message that a commit has been made to I tried that at [3]. Works OK, but of course just with a You're right thinking that the biggest downside of this method is that it requires running a server. I had shut down my own server running fitnesse.selimcan.org few weeks ago, with the intention of moving it from DigitalOcean (since some of its IPs stopped working in Russia), but haven't gotten to it yet. Running a server is indeed a hassle which should probably be avoided, given that free CI is available. [1] https://github.com/taruen/apertiumpp/tree/master/data4apertium/corpora |
I think I agree with most of what you've said! Couple things I'm not sure about regarding the mechanics:
Travis is open-source? Not as far as I can tell?
This is just a matter of generating some SVGs. A simple Python script could generate SVGs and insert them into the artifacts. If there's some URL munging required, a simple Python server could handle redirecting URLs or even generating the SVGs in real time. https://github.com/apertium/apertium-stats-service could even handle it.
Should be possible just with a different script for the monolingual repos?
This is almost certainly possible through Travis/Circle APIs but yeah, it's definitely a apertium/lttoolbox specific script.
Spinning up a Docker image with core packages installed is equivalent to booting a VM. Can't really get any less resource intensive than that for a CI service.
This part can be solved with a caching layer. Both Travis and Circle support that.
Having used Jenkins, I would not be interested in supporting it :) IMO, it's better than a custom solution though. It would scale somewhat better because Travis/Circle have limits on free concurrent builds but I'm not sure we hit them anyway. I think that unless we have a workload that requires a custom solution, we shouldn't use one for a couple reasons:
I don't think there's anything in our workload that can't be solved by starting with a Docker image that already has core tools installed and then caching any downloads that are required. Of course, there's overhead in having things virtualized but practically speaking, the performance overhead is negligible and its carbon output overhead in perpetuity is probably far less than the cost of a single steak dinner :) It's basically just the cost of pulling down some bytes from cache. |
Thanks for clearing this up @sushain97 .
Well, should be, they say it is: https://blog.travis-ci.com/2019-01-23-travis-ci-joins-idera-inc Either way, I was inclined to avoid anything custom or Jenkins-based semi-custom, and the arguments you gave are quite convincing. That option is pretty much ruled out now, I think. I'll definitely play around with circleci and try to make it do those 3 bare minimum things .travis.yaml in apertium-tat is currently (half-assedly) doing. I think I've seen you to have both |
Hmm, we do? That seems odd... I think I transitioned from one to another at some point in some repos though.
This would be cool.
I think you meant circleci vs travis here since for travis, they're combining both the .org and .com (the former was reserved for open source and the latter paying customers until recently). |
We now also have github actions to consider as well, which does have a straightforward way to generate custom badges (see https://docs.github.com/en/actions/managing-workflow-runs/adding-a-workflow-status-badge). |
the github action's badges are neat but nowadays also shields.io provides pretty arbitrary badges for free. |
I was looking at those, and my guess would be that maybe the github action could calculate the appropriate numbers and use shields.io to generate the appropriate image and then store it for use in the readme |
yeah that's on my todo list, the way I see it really is is that you have to generate / sed the markdown snippet in readme in the github action which means the github action should probably be able to commit and push and without looping back, I haven't found simple enough example (in actions marketplace or whatever) so I haven't gone down that route yet. |
Upon closer inspection, the link I posted is for a rather more limited feature than I thought. https://github.com/marketplace/actions/bring-your-own-badge is closer, but commits the updated results to a side branch - not sure what to think of that solution |
If https://github.com/apertium/apertium-stats-service worked, it could rather easily provide various badges. APy could also. Or nightly builder. Or Github Action could publish numbers to apertium.org and generate badges that way. Lots of methods to get badges and store them in ways that don't involve committing to the repos. We just need to implement one of them. Stats-service was it, when it worked... |
What broke? I can potentially try and go fix it. |
Don't know. It builds and runs, but does nothing. And only after adjusting the Rust version a few months forward, because as-is it wouldn't even build. But I've just 2 hours ago concluded on IRC that stats-service is superfluous - these kinds of stats can be done better by the build system, so I'll do that. That will also let us generate all the badges we want and host them ourselves. |
Hm, okay. I'll go poke at it.
|
I was going to store them as timestamp + commit hash for that purpose. |
I was going to give my two cents here, but saw the conversation is already going. My first point was going to be this
I can't agree more. Repo metadata shouldn't require commits with data on repos.
I personally like a lot the idea of having all this information available via a REST API.
But I want to reinforce the idea of having data available via APIs is desirable |
FWIW, I plan to look into why |
I just went ahead and bumped the Rust version all the way to today's nightly: apertium/apertium-stats-service@c31f54c. The tests are more or less passing aside from some bitrot due to them depending on actual language modules rather than fixtures. I also verified that the Logs:
If I remember correctly, the If updating works and we still want it, I can switch it to build with GH actions and produce a proper static binary release. The |
No description provided.
The text was updated successfully, but these errors were encountered: