Question about compatibility and scope of the project with our setup

[adamryczkowski]

Hello,

Your project was referred to me by Hannes Vogt, and I am considering using pycicle in our organization (Polish meteorological services, IMGW) for testing development of our new dynamic core for the weather forecast.

Our project is built and tested exclusively by CMake and CTest (with make and make test). Besides C++ and CUDA it does have Fortran code. The git repository containing the project itself contains git submodules. The project itself is hosted on our internal GitLab instance, and some submodules are on GitHub. We wish to build the project on our supercomputer managed by the Torque (not Slurm), where compute nodes do not have access to the internet. We don't use CDash (but may start using it, if it is recommended).

Do you think this kind of project is within reach of the pycicle (or at least pycicle can be easily expanded to cover it)?

[biddisco]

If the project can be configured with cmake and tests run using ctest, then all should be fine from the point of view of actually performing builds and running tests, so cuda/fortran themselves should not cause any issues beyond the usual pain of getting the builds to work.

Problem 1 will be that if the compute nodes cannot drop an http request onto a cdash server, then you won't get them on a dashboard. This could be fixed, but it would require an extra scrape function. (If you are not using cdash, then I'm not sure how to get the results to a different CI front end).
Pycicle periodically checks the build dirs for the build/test result summaries and uses them to set the build status on github, so that a branch can be marked as good/bad. What we'd need to do is add an extra step to retrieve the xml files generated by ctest and then submit them to a dashboard from the machine running the pycicle 'front end'. This would be an afternoon's work hopefully - should be easy to do, but I've not tried it yet.

Problem 2 is that if you are using gitlab, the pygithub module used by pycicle won't do the right thing. I've not looked to see if there's a drop in replacement that could be used for gitlab instead of github.
([pause], https://github.com/python-gitlab/python-gitlab looks like it could be used).
We only make a half-dozen or so actual calls to pygithub, so it might be that converting to gitlab is quite doable, but I'm not sure until someone tries it.

"Do you think this kind of project is within reach of the pycicle (or at least pycicle can be easily expanded to cover it)?"

I'd say probably yes, the development of pycicle backbone was done mostly over a weekend and has been improved over the last month or so (and I'm not a python programmer). If you are prepared to put in a few days of work (and I will happily help where I can), then it should be doable.

I decided to create pycicle initially because I estimated it'd be quicker to implement something like this, than to get familiar with other CI tools and set them up (accounts/permissions/etc). If you think the same, then I'd say, give it a try.

[biddisco]

I forgot to address the questions of

git submodules : They should not be too much of a problem, the update step is a little complicated at the moment, because I wanted cdash to display the changed files. with submodules, we'd probably need to tweak the update slightly to get things right, and cdash might not show all changed files in subdirs.

Torque: Since all pycicle does is wrap stuff into scripts and then launch them, I would imagine that replacing the slum wrapper generation with torque should be straightforward.

Quite a few new command line options are going to be needed to address all your needs!

[adamryczkowski]

Thank you very much for the info!

[adamryczkowski]

Thank you very much for the info!