Build on official GIMP infrastructure #9
Comments
|
Relevant discussion also here: |
|
@probonopd, ask @prokoudine P.S.: GIMP 2.10 released! |
|
@probonopd I already proposed a couple of times on the gimp_developer mailing list to make my AppImage packages more "official", but never really got feedback on how to proceed in practice. I need to understand how much work would be to port my Travis CI scripts to the GIMP build environment. If they support Docker containers than it should not be too difficult, as I am using a custom container in the Travis case. Unfortunately I can rarely spend time on IRC, for me emails work much better because my available time is very scattered. |
|
This is unfortunate, because IRC works best for us. I hope schumaml and you can work something out. |
|
Ok, let's say we could maybe arrange an appointment? I do not want to look like I am trying to block things out... what are the "usual" time slots for GIMP discussions on IRC? |
|
We are usually around between 6pm and 1am in +2:00GMT (think Berlin). This week is different, because part of the team is at LGM in Spain. |
|
Hi!
I must have missed this, mostly because I don't see all ML emails (as many, I'm subscribed to too many lists). The best is to make bug reports, then it won't disappear slowly as time passes. And I read all bug reports (at least the newly created ones). Basically the requirement to make your AppImage official are:
In other words: don't make our life difficult by giving us more work. Contributing is not about giving code, it is actually about giving time. Many people can do the first, few do the second. Last point is that ideally builds are made on a "neutral" place, which means done by the automated scripts (which you would have committed in the repo) on a server controlled by the team. This may be a bit difficult since time is a bit missing on our side. Actually we have been accepting binaries built on contributors machine for years, but the reason was also that we have known these contributors for years too (so that's a trust thing). Anyway as you say:
I know we can do this (because I have been told so). So I propose such course of action: (1) Make a patch against our source tree to add your scripts and howto texts in there ( |
|
Thanks for the hints!
Regarding the support, at the moment I do not expect to step away, and I
have been supporting the packages for several months.
One issue I am aware of and I am trying to fix is the fact that python
plugins don't work on some systems.
Just to understand, when you say:
" Last point is that ideally builds are made on a "neutral" place, which
means done by the automated scripts (which you would have committed in the
repo) on a server controlled by the team. This may be a bit difficult since
time is a bit missing on our side."
I wonder if the current setup would be acceptable, as it is based on Travis
CI and packages are automatically uploaded. No private computer is involved
in the process, except for updating the Docker container when needed.
Everything is documented in public github repositories, including the
Docker configuration, so anybody can in principle take over in case I would
not be available for any reason.
|
I know and that's a good point for you, because (1) it means you are dedicated (2) your build has been used quite a bit, and that adds to the trust of publishing binaries. :-)
Ok. Well I don't mind non-perfect builds. As long as we are transparent about the issues (the flatpak also has a few issues; I need to make a post about this) and that we try to improve this with time (continuous support!).
I am not an admin, but from what you say, yes it looks great. |
I also would prefer solution (2), as it would minimize the amount of work (and I think we are all short in free time). Travis CI is AFAIK a trusted and reliable build infrastructure, and I have no problem at all to give full access to this repository to any interested/relevant GIMP developer. What is probably missing is some automated mechanism to upload the packages to the GIMP file servers, so that they become easy to access. Note that I am using a similar setup to maintain the AppImage packages of few other projects, namely:
The first two, together with GIMP, are the more advanced ones (I would say production-ready), the last two still need a bit of polishing (only in the build infrastructure, the AppImage packages are working well in all cases AFAIK). |
|
Meanwhile, I will put some short documentation of the build process in the README file of the repository. |
|
Glad to see this coming together.
@aferrero2707 has a long-standing, proven track record over at pixls.us where he as been working on GIMP and other AppImages and has been interacting with users very interactively for quite some time. 👍 https://www.google.com/search?q=pixls.us+appimage So I'm really confident that the opposite of "drop a binary on us then run away" will happen :-) |
|
Having a docker file that is set up to install all the required packages, clone and build GIMP and anything else and then the packages would be nice.
There is one in git master already: that one gets Debian testing and then just installs all packages for a GIMP build. Treat it as a proof of concept and not finished in any way, it's just a test to see whether we can build on our virtual machine. It's also not related to build.gimp.org in any way. This was done in about two hours and unless we want to base images on it, its usefulness might be over.
Do not make this overly complicated - on one end it is just someone without any package system experience, but access to a docker instance which can easily put things to download.gimp.org. On the other it would be people like you who could answrr my stupid questions about why a build of a packaging format does what it is doing.
…On April 28, 2018 4:49:58 PM GMT+02:00, probonopd ***@***.***> wrote:
Glad to see this coming together.
> Supported! This is maybe the single most important point. When there
will be bug reports on our bug tracker, we expect you to fix what needs
to be. Of course, everyone has one's own schedule, we are not asking
for "busy work schedule", just "don't drop a binary on us then run
away", that's all. :-)
@aferrero2707 has a long-standing, proven track record over at pixls.us
where he as been working on GIMP and other AppImages and has been
interacting with users very interactively for quite some time. 👍
https://www.google.com/search?q=pixls.us+appimage
So I'm really confident that the opposite of "drop a binary on us then
run away" will happen :-)
--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#9 (comment)
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
|
Actually after this sentence, I just realized that Travis CI was a third-party service. Somehow I thought (mistakenly) it was just a software for continuous integration (and you installed it on a server of your own). We'll have to discuss but somehow I consider it may be better to continue delegate all the building part to such third-party service (same as our flatpak is built by flathub, which makes it a lot easier because a lot less to care about). Well… at least in our case where we are lacking a bit of contributor time. |
I think that some technical explanation can be useful to explain how things are done. The Travis CI configuration fetches the Docker container and the build scripts, and uses the container to run (almost) all the build steps. Hence, each Travis job fetches the BABL/GEGL/GIMP sources as well as several popular GIMP plugins, compiles the code and bundles all the required binaries, libraries and configuration files into the AppImage structure. The Travis jobs are configured to automatically run once per week. If the build fails I get an email notification, otherwise the new packages automagically appear in the release page without any need of human intervention. @Jehan my feeling is that relying on a third-party service like Travis is better than using GIMP's own build servers, even just for releasing some unneeded workload from the team... |
|
@Jehan @schumaml by the way, if you are interested we could push the mechanism even further, and do the same for Windows and OSX packages. Notice that I am routinely building my PhotoFlow Win and OSX packages on Travis, and we have been testing the same for RT Win builds as well. I must confess that I am a big fan of such automated package creation, because it releases a lot of repetitive workload from the developers. Do you think it would be worth to open a bug report to discuss about this ideas in a proper way? |
|
@aferrero2707 We are interested by Windows/macOS continuous integration. This is something we have often be hoping to have one day, but nobody ever had the time to do this (and I fear nobody from current team will ever have if we don't get some new blood).
Ahahah. Yeah, I do. :P As for Win/macOS packages (assuming you mean "stable packages" for releases on our download page, not just CI), we'd be happy to automatize them a bit too though you'd have to discuss with our current Windows installer maintainers (Jernej and Ell) and macOS maintainer (Kris) because a lot of work have been done over the year to make these packages as good as possible (it is not just about building the software). We don't want to waste these efforts. It would be really better if you could come discuss this on IRC. |
|
Let's do one thing at a time: would it be possible to give me keys to the build system for AppImage, just enough to review things and check that everything is sane? |
What do you mean by "keys to the build system"? The builds are running in public on Travis CI |
|
Well I don't really know how things work here. I was thinking that seeing from the inside how it is set up would be helpful. But from what you are saying, I guess we can do this without account, which is even better (the less account the easier). For the record: I am not an admin and don't know much about CI (apart from what it does, but not how it is set up) and am only doing a review because someone has to step up if we want to get this done. Otherwise, I can tell you: I would prefer to not do this! Coding is 100 times much funnier! :P Anyway I'll have a look when I can have some time. Hopefully some time soon. |
|
@Jehan I have updated the Please do not hesitate to ask questions if something is not clear... |
|
Hi @aferrero2707 ! I really hoped I could make some time to review and understand how this all works, but history shows us I have not been able in the last month. :P I am unsure when I will, but I would really suggest you to come on IRC and discuss with other contributors. In particular maybe with Sam Gleske (samrocketman on IRC) who is our CI guy and is very active lately. Though really we would also appreciate if you could come to IRC sometimes too. I am not a big IRC person as well, and therefore am not there all the time. But I still make appearances a few times a week because I just know it makes things easier. If we include your AppImage officially, we'll work as a team, which means you'll have to support it in our gitlab, and if sometimes we need to discuss things, being available on IRC is much better, etc. |
|
I will try to show up on IRC during next weekend, if time allows... meanwhile, there is one basic aspect that needs to be sorted out: do we want the AppImage CI script to reside in the GIMP trunk, or in a separate repository? |
|
Unless the AppImage is a very complicated setup with hundreds of files or whatever (I hope it's not the case! Many files is what would make difficult to review and understand; the flatpak for instance is all in a single, and quite short/understandable file), I'd prefer that the scripts are inside the main GIMP repo. Yet it could be in a separate repository. If so, that should be one hosted by us, alongside the GIMP repo (for instance, we have the gimp-web repo for the website or the gimp-help-2 repo for the documentation, etc.).
I don't understand this one. Why wouldn't it be possible to build several branches if the scripts were all in the main GIMP repo. I used to have several flatpak manifests (building different branches, up to 3!) in the GIMP repo, so that I could indeed build various branches. |
|
@aferrero2707 has cleaned up the repository root during our discussion on IRC yesterday evening - many thanks for that. A similar cleanup to the modulesets and patches would make it easier to spot what is happening - most of the contents there are copied straight from the jhbuild and shouldn't be relevant for building GIMP at all. For an official AppImage, I'd like to see all third-party plug-ins in an optional build step which is skippable. |
|
@aferrero2707 @probonopd I have documented how to provision Jenkins locally. One of the benefits of the new Jenkins infrastructure is that one is able to have a complete copy of https://build.gimp.org/ running locally on their own development machine. Here are some tips:
TL;DR instructionsAssuming you have Vagrant and VirtualBox installed. git clone --recursive https://gitlab.gnome.org/World/gimp-ci/jenkins.git
cd jenkins/
export VAGRANT_JENKINS=1
vagrant up
./jenkins_bootstrap.sh^^ will automatically provision a VM and inside of the VM: install docker, pull down the GIMP docker image, install and start Jenkins. Developing locally inside of VagrantI still need to document how to develop in Vagrant. But basically you can SSH to the Vagrant VM the following ways. # SSH into the VM with vagrant command
vagrant ssh
# alternately SSH into vagrant VM using SSH command
vagrant ssh-config > ssh_config
ssh -F ./ssh_config defaultIf you add diff --git a/configs/job_generator.xml b/configs/job_generator.xml
index 206ae71..cfcb778 100644
--- a/configs/job_generator.xml
+++ b/configs/job_generator.xml
@@ -9,7 +9,7 @@
<configVersion>2</configVersion>
<userRemoteConfigs>
<hudson.plugins.git.UserRemoteConfig>
- <url>https://gitlab.gnome.org/World/gimp-ci/jenkins-dsl.git</url>
+ <url>/tmp/jenkins-dsl</url>
</hudson.plugins.git.UserRemoteConfig>
</userRemoteConfigs>
<branches>
diff --git a/configs/shared-pipelines.groovy b/configs/shared-pipelines.groovy
index 0e02bdd..e57153d 100644
--- a/configs/shared-pipelines.groovy
+++ b/configs/shared-pipelines.groovy
@@ -5,7 +5,7 @@ pipeline_shared_libraries = [
'allowVersionOverride': false,
'includeInChangesets': true,
'scm': [
- 'remote': 'https://gitlab.gnome.org/World/gimp-ci/jenkins-dsl.git'
+ 'remote': '/tmp/jenkins-dsl'
]
]
]Don't forget to run Inside of the Vagrant VM I would create a bare repository at Then in my locally cloned copy of jenkins-dsl I would add it as a development remote to push code for testing out new pipelines. git clone https://gitlab.gnome.org/World/gimp-ci/jenkins-dsl.git
cd jenkins-dsl/
git remote add vagrant vagrant@default:/tmp/jenkins-dsl
# push your code to the Vagrant VM so Jenkins uses your development copy.
git push vagrant master |
|
I went ahead and wrote up a local development guide. https://gitlab.gnome.org/World/gimp-ci/documentation/blob/master/develop-locally.md |
|
@samrocketman from my side, I will continue cleaning up this repository as well as the docker-trusty-gimp one, and introduce the required changes such that the AppImage creation can run entirely inside the docker container. Once this is ready, it will be possible to run the docker container on any CI system supporting Docker, and this will provide the final AppImage artefact, ready for uploading in some official place. |
|
Hi! As said on the mailing list, I'd be interested to resume discussing you going upstream. To be clear, one of the main issue is that your build has so many files and scripts that it's hard to review, unlike flatpak or snap where there is basically a single easy manifest file to review (which can be done in a few minutes). Anyway… can you confirm me the start point is Also you won't be able to package the additional (non-core) plug-ins when you go upstream. Releasing third-party plug-ins basically makes us liable and we'd have to support these. This is not possible (it is already hard to support our current core plug-ins with the few we are). Also is the built GIMP as complete as possible? Does it have all the core plug-ins? Looking at About Also as a general rule, even when downloading from known upstream location, you should check the downloaded files. In particular it means to check a tarball against an expected checksum (SHA256 or similar preferably) and to check commit hashes too when cloning git repositories. Moreover why are you saying our "desktop file is a bit strange"? Our desktop file is validated in unit tests and is perfectly fine AFAIK. And even if it is not (bugs are always possible!), then we must fix whatever needs to be. Don't just override our desktop file. Even more as it took a lot of work and dedication from our translators to translate all fields in 81 languages and you are basically just throwing away all this work on a whim. So please use the official desktop file (and let's figure out any issue together if needed) for official builds. I will have probably a lot more to say. As I said, there are so many files, and I have not really looked in details to how there are all linked. This comment is only the result of a really quick review based on a 5-minute check, following the execution flow starting from Anyway that's already quite a lot to improve/clean/clarify. |
|
@Jehan
Yes, that's correct.
This is already the case for the latest packages. For example, GIMP_AppImage-git-2.10.7-20180913-x86_64.AppImage is the bare package, while GIMP_AppImage-git-2.10.7-withplugins-20180913-x86_64.AppImage ships additional plug-ins.
Obsolete, I will remove it
I will incorporate the scripts directly into the gimp-appimage repository, so that changes can be traced more easily.
This helper script is only needed to upload the packages from Travis and back to GitHub. If we go for a different CI/hosting system, this will certainly change. But I can add this script to the gimp-appimage repository as well, if preferred.
Is this also needed when cloning the git HEAD? In this case the commit hash is not known/predictable...
Again, I suspect this is some old relic. I will test with the official desktop file and see if I get problems. Probably not... Let me do the clean-up, I will post a message here when finished. Anyway, it will be good if we can progress on this! |
|
@Jehan I guess that the most useful information comes from the messages provided by the Here are few things I noticed. and in the summary GIMP: and in the summary I will look in detail into each of them. |
|
The GEGL output is less important. You may have unneeded things there. So focus on GIMP configure output first. As I feared, you have barely any of the optional feature. :-/ Note: on GEGL side, there is at least one optional feature which is not checked in GIMP configure (and should, I'll update this): it needs libumfpack (SuiteSparse) dependency to build "gegl:matting-levin" operation (though you have this one already, so it's all good). |
Yeah
Cool. :-) |
|
It may be worth looking at the gimp-docker image which has installed packages for optional features. https://gitlab.gnome.org/World/gimp-ci/docker-gimp/blob/master/debian-testing/Dockerfile Noted as The best thing you can do is wrap your process in a docker image like the other GIMP CI builds. It will be more clear how they work since the infrastructure will live as code. We’ll also be able to publish you docker image to Dockerhub so people can pull it down and build it themselves. |
|
Seeing you collaborate on this makes me happy. Thanks for doing it. 👍
Imho AppImages are especially useful for the testing of nightly or continuous builds, especially when used in conjunction with AppImageUpdate binary delta updates (just download a few MB rather the whole thing to go from build to build).
Agree, let's go step-by-step (but I'm super excited to eventually see the continuous builds, too. Should make the implement/test/feedback cycle much more "fluid"). |
This is already the case, as you can see in the This should allow for a smooth transition to any other CI environment that supports Docker images... |
I’m referring to the sources published on GitLab. The GIMP CI infrastucture can be provisioned locally on a laptop for testing (including adding official builds). It’s good that you already have the docker bits sorted out. When I mentioned publishing to dockerhub I was referring to GIMP official dockerhub. |
|
@Jehan I have also removed all the external dependencies from the build/packaging scripts, so they should be ready to be reviewed if you have the time. |
|
@Jehan How would you suggest to proceed? I really would like to see this making its way upstream, if it is considered robust and reliable enough... |
|
Great to see this conversation, just curious to learn: what needs to be done next to bring this upstream? |
|
For the record, after meeting @aferrero2707 at LGM, we advanced on the discussion. I have started writing some "How to be an official package maintainer" rules where we will write all expectations (as discussed with @aferrero2707) from such trusted maintenance. Anyway just wait a little bit more. It's all just a question of when I can make the time to finish. :-) |
|
@Jehan indeed we had a very fruitful discussion at LGM, and it was great to sit down and wrap things up together! There is no big hurry, take your time, the packages are being updated in all cases ;-) |
|
The AppImages for the "Libre Grahics Suite" are coming together nicely. I hope the GIMP will make the jump to the top category one day. https://github.com/AppImage/AppImageKit/wiki/Libre-Graphics-Suite |
|
Looks like GIMP is now being built on GitLab CI, which should greatly simplify things. No more custom Jenkins setup to study. https://gitlab.gnome.org/GNOME/gimp/blob/master/.gitlab-ci.yml https://twitter.com/GIMP_Official/status/1190731951080116224
I guess we really can do it now 👍 |
|
Hi all! I'd like to say I am really sorry this whole review thing takes so much time. I'll try to make some time, I really want to. It's just hard to promise a deadline, and I must admit reviewing build scripts is not my favorite thing ever (even though I do far too much of this these last years for the Gitlab CI and for flatpak!) so I procrastinate. Now I have a few questions. I'm sure we must have discussed this earlier (either here in one of the 40+ comments but I am scared to re-read them or when we met at LGM) but if so, I forgot:
|
|
I don't think there is anything on Travis CI that can't be done on GitLab CI. And yes, that would be the plan - a CI AppImage build for every commit, and for every merge request. https://gitlab.com/inkscape/inkscape is already doing it, for example. |
|
@aferrero2707 @probonopd Could you then make a merge request with your build scripts and with the update of the gitlab-ci? Then I'll try to kick myself into doing a proper review not in 10 years. Of course as discussed, it will have to be bare GIMP (no third-party plug-ins; you are welcome to continue doing a with third-party package here 😉) and maximum options ON if possible. And obviously, @aferrero2707, I'll expect you to continue maintaining your code for the AppImage as time goes on. Basically be a part of the GIMP team. :-) |
|
@Jehan @probonopd I confirm that there should be nothing preventing the use of GitLab CI for building the GIMP appimages. The CI script is actually rather simple, as the whole build process runs in a Docker container that can be ported from one system to another. It will probably take the whole week for me to adapt the scripts and configurations to GitLab CI and prepare the merge request, but I promise I will work on that. We will also need to think how to handle the release vs. git head packages, particularly regarding the naming of the output package and the BABL/GEGL versions that are used. |
|
@aferrero2707 did you find some time to look into this? |
|
@schumaml sorry, no time yet to work on this, hopefully I will finally manage to put my hands on it next week... |
|
@aferrero2707 I've made a recipe for the Glimpse project (which uses Gimp as code base), you may want to take a look into it.
This technology also support python and perl for the plugin system. Here are some useful links:
|
It would be great if this could eventually be made official by the GIMP team. I asked on Twitter, and they said for this to become official, it would need to be built on the official GIMP infrastructure. @schumaml suggested to join #gimp on IRC to kick off the process.
https://twitter.com/probonopd/status/988170965224960000
The text was updated successfully, but these errors were encountered: