New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build on official GIMP infrastructure #9

Open
probonopd opened this Issue Apr 24, 2018 · 39 comments

Comments

Projects
None yet
7 participants
@probonopd

probonopd commented Apr 24, 2018

It would be great if this could eventually be made official by the GIMP team. I asked on Twitter, and they said for this to become official, it would need to be built on the official GIMP infrastructure. @schumaml suggested to join #gimp on IRC to kick off the process.

https://twitter.com/probonopd/status/988170965224960000

@probonopd

This comment has been minimized.

Show comment
Hide comment
@probonopd

probonopd commented Apr 26, 2018

@Symbian9

This comment has been minimized.

Show comment
Hide comment
@Symbian9

Symbian9 Apr 27, 2018

@probonopd, ask @prokoudine

P.S.: GIMP 2.10 released!

Symbian9 commented Apr 27, 2018

@probonopd, ask @prokoudine

P.S.: GIMP 2.10 released!

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Apr 27, 2018

Owner

@probonopd I already proposed a couple of times on the gimp_developer mailing list to make my AppImage packages more "official", but never really got feedback on how to proceed in practice.

I need to understand how much work would be to port my Travis CI scripts to the GIMP build environment. If they support Docker containers than it should not be too difficult, as I am using a custom container in the Travis case.

Unfortunately I can rarely spend time on IRC, for me emails work much better because my available time is very scattered.

Owner

aferrero2707 commented Apr 27, 2018

@probonopd I already proposed a couple of times on the gimp_developer mailing list to make my AppImage packages more "official", but never really got feedback on how to proceed in practice.

I need to understand how much work would be to port my Travis CI scripts to the GIMP build environment. If they support Docker containers than it should not be too difficult, as I am using a custom container in the Travis case.

Unfortunately I can rarely spend time on IRC, for me emails work much better because my available time is very scattered.

@prokoudine

This comment has been minimized.

Show comment
Hide comment
@prokoudine

prokoudine Apr 27, 2018

This is unfortunate, because IRC works best for us. I hope schumaml and you can work something out.

prokoudine commented Apr 27, 2018

This is unfortunate, because IRC works best for us. I hope schumaml and you can work something out.

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Apr 27, 2018

Owner

Ok, let's say we could maybe arrange an appointment? I do not want to look like I am trying to block things out... what are the "usual" time slots for GIMP discussions on IRC?

Owner

aferrero2707 commented Apr 27, 2018

Ok, let's say we could maybe arrange an appointment? I do not want to look like I am trying to block things out... what are the "usual" time slots for GIMP discussions on IRC?

@prokoudine

This comment has been minimized.

Show comment
Hide comment
@prokoudine

prokoudine Apr 27, 2018

We are usually around between 6pm and 1am in +2:00GMT (think Berlin). This week is different, because part of the team is at LGM in Spain.

prokoudine commented Apr 27, 2018

We are usually around between 6pm and 1am in +2:00GMT (think Berlin). This week is different, because part of the team is at LGM in Spain.

@Jehan

This comment has been minimized.

Show comment
Hide comment
@Jehan

Jehan Apr 28, 2018

Collaborator

Hi!

I already proposed a couple of times on the gimp_developer mailing list to make my AppImage packages more "official", but never really got feedback on how to proceed in practice.

I must have missed this, mostly because I don't see all ML emails (as many, I'm subscribed to too many lists). The best is to make bug reports, then it won't disappear slowly as time passes. And I read all bug reports (at least the newly created ones).

Basically the requirement to make your AppImage official are:

  • Made with scripts which have to be publicly committed to our repository, and therefore can be reviewed by anyone.
  • Documented (just a "howto" or "README" text file next to the script which explains how to build, or main specific points of the build).
  • Supported! This is maybe the single most important point. When there will be bug reports on our bug tracker, we expect you to fix what needs to be. Of course, everyone has one's own schedule, we are not asking for "busy work schedule", just "don't drop a binary on us then run away", that's all. :-)
    We are also not asking for you to promise to stay with us for dozens of years (life happens! We never know of tomorrow! This is also why the second point "documentation" is important so that someone may continue your work if you could not stay with us). But at least don't plan to abandon us a few weeks after you drop an AppImage on us please.

In other words: don't make our life difficult by giving us more work. Contributing is not about giving code, it is actually about giving time. Many people can do the first, few do the second.

Last point is that ideally builds are made on a "neutral" place, which means done by the automated scripts (which you would have committed in the repo) on a server controlled by the team. This may be a bit difficult since time is a bit missing on our side. Actually we have been accepting binaries built on contributors machine for years, but the reason was also that we have known these contributors for years too (so that's a trust thing).

Anyway as you say:

If they support Docker containers than it should not be too difficult, as I am using a custom container in the Travis case.

I know we can do this (because I have been told so).

So I propose such course of action:

(1) Make a patch against our source tree to add your scripts and howto texts in there (build/appimage/ is where it makes sense) and make a bug report where you'd upload your patch.
(2) Prepare your docker container meant to run your script and we'll see how we can have this run.

Collaborator

Jehan commented Apr 28, 2018

Hi!

I already proposed a couple of times on the gimp_developer mailing list to make my AppImage packages more "official", but never really got feedback on how to proceed in practice.

I must have missed this, mostly because I don't see all ML emails (as many, I'm subscribed to too many lists). The best is to make bug reports, then it won't disappear slowly as time passes. And I read all bug reports (at least the newly created ones).

Basically the requirement to make your AppImage official are:

  • Made with scripts which have to be publicly committed to our repository, and therefore can be reviewed by anyone.
  • Documented (just a "howto" or "README" text file next to the script which explains how to build, or main specific points of the build).
  • Supported! This is maybe the single most important point. When there will be bug reports on our bug tracker, we expect you to fix what needs to be. Of course, everyone has one's own schedule, we are not asking for "busy work schedule", just "don't drop a binary on us then run away", that's all. :-)
    We are also not asking for you to promise to stay with us for dozens of years (life happens! We never know of tomorrow! This is also why the second point "documentation" is important so that someone may continue your work if you could not stay with us). But at least don't plan to abandon us a few weeks after you drop an AppImage on us please.

In other words: don't make our life difficult by giving us more work. Contributing is not about giving code, it is actually about giving time. Many people can do the first, few do the second.

Last point is that ideally builds are made on a "neutral" place, which means done by the automated scripts (which you would have committed in the repo) on a server controlled by the team. This may be a bit difficult since time is a bit missing on our side. Actually we have been accepting binaries built on contributors machine for years, but the reason was also that we have known these contributors for years too (so that's a trust thing).

Anyway as you say:

If they support Docker containers than it should not be too difficult, as I am using a custom container in the Travis case.

I know we can do this (because I have been told so).

So I propose such course of action:

(1) Make a patch against our source tree to add your scripts and howto texts in there (build/appimage/ is where it makes sense) and make a bug report where you'd upload your patch.
(2) Prepare your docker container meant to run your script and we'll see how we can have this run.

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Apr 28, 2018

Owner
Owner

aferrero2707 commented Apr 28, 2018

@Jehan

This comment has been minimized.

Show comment
Hide comment
@Jehan

Jehan Apr 28, 2018

Collaborator

and I have been supporting the packages for several months.

I know and that's a good point for you, because (1) it means you are dedicated (2) your build has been used quite a bit, and that adds to the trust of publishing binaries. :-)

One issue I am aware of and I am trying to fix is the fact that python plugins don't work on some systems.

Ok. Well I don't mind non-perfect builds. As long as we are transparent about the issues (the flatpak also has a few issues; I need to make a post about this) and that we try to improve this with time (continuous support!).

I wonder if the current setup would be acceptable, as it is based on Travis CI and packages are automatically uploaded. No private computer is involved in the process, except for updating the Docker container when needed. Everything is documented in public github repositories, including the Docker configuration, so anybody can in principle take over in case I would not be available for any reason.

I am not an admin, but from what you say, yes it looks great.
Don't take my word for a promise because I am not in charge of server administration, but I could see 2 ways to make the situation viable: (1) either you move your setup to one of our server (2) or you give some other GIMP people access to your server.
We would probably not do much. That's just about "being able to check if something is wrong". Obviously (2) would be the faster solution because it would not rely on some other contributors (who already really lack time) to step in very actively.

Collaborator

Jehan commented Apr 28, 2018

and I have been supporting the packages for several months.

I know and that's a good point for you, because (1) it means you are dedicated (2) your build has been used quite a bit, and that adds to the trust of publishing binaries. :-)

One issue I am aware of and I am trying to fix is the fact that python plugins don't work on some systems.

Ok. Well I don't mind non-perfect builds. As long as we are transparent about the issues (the flatpak also has a few issues; I need to make a post about this) and that we try to improve this with time (continuous support!).

I wonder if the current setup would be acceptable, as it is based on Travis CI and packages are automatically uploaded. No private computer is involved in the process, except for updating the Docker container when needed. Everything is documented in public github repositories, including the Docker configuration, so anybody can in principle take over in case I would not be available for any reason.

I am not an admin, but from what you say, yes it looks great.
Don't take my word for a promise because I am not in charge of server administration, but I could see 2 ways to make the situation viable: (1) either you move your setup to one of our server (2) or you give some other GIMP people access to your server.
We would probably not do much. That's just about "being able to check if something is wrong". Obviously (2) would be the faster solution because it would not rely on some other contributors (who already really lack time) to step in very actively.

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Apr 28, 2018

Owner

I am not an admin, but from what you say, yes it looks great.
Don't take my word for a promise because I am not in charge of server administration, but I could see 2 ways to make the situation viable: (1) either you move your setup to one of our server (2) or you give some other GIMP people access to your server.
We would probably not do much. That's just about "being able to check if something is wrong". Obviously (2) would be the faster solution because it would not rely on some other contributors (who already really lack time) to step in very actively.

I also would prefer solution (2), as it would minimize the amount of work (and I think we are all short in free time). Travis CI is AFAIK a trusted and reliable build infrastructure, and I have no problem at all to give full access to this repository to any interested/relevant GIMP developer.

What is probably missing is some automated mechanism to upload the packages to the GIMP file servers, so that they become easy to access.

Note that I am using a similar setup to maintain the AppImage packages of few other projects, namely:

The first two, together with GIMP, are the more advanced ones (I would say production-ready), the last two still need a bit of polishing (only in the build infrastructure, the AppImage packages are working well in all cases AFAIK).

Owner

aferrero2707 commented Apr 28, 2018

I am not an admin, but from what you say, yes it looks great.
Don't take my word for a promise because I am not in charge of server administration, but I could see 2 ways to make the situation viable: (1) either you move your setup to one of our server (2) or you give some other GIMP people access to your server.
We would probably not do much. That's just about "being able to check if something is wrong". Obviously (2) would be the faster solution because it would not rely on some other contributors (who already really lack time) to step in very actively.

I also would prefer solution (2), as it would minimize the amount of work (and I think we are all short in free time). Travis CI is AFAIK a trusted and reliable build infrastructure, and I have no problem at all to give full access to this repository to any interested/relevant GIMP developer.

What is probably missing is some automated mechanism to upload the packages to the GIMP file servers, so that they become easy to access.

Note that I am using a similar setup to maintain the AppImage packages of few other projects, namely:

The first two, together with GIMP, are the more advanced ones (I would say production-ready), the last two still need a bit of polishing (only in the build infrastructure, the AppImage packages are working well in all cases AFAIK).

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Apr 28, 2018

Owner

Meanwhile, I will put some short documentation of the build process in the README file of the repository.

Owner

aferrero2707 commented Apr 28, 2018

Meanwhile, I will put some short documentation of the build process in the README file of the repository.

@probonopd

This comment has been minimized.

Show comment
Hide comment
@probonopd

probonopd Apr 28, 2018

Glad to see this coming together.

Supported! This is maybe the single most important point. When there will be bug reports on our bug tracker, we expect you to fix what needs to be. Of course, everyone has one's own schedule, we are not asking for "busy work schedule", just "don't drop a binary on us then run away", that's all. :-)

@aferrero2707 has a long-standing, proven track record over at pixls.us where he as been working on GIMP and other AppImages and has been interacting with users very interactively for quite some time. 👍

https://www.google.com/search?q=pixls.us+appimage

So I'm really confident that the opposite of "drop a binary on us then run away" will happen :-)

probonopd commented Apr 28, 2018

Glad to see this coming together.

Supported! This is maybe the single most important point. When there will be bug reports on our bug tracker, we expect you to fix what needs to be. Of course, everyone has one's own schedule, we are not asking for "busy work schedule", just "don't drop a binary on us then run away", that's all. :-)

@aferrero2707 has a long-standing, proven track record over at pixls.us where he as been working on GIMP and other AppImages and has been interacting with users very interactively for quite some time. 👍

https://www.google.com/search?q=pixls.us+appimage

So I'm really confident that the opposite of "drop a binary on us then run away" will happen :-)

@schumaml

This comment has been minimized.

Show comment
Hide comment
@schumaml

schumaml Apr 28, 2018

schumaml commented Apr 28, 2018

@Jehan

This comment has been minimized.

Show comment
Hide comment
@Jehan

Jehan Apr 28, 2018

Collaborator

Travis CI is AFAIK a trusted and reliable build infrastructure, and I have no problem at all to give full access to this repository to any interested/relevant GIMP developer.

Actually after this sentence, I just realized that Travis CI was a third-party service. Somehow I thought (mistakenly) it was just a software for continuous integration (and you installed it on a server of your own).
That makes it even more straightforward as long as the service keeps a trustworthy track.

We'll have to discuss but somehow I consider it may be better to continue delegate all the building part to such third-party service (same as our flatpak is built by flathub, which makes it a lot easier because a lot less to care about). Well… at least in our case where we are lacking a bit of contributor time.

Collaborator

Jehan commented Apr 28, 2018

Travis CI is AFAIK a trusted and reliable build infrastructure, and I have no problem at all to give full access to this repository to any interested/relevant GIMP developer.

Actually after this sentence, I just realized that Travis CI was a third-party service. Somehow I thought (mistakenly) it was just a software for continuous integration (and you installed it on a server of your own).
That makes it even more straightforward as long as the service keeps a trustworthy track.

We'll have to discuss but somehow I consider it may be better to continue delegate all the building part to such third-party service (same as our flatpak is built by flathub, which makes it a lot easier because a lot less to care about). Well… at least in our case where we are lacking a bit of contributor time.

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Apr 28, 2018

Owner

@schumaml

Having a docker file that is set up to install all the required packages, clone and build GIMP and anything else and then the packages would be nice.

There is one in git master already: that one gets Debian testing and then just installs all packages for a GIMP build. Treat it as a proof of concept and not finished in any way, it's just a test to see whether we can build on our virtual machine. It's also not related to build.gimp.org in any way. This was done in about two hours and unless we want to base images on it, its usefulness might be over.

I think that some technical explanation can be useful to explain how things are done.
One of the advantages of the AppImage packages is that they allow to run bleeding-edge software also on relatively old systems, for which up-to-date packages are not available.
This is achieved by compiling the software on the oldest supported system, in the GIMP case Ubuntu 14.04. Due to the lack of up-to-date dependencies, this requires compiling many packages from sources, hence the use of an ad-hoc Docker container. In this case, I am using jhbuild to handle the source dependencies when preparing the container.

The Travis CI configuration fetches the Docker container and the build scripts, and uses the container to run (almost) all the build steps. Hence, each Travis job fetches the BABL/GEGL/GIMP sources as well as several popular GIMP plugins, compiles the code and bundles all the required binaries, libraries and configuration files into the AppImage structure.
Once ready, the AppImage is uploaded back to the dedicated GitHub release page.

The Travis jobs are configured to automatically run once per week. If the build fails I get an email notification, otherwise the new packages automagically appear in the release page without any need of human intervention.

@Jehan my feeling is that relying on a third-party service like Travis is better than using GIMP's own build servers, even just for releasing some unneeded workload from the team...

Owner

aferrero2707 commented Apr 28, 2018

@schumaml

Having a docker file that is set up to install all the required packages, clone and build GIMP and anything else and then the packages would be nice.

There is one in git master already: that one gets Debian testing and then just installs all packages for a GIMP build. Treat it as a proof of concept and not finished in any way, it's just a test to see whether we can build on our virtual machine. It's also not related to build.gimp.org in any way. This was done in about two hours and unless we want to base images on it, its usefulness might be over.

I think that some technical explanation can be useful to explain how things are done.
One of the advantages of the AppImage packages is that they allow to run bleeding-edge software also on relatively old systems, for which up-to-date packages are not available.
This is achieved by compiling the software on the oldest supported system, in the GIMP case Ubuntu 14.04. Due to the lack of up-to-date dependencies, this requires compiling many packages from sources, hence the use of an ad-hoc Docker container. In this case, I am using jhbuild to handle the source dependencies when preparing the container.

The Travis CI configuration fetches the Docker container and the build scripts, and uses the container to run (almost) all the build steps. Hence, each Travis job fetches the BABL/GEGL/GIMP sources as well as several popular GIMP plugins, compiles the code and bundles all the required binaries, libraries and configuration files into the AppImage structure.
Once ready, the AppImage is uploaded back to the dedicated GitHub release page.

The Travis jobs are configured to automatically run once per week. If the build fails I get an email notification, otherwise the new packages automagically appear in the release page without any need of human intervention.

@Jehan my feeling is that relying on a third-party service like Travis is better than using GIMP's own build servers, even just for releasing some unneeded workload from the team...

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Apr 28, 2018

Owner

@Jehan @schumaml by the way, if you are interested we could push the mechanism even further, and do the same for Windows and OSX packages.

Notice that I am routinely building my PhotoFlow Win and OSX packages on Travis, and we have been testing the same for RT Win builds as well.
The Win packages are cross-compiled with ming-w64 using crossroad, which you should know very well ;-).
The OSX ones are based on homebrew, and shipped as a standard OSX application bundle.

I must confess that I am a big fan of such automated package creation, because it releases a lot of repetitive workload from the developers. Do you think it would be worth to open a bug report to discuss about this ideas in a proper way?

Owner

aferrero2707 commented Apr 28, 2018

@Jehan @schumaml by the way, if you are interested we could push the mechanism even further, and do the same for Windows and OSX packages.

Notice that I am routinely building my PhotoFlow Win and OSX packages on Travis, and we have been testing the same for RT Win builds as well.
The Win packages are cross-compiled with ming-w64 using crossroad, which you should know very well ;-).
The OSX ones are based on homebrew, and shipped as a standard OSX application bundle.

I must confess that I am a big fan of such automated package creation, because it releases a lot of repetitive workload from the developers. Do you think it would be worth to open a bug report to discuss about this ideas in a proper way?

@Symbian9 Symbian9 referenced this issue Apr 28, 2018

Open

Setup Travis CI for build DMG (*.dmg) #41

1 of 4 tasks complete
@Jehan

This comment has been minimized.

Show comment
Hide comment
@Jehan

Jehan May 2, 2018

Collaborator

@aferrero2707 We are interested by Windows/macOS continuous integration. This is something we have often be hoping to have one day, but nobody ever had the time to do this (and I fear nobody from current team will ever have if we don't get some new blood).

The Win packages are cross-compiled with ming-w64 using crossroad, which you should know very well ;-).

Ahahah. Yeah, I do. :P
Nice to know this is used. ;-)

As for Win/macOS packages (assuming you mean "stable packages" for releases on our download page, not just CI), we'd be happy to automatize them a bit too though you'd have to discuss with our current Windows installer maintainers (Jernej and Ell) and macOS maintainer (Kris) because a lot of work have been done over the year to make these packages as good as possible (it is not just about building the software). We don't want to waste these efforts. It would be really better if you could come discuss this on IRC.

Collaborator

Jehan commented May 2, 2018

@aferrero2707 We are interested by Windows/macOS continuous integration. This is something we have often be hoping to have one day, but nobody ever had the time to do this (and I fear nobody from current team will ever have if we don't get some new blood).

The Win packages are cross-compiled with ming-w64 using crossroad, which you should know very well ;-).

Ahahah. Yeah, I do. :P
Nice to know this is used. ;-)

As for Win/macOS packages (assuming you mean "stable packages" for releases on our download page, not just CI), we'd be happy to automatize them a bit too though you'd have to discuss with our current Windows installer maintainers (Jernej and Ell) and macOS maintainer (Kris) because a lot of work have been done over the year to make these packages as good as possible (it is not just about building the software). We don't want to waste these efforts. It would be really better if you could come discuss this on IRC.

@Jehan

This comment has been minimized.

Show comment
Hide comment
@Jehan

Jehan May 2, 2018

Collaborator

Let's do one thing at a time: would it be possible to give me keys to the build system for AppImage, just enough to review things and check that everything is sane?
I am pretty sure everything is fine, but with just written discussion, we won't go anywhere. I guess if I can see things, we can go forward.

Collaborator

Jehan commented May 2, 2018

Let's do one thing at a time: would it be possible to give me keys to the build system for AppImage, just enough to review things and check that everything is sane?
I am pretty sure everything is fine, but with just written discussion, we won't go anywhere. I guess if I can see things, we can go forward.

@probonopd

This comment has been minimized.

Show comment
Hide comment
@probonopd

probonopd May 2, 2018

would it be possible to give me keys to the build system for AppImage

What do you mean by "keys to the build system"? The builds are running in public on Travis CI
https://travis-ci.org/aferrero2707/gimp-appimage/
using
https://github.com/aferrero2707/gimp-appimage/blob/master/.travis.yml

probonopd commented May 2, 2018

would it be possible to give me keys to the build system for AppImage

What do you mean by "keys to the build system"? The builds are running in public on Travis CI
https://travis-ci.org/aferrero2707/gimp-appimage/
using
https://github.com/aferrero2707/gimp-appimage/blob/master/.travis.yml

@Jehan

This comment has been minimized.

Show comment
Hide comment
@Jehan

Jehan May 2, 2018

Collaborator

Well I don't really know how things work here. I was thinking that seeing from the inside how it is set up would be helpful. But from what you are saying, I guess we can do this without account, which is even better (the less account the easier).

For the record: I am not an admin and don't know much about CI (apart from what it does, but not how it is set up) and am only doing a review because someone has to step up if we want to get this done. Otherwise, I can tell you: I would prefer to not do this! Coding is 100 times much funnier! :P

Anyway I'll have a look when I can have some time. Hopefully some time soon.

Collaborator

Jehan commented May 2, 2018

Well I don't really know how things work here. I was thinking that seeing from the inside how it is set up would be helpful. But from what you are saying, I guess we can do this without account, which is even better (the less account the easier).

For the record: I am not an admin and don't know much about CI (apart from what it does, but not how it is set up) and am only doing a review because someone has to step up if we want to get this done. Otherwise, I can tell you: I would prefer to not do this! Coding is 100 times much funnier! :P

Anyway I'll have a look when I can have some time. Hopefully some time soon.

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 May 4, 2018

Owner

@Jehan I have updated the README.md file of the repository to describe the current building procedure, and added you as a collaborator.

Please do not hesitate to ask questions if something is not clear...

Owner

aferrero2707 commented May 4, 2018

@Jehan I have updated the README.md file of the repository to describe the current building procedure, and added you as a collaborator.

Please do not hesitate to ask questions if something is not clear...

@Jehan

This comment has been minimized.

Show comment
Hide comment
@Jehan

Jehan May 31, 2018

Collaborator

Hi @aferrero2707 ! I really hoped I could make some time to review and understand how this all works, but history shows us I have not been able in the last month. :P

I am unsure when I will, but I would really suggest you to come on IRC and discuss with other contributors. In particular maybe with Sam Gleske (samrocketman on IRC) who is our CI guy and is very active lately.
Or maybe you could make a patch/merge request with all the files which need to go in our repository (put them under build/appimage/, you'll see there are already various build directories under build/: for flatpak, Windows, etc.) and a good overview on how this would work for the AppImage.

Though really we would also appreciate if you could come to IRC sometimes too. I am not a big IRC person as well, and therefore am not there all the time. But I still make appearances a few times a week because I just know it makes things easier. If we include your AppImage officially, we'll work as a team, which means you'll have to support it in our gitlab, and if sometimes we need to discuss things, being available on IRC is much better, etc.

Collaborator

Jehan commented May 31, 2018

Hi @aferrero2707 ! I really hoped I could make some time to review and understand how this all works, but history shows us I have not been able in the last month. :P

I am unsure when I will, but I would really suggest you to come on IRC and discuss with other contributors. In particular maybe with Sam Gleske (samrocketman on IRC) who is our CI guy and is very active lately.
Or maybe you could make a patch/merge request with all the files which need to go in our repository (put them under build/appimage/, you'll see there are already various build directories under build/: for flatpak, Windows, etc.) and a good overview on how this would work for the AppImage.

Though really we would also appreciate if you could come to IRC sometimes too. I am not a big IRC person as well, and therefore am not there all the time. But I still make appearances a few times a week because I just know it makes things easier. If we include your AppImage officially, we'll work as a team, which means you'll have to support it in our gitlab, and if sometimes we need to discuss things, being available on IRC is much better, etc.

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 May 31, 2018

Owner

I will try to show up on IRC during next weekend, if time allows... meanwhile, there is one basic aspect that needs to be sorted out: do we want the AppImage CI script to reside in the GIMP trunk, or in a separate repository?
The second option has the advantage that the same set of scripts can be used to build in parallel several branches, while the first option has the advantage of keeping everything in a single place... For most of the projects I am sticking to external scripts.

Owner

aferrero2707 commented May 31, 2018

I will try to show up on IRC during next weekend, if time allows... meanwhile, there is one basic aspect that needs to be sorted out: do we want the AppImage CI script to reside in the GIMP trunk, or in a separate repository?
The second option has the advantage that the same set of scripts can be used to build in parallel several branches, while the first option has the advantage of keeping everything in a single place... For most of the projects I am sticking to external scripts.

@Jehan

This comment has been minimized.

Show comment
Hide comment
@Jehan

Jehan Jun 4, 2018

Collaborator

Unless the AppImage is a very complicated setup with hundreds of files or whatever (I hope it's not the case! Many files is what would make difficult to review and understand; the flatpak for instance is all in a single, and quite short/understandable file), I'd prefer that the scripts are inside the main GIMP repo. Yet it could be in a separate repository. If so, that should be one hosted by us, alongside the GIMP repo (for instance, we have the gimp-web repo for the website or the gimp-help-2 repo for the documentation, etc.).

The second option has the advantage that the same set of scripts can be used to build in parallel several branches

I don't understand this one. Why wouldn't it be possible to build several branches if the scripts were all in the main GIMP repo. I used to have several flatpak manifests (building different branches, up to 3!) in the GIMP repo, so that I could indeed build various branches.

Collaborator

Jehan commented Jun 4, 2018

Unless the AppImage is a very complicated setup with hundreds of files or whatever (I hope it's not the case! Many files is what would make difficult to review and understand; the flatpak for instance is all in a single, and quite short/understandable file), I'd prefer that the scripts are inside the main GIMP repo. Yet it could be in a separate repository. If so, that should be one hosted by us, alongside the GIMP repo (for instance, we have the gimp-web repo for the website or the gimp-help-2 repo for the documentation, etc.).

The second option has the advantage that the same set of scripts can be used to build in parallel several branches

I don't understand this one. Why wouldn't it be possible to build several branches if the scripts were all in the main GIMP repo. I used to have several flatpak manifests (building different branches, up to 3!) in the GIMP repo, so that I could indeed build various branches.

@schumaml

This comment has been minimized.

Show comment
Hide comment
@schumaml

schumaml Jun 11, 2018

@aferrero2707 has cleaned up the repository root during our discussion on IRC yesterday evening - many thanks for that.

A similar cleanup to the modulesets and patches would make it easier to spot what is happening - most of the contents there are copied straight from the jhbuild and shouldn't be relevant for building GIMP at all.

For an official AppImage, I'd like to see all third-party plug-ins in an optional build step which is skippable.

schumaml commented Jun 11, 2018

@aferrero2707 has cleaned up the repository root during our discussion on IRC yesterday evening - many thanks for that.

A similar cleanup to the modulesets and patches would make it easier to spot what is happening - most of the contents there are copied straight from the jhbuild and shouldn't be relevant for building GIMP at all.

For an official AppImage, I'd like to see all third-party plug-ins in an optional build step which is skippable.

@samrocketman

This comment has been minimized.

Show comment
Hide comment
@samrocketman

samrocketman Jun 12, 2018

@aferrero2707 @probonopd I have documented how to provision Jenkins locally. One of the benefits of the new Jenkins infrastructure is that one is able to have a complete copy of https://build.gimp.org/ running locally on their own development machine. Here are some tips:

TL;DR instructions

Assuming you have Vagrant and VirtualBox installed.

git clone --recursive https://gitlab.gnome.org/World/gimp-ci/jenkins.git
cd jenkins/
export VAGRANT_JENKINS=1
vagrant up
./jenkins_bootstrap.sh

^^ will automatically provision a VM and inside of the VM: install docker, pull down the GIMP docker image, install and start Jenkins. jenkins_bootstrap.sh will configure Jenkins and automatically start the build pipelines to create a GIMP artifact.

Developing locally inside of Vagrant

I still need to document how to develop in Vagrant. But basically you can SSH to the Vagrant VM the following ways.

# SSH into the VM with vagrant command
vagrant ssh
# alternately SSH into vagrant VM using SSH command
vagrant ssh-config > ssh_config
ssh -F ./ssh_config default

If you add ssh_config to ~/.ssh/config you can push and pull git source code to the default host. For example, if I wanted to develop jenkins-dsl I would apply the following patch to https://gitlab.gnome.org/World/gimp-ci/jenkins

diff --git a/configs/job_generator.xml b/configs/job_generator.xml
index 206ae71..cfcb778 100644
--- a/configs/job_generator.xml
+++ b/configs/job_generator.xml
@@ -9,7 +9,7 @@
     <configVersion>2</configVersion>
     <userRemoteConfigs>
       <hudson.plugins.git.UserRemoteConfig>
-        <url>https://gitlab.gnome.org/World/gimp-ci/jenkins-dsl.git</url>
+        <url>/tmp/jenkins-dsl</url>
       </hudson.plugins.git.UserRemoteConfig>
     </userRemoteConfigs>
     <branches>
diff --git a/configs/shared-pipelines.groovy b/configs/shared-pipelines.groovy
index 0e02bdd..e57153d 100644
--- a/configs/shared-pipelines.groovy
+++ b/configs/shared-pipelines.groovy
@@ -5,7 +5,7 @@ pipeline_shared_libraries = [
         'allowVersionOverride': false,
         'includeInChangesets': true,
         'scm': [
-            'remote': 'https://gitlab.gnome.org/World/gimp-ci/jenkins-dsl.git'
+            'remote': '/tmp/jenkins-dsl'
         ]
     ]
 ]

Don't forget to run ./jenkins_bootstrap.sh to update the configuration of Jenkins.

Inside of the Vagrant VM I would create a bare repository at /tmp/jenkins-dsl.

vagrant ssh
mkdir /tmp/jenkins-dsl
cd /tmp/jenkins-dsl
git init --bare

Then in my locally cloned copy of jenkins-dsl I would add it as a development remote to push code for testing out new pipelines.

git clone https://gitlab.gnome.org/World/gimp-ci/jenkins-dsl.git
cd jenkins-dsl/
git remote add vagrant vagrant@default:/tmp/jenkins-dsl
# push your code to the Vagrant VM so Jenkins uses your development copy.
git push vagrant master

samrocketman commented Jun 12, 2018

@aferrero2707 @probonopd I have documented how to provision Jenkins locally. One of the benefits of the new Jenkins infrastructure is that one is able to have a complete copy of https://build.gimp.org/ running locally on their own development machine. Here are some tips:

TL;DR instructions

Assuming you have Vagrant and VirtualBox installed.

git clone --recursive https://gitlab.gnome.org/World/gimp-ci/jenkins.git
cd jenkins/
export VAGRANT_JENKINS=1
vagrant up
./jenkins_bootstrap.sh

^^ will automatically provision a VM and inside of the VM: install docker, pull down the GIMP docker image, install and start Jenkins. jenkins_bootstrap.sh will configure Jenkins and automatically start the build pipelines to create a GIMP artifact.

Developing locally inside of Vagrant

I still need to document how to develop in Vagrant. But basically you can SSH to the Vagrant VM the following ways.

# SSH into the VM with vagrant command
vagrant ssh
# alternately SSH into vagrant VM using SSH command
vagrant ssh-config > ssh_config
ssh -F ./ssh_config default

If you add ssh_config to ~/.ssh/config you can push and pull git source code to the default host. For example, if I wanted to develop jenkins-dsl I would apply the following patch to https://gitlab.gnome.org/World/gimp-ci/jenkins

diff --git a/configs/job_generator.xml b/configs/job_generator.xml
index 206ae71..cfcb778 100644
--- a/configs/job_generator.xml
+++ b/configs/job_generator.xml
@@ -9,7 +9,7 @@
     <configVersion>2</configVersion>
     <userRemoteConfigs>
       <hudson.plugins.git.UserRemoteConfig>
-        <url>https://gitlab.gnome.org/World/gimp-ci/jenkins-dsl.git</url>
+        <url>/tmp/jenkins-dsl</url>
       </hudson.plugins.git.UserRemoteConfig>
     </userRemoteConfigs>
     <branches>
diff --git a/configs/shared-pipelines.groovy b/configs/shared-pipelines.groovy
index 0e02bdd..e57153d 100644
--- a/configs/shared-pipelines.groovy
+++ b/configs/shared-pipelines.groovy
@@ -5,7 +5,7 @@ pipeline_shared_libraries = [
         'allowVersionOverride': false,
         'includeInChangesets': true,
         'scm': [
-            'remote': 'https://gitlab.gnome.org/World/gimp-ci/jenkins-dsl.git'
+            'remote': '/tmp/jenkins-dsl'
         ]
     ]
 ]

Don't forget to run ./jenkins_bootstrap.sh to update the configuration of Jenkins.

Inside of the Vagrant VM I would create a bare repository at /tmp/jenkins-dsl.

vagrant ssh
mkdir /tmp/jenkins-dsl
cd /tmp/jenkins-dsl
git init --bare

Then in my locally cloned copy of jenkins-dsl I would add it as a development remote to push code for testing out new pipelines.

git clone https://gitlab.gnome.org/World/gimp-ci/jenkins-dsl.git
cd jenkins-dsl/
git remote add vagrant vagrant@default:/tmp/jenkins-dsl
# push your code to the Vagrant VM so Jenkins uses your development copy.
git push vagrant master
@samrocketman

This comment has been minimized.

Show comment
Hide comment
@samrocketman

samrocketman commented Jun 12, 2018

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Jun 12, 2018

Owner

@samrocketman from my side, I will continue cleaning up this repository as well as the docker-trusty-gimp one, and introduce the required changes such that the AppImage creation can run entirely inside the docker container.

Once this is ready, it will be possible to run the docker container on any CI system supporting Docker, and this will provide the final AppImage artefact, ready for uploading in some official place.

Owner

aferrero2707 commented Jun 12, 2018

@samrocketman from my side, I will continue cleaning up this repository as well as the docker-trusty-gimp one, and introduce the required changes such that the AppImage creation can run entirely inside the docker container.

Once this is ready, it will be possible to run the docker container on any CI system supporting Docker, and this will provide the final AppImage artefact, ready for uploading in some official place.

@Jehan

This comment has been minimized.

Show comment
Hide comment
@Jehan

Jehan Sep 16, 2018

Collaborator

Hi!

As said on the mailing list, I'd be interested to resume discussing you going upstream. To be clear, one of the main issue is that your build has so many files and scripts that it's hard to review, unlike flatpak or snap where there is basically a single easy manifest file to review (which can be done in a few minutes).

Anyway… can you confirm me the start point is mkappimage.sh? (that's what I understood from .travis.yml)

Also you won't be able to package the additional (non-core) plug-ins when you go upstream. Releasing third-party plug-ins basically makes us liable and we'd have to support these. This is not possible (it is already hard to support our current core plug-ins with the few we are).
You are welcome to continue providing a separate AppImage on the side with all the plug-ins you want, but the official AppImage has to be bare.

Also is the built GIMP as complete as possible? Does it have all the core plug-ins? Looking at build-appimage-centos7.sh, I see very few dependencies installed (but maybe they are installed otherwise).
We want a bare GIMP, yet obviously as full-featured as possible (the flatpak for instance is full-featured, apart from whatever the sandbox prevents or what I could not resolve yet).

About package-appimage.sh, what is this move_blacklisted() function for?
I see you clone the appimage-helper-scripts.git, but for security reason, it is preferred to not depend on non-upstream files (this makes harder to review too, makes non-reproducible builds if the contents change, may break more easily, etc.).
I actually noticed the same issue when calling upload_rotate.sh downloaded remotely as well.
Basically apart from well known projects, it is preferred to not download remote files in build scripts.

Also as a general rule, even when downloading from known upstream location, you should check the downloaded files. In particular it means to check a tarball against an expected checksum (SHA256 or similar preferably) and to check commit hashes too when cloning git repositories.
Currently your script are completely subject to man-in-the-middle attacks and this won't do.

Moreover why are you saying our "desktop file is a bit strange"? Our desktop file is validated in unit tests and is perfectly fine AFAIK. And even if it is not (bugs are always possible!), then we must fix whatever needs to be. Don't just override our desktop file. Even more as it took a lot of work and dedication from our translators to translate all fields in 81 languages and you are basically just throwing away all this work on a whim. So please use the official desktop file (and let's figure out any issue together if needed) for official builds.
Also don't rename the application. In the desktop file, it is called "GNU Image Manipulation Program" not "GIMP AppImage".

I will have probably a lot more to say. As I said, there are so many files, and I have not really looked in details to how there are all linked. This comment is only the result of a really quick review based on a 5-minute check, following the execution flow starting from .travis.yml (as you say this is the starting point of Travis build in your README). Some of the files, I haven't even seen yet when you run them.

Anyway that's already quite a lot to improve/clean/clarify.

Collaborator

Jehan commented Sep 16, 2018

Hi!

As said on the mailing list, I'd be interested to resume discussing you going upstream. To be clear, one of the main issue is that your build has so many files and scripts that it's hard to review, unlike flatpak or snap where there is basically a single easy manifest file to review (which can be done in a few minutes).

Anyway… can you confirm me the start point is mkappimage.sh? (that's what I understood from .travis.yml)

Also you won't be able to package the additional (non-core) plug-ins when you go upstream. Releasing third-party plug-ins basically makes us liable and we'd have to support these. This is not possible (it is already hard to support our current core plug-ins with the few we are).
You are welcome to continue providing a separate AppImage on the side with all the plug-ins you want, but the official AppImage has to be bare.

Also is the built GIMP as complete as possible? Does it have all the core plug-ins? Looking at build-appimage-centos7.sh, I see very few dependencies installed (but maybe they are installed otherwise).
We want a bare GIMP, yet obviously as full-featured as possible (the flatpak for instance is full-featured, apart from whatever the sandbox prevents or what I could not resolve yet).

About package-appimage.sh, what is this move_blacklisted() function for?
I see you clone the appimage-helper-scripts.git, but for security reason, it is preferred to not depend on non-upstream files (this makes harder to review too, makes non-reproducible builds if the contents change, may break more easily, etc.).
I actually noticed the same issue when calling upload_rotate.sh downloaded remotely as well.
Basically apart from well known projects, it is preferred to not download remote files in build scripts.

Also as a general rule, even when downloading from known upstream location, you should check the downloaded files. In particular it means to check a tarball against an expected checksum (SHA256 or similar preferably) and to check commit hashes too when cloning git repositories.
Currently your script are completely subject to man-in-the-middle attacks and this won't do.

Moreover why are you saying our "desktop file is a bit strange"? Our desktop file is validated in unit tests and is perfectly fine AFAIK. And even if it is not (bugs are always possible!), then we must fix whatever needs to be. Don't just override our desktop file. Even more as it took a lot of work and dedication from our translators to translate all fields in 81 languages and you are basically just throwing away all this work on a whim. So please use the official desktop file (and let's figure out any issue together if needed) for official builds.
Also don't rename the application. In the desktop file, it is called "GNU Image Manipulation Program" not "GIMP AppImage".

I will have probably a lot more to say. As I said, there are so many files, and I have not really looked in details to how there are all linked. This comment is only the result of a really quick review based on a 5-minute check, following the execution flow starting from .travis.yml (as you say this is the starting point of Travis build in your README). Some of the files, I haven't even seen yet when you run them.

Anyway that's already quite a lot to improve/clean/clarify.

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Sep 17, 2018

Owner

@Jehan
Let's go through your questions and comments, one-by-one...

can you confirm me the start point is mkappimage.sh?

Yes, that's correct.

You are welcome to continue providing a separate AppImage on the side with all the plug-ins you want, but the official AppImage has to be bare.

This is already the case for the latest packages. For example, GIMP_AppImage-git-2.10.7-20180913-x86_64.AppImage is the bare package, while GIMP_AppImage-git-2.10.7-withplugins-20180913-x86_64.AppImage ships additional plug-ins.

About package-appimage.sh, what is this move_blacklisted() function for?

Obsolete, I will remove it

I see you clone the appimage-helper-scripts.git, but for security reason, it is preferred to not depend on non-upstream files

I will incorporate the scripts directly into the gimp-appimage repository, so that changes can be traced more easily.

I actually noticed the same issue when calling upload_rotate.sh downloaded remotely as well.

This helper script is only needed to upload the packages from Travis and back to GitHub. If we go for a different CI/hosting system, this will certainly change. But I can add this script to the gimp-appimage repository as well, if preferred.

to check commit hashes too when cloning git repositories.

Is this also needed when cloning the git HEAD? In this case the commit hash is not known/predictable...
For example, jhbuild allows to check the hash of downloaded tar archives, but AFAIK considers git clones as trustworthy.

why are you saying our "desktop file is a bit strange"?

Again, I suspect this is some old relic. I will test with the official desktop file and see if I get problems. Probably not...

Let me do the clean-up, I will post a message here when finished.

Anyway, it will be good if we can progress on this!

Owner

aferrero2707 commented Sep 17, 2018

@Jehan
Let's go through your questions and comments, one-by-one...

can you confirm me the start point is mkappimage.sh?

Yes, that's correct.

You are welcome to continue providing a separate AppImage on the side with all the plug-ins you want, but the official AppImage has to be bare.

This is already the case for the latest packages. For example, GIMP_AppImage-git-2.10.7-20180913-x86_64.AppImage is the bare package, while GIMP_AppImage-git-2.10.7-withplugins-20180913-x86_64.AppImage ships additional plug-ins.

About package-appimage.sh, what is this move_blacklisted() function for?

Obsolete, I will remove it

I see you clone the appimage-helper-scripts.git, but for security reason, it is preferred to not depend on non-upstream files

I will incorporate the scripts directly into the gimp-appimage repository, so that changes can be traced more easily.

I actually noticed the same issue when calling upload_rotate.sh downloaded remotely as well.

This helper script is only needed to upload the packages from Travis and back to GitHub. If we go for a different CI/hosting system, this will certainly change. But I can add this script to the gimp-appimage repository as well, if preferred.

to check commit hashes too when cloning git repositories.

Is this also needed when cloning the git HEAD? In this case the commit hash is not known/predictable...
For example, jhbuild allows to check the hash of downloaded tar archives, but AFAIK considers git clones as trustworthy.

why are you saying our "desktop file is a bit strange"?

Again, I suspect this is some old relic. I will test with the official desktop file and see if I get problems. Probably not...

Let me do the clean-up, I will post a message here when finished.

Anyway, it will be good if we can progress on this!

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Sep 17, 2018

Owner

@Jehan
regarding the functionalities that are enabled in the GIMP version shipped with the AppImage, this is indeed something to be checked in detail.

I guess that the most useful information comes from the messages provided by the configure script, both for GEGL and GIMP.
You can access an example of a recent build log here.

Here are few things I noticed.
GEGL:

checking for MRG... no
checking for LENSFUN... no
checking for V4L2... no
checking for sdl-config... no
checking for LIBRAW... no
checking for dot... no
checking for LUA... no
checking for V4l... no
checking for run_spiro in -lspiro... no
checking for WEBP... no
checking for P2TC... no
checking for w3m... no

and in the summary

Optional features:
  GEGL docs:       no
  Build workshop:  no
  Build website:   no  (asciidoc not found)
  SIMD:            sse:yes mmx:yes
  Vala support:    yes

Optional dependencies:
  asciidoc:        no  (asciidoc not found)
  enscript:        no  (enscript not found)
  mrg:             no  (mrg not found)
  Ruby:            no
  Lua:             no  (usable lua not found)
  Cairo:           yes
  Pango:           yes
  pangocairo:      yes
  GDKPixbuf:       yes
  JPEG:            yes
  PNG:             yes
  OpenEXR:         yes
  rsvg:            yes
  SDL:             no  (SDL library not found)
  libraw:          no  (libraw library not found)
  Jasper:          yes
  graphviz:        no  (graphviz not found)
  avformat:        no  (disabled)
  V4L:             no   (v4l library not found)
  V4L2:            no  (usable libv4l2 not found)
  spiro:           no  (usable SPIRO library not found)
  EXIV:            yes
  gexiv2:          yes
  umfpack:         yes
  TIFF             yes
  webp:            no  (webp library not found)
  poly2tri-c:      yes (internal)

GIMP:

checking for WEBP... no
checking for LIBHEIF... no
checking for LIBUNWIND... no
checking for WEBKIT... no
checking for libwmf-config... no
checking for OPENJPEG... no
checking for perl5... no
checking for perl... /usr/bin/perl
checking for perl version... 5.16.3
checking for xvfb-run... no
checking for appstream-util... no
checking for desktop-file-validate... no

and in the summary

Optional Features:
  Language selection:        no (iso-codes package not found)
  Vector icons:              yes
  Dr. Mingw (Win32):         no
  Relocatable bundle:        no
  Default ICC directory:     /usr/share/color/icc
  Debug console (Win32):     no
  32-bit DLL folder (Win32): no
  Detailed backtraces:       no (libunwind not found)

Optional Plug-Ins:
  Ascii Art:                 no (AA library not found)
  Ghostscript:               yes
  Help Browser:              no (WebKit not found)
  JPEG 2000:                 no (OpenJPEG not found)
  MNG:                       no (MNG header file not found)
  OpenEXR:                   yes
  WebP:                      no (WebP not found)
  Heif:                      no (libheif not found)
  PDF (export):              yes
  Print:                     yes
  Python 2:                  yes
  TWAIN (Win32):             no
  Webpage:                   no (WebKit not found)
  WMF:                       no (libwmf not found)
  X11 Mouse Cursor:          yes
  XPM:                       no (XPM library not found)
  Email:                     needs runtime dependency: xdg-email

Optional Modules:
  ALSA (MIDI Input):         no (libasound not found or unusable)
  Linux Input:               yes (GUdev support: no (libgudev not found))
  DirectInput (Win32):       no

Tests:
  Use xvfb-run               no (not found)
  Test appdata               no (appstream-util not found)
  Test desktop file          no (desktop-file-validate not found)

I will look in detail into each of them.

Owner

aferrero2707 commented Sep 17, 2018

@Jehan
regarding the functionalities that are enabled in the GIMP version shipped with the AppImage, this is indeed something to be checked in detail.

I guess that the most useful information comes from the messages provided by the configure script, both for GEGL and GIMP.
You can access an example of a recent build log here.

Here are few things I noticed.
GEGL:

checking for MRG... no
checking for LENSFUN... no
checking for V4L2... no
checking for sdl-config... no
checking for LIBRAW... no
checking for dot... no
checking for LUA... no
checking for V4l... no
checking for run_spiro in -lspiro... no
checking for WEBP... no
checking for P2TC... no
checking for w3m... no

and in the summary

Optional features:
  GEGL docs:       no
  Build workshop:  no
  Build website:   no  (asciidoc not found)
  SIMD:            sse:yes mmx:yes
  Vala support:    yes

Optional dependencies:
  asciidoc:        no  (asciidoc not found)
  enscript:        no  (enscript not found)
  mrg:             no  (mrg not found)
  Ruby:            no
  Lua:             no  (usable lua not found)
  Cairo:           yes
  Pango:           yes
  pangocairo:      yes
  GDKPixbuf:       yes
  JPEG:            yes
  PNG:             yes
  OpenEXR:         yes
  rsvg:            yes
  SDL:             no  (SDL library not found)
  libraw:          no  (libraw library not found)
  Jasper:          yes
  graphviz:        no  (graphviz not found)
  avformat:        no  (disabled)
  V4L:             no   (v4l library not found)
  V4L2:            no  (usable libv4l2 not found)
  spiro:           no  (usable SPIRO library not found)
  EXIV:            yes
  gexiv2:          yes
  umfpack:         yes
  TIFF             yes
  webp:            no  (webp library not found)
  poly2tri-c:      yes (internal)

GIMP:

checking for WEBP... no
checking for LIBHEIF... no
checking for LIBUNWIND... no
checking for WEBKIT... no
checking for libwmf-config... no
checking for OPENJPEG... no
checking for perl5... no
checking for perl... /usr/bin/perl
checking for perl version... 5.16.3
checking for xvfb-run... no
checking for appstream-util... no
checking for desktop-file-validate... no

and in the summary

Optional Features:
  Language selection:        no (iso-codes package not found)
  Vector icons:              yes
  Dr. Mingw (Win32):         no
  Relocatable bundle:        no
  Default ICC directory:     /usr/share/color/icc
  Debug console (Win32):     no
  32-bit DLL folder (Win32): no
  Detailed backtraces:       no (libunwind not found)

Optional Plug-Ins:
  Ascii Art:                 no (AA library not found)
  Ghostscript:               yes
  Help Browser:              no (WebKit not found)
  JPEG 2000:                 no (OpenJPEG not found)
  MNG:                       no (MNG header file not found)
  OpenEXR:                   yes
  WebP:                      no (WebP not found)
  Heif:                      no (libheif not found)
  PDF (export):              yes
  Print:                     yes
  Python 2:                  yes
  TWAIN (Win32):             no
  Webpage:                   no (WebKit not found)
  WMF:                       no (libwmf not found)
  X11 Mouse Cursor:          yes
  XPM:                       no (XPM library not found)
  Email:                     needs runtime dependency: xdg-email

Optional Modules:
  ALSA (MIDI Input):         no (libasound not found or unusable)
  Linux Input:               yes (GUdev support: no (libgudev not found))
  DirectInput (Win32):       no

Tests:
  Use xvfb-run               no (not found)
  Test appdata               no (appstream-util not found)
  Test desktop file          no (desktop-file-validate not found)

I will look in detail into each of them.

@Jehan

This comment has been minimized.

Show comment
Hide comment
@Jehan

Jehan Sep 17, 2018

Collaborator

The GEGL output is less important. You may have unneeded things there. So focus on GIMP configure output first. As I feared, you have barely any of the optional feature. :-/

Note: on GEGL side, there is at least one optional feature which is not checked in GIMP configure (and should, I'll update this): it needs libumfpack (SuiteSparse) dependency to build "gegl:matting-levin" operation (though you have this one already, so it's all good).

Collaborator

Jehan commented Sep 17, 2018

The GEGL output is less important. You may have unneeded things there. So focus on GIMP configure output first. As I feared, you have barely any of the optional feature. :-/

Note: on GEGL side, there is at least one optional feature which is not checked in GIMP configure (and should, I'll update this): it needs libumfpack (SuiteSparse) dependency to build "gegl:matting-levin" operation (though you have this one already, so it's all good).

@Jehan

This comment has been minimized.

Show comment
Hide comment
@Jehan

Jehan Sep 17, 2018

Collaborator

Is this also needed when cloning the git HEAD? In this case the commit hash is not known/predictable...

Yeah git HEAD obviously cannot be predicted. That would be a dev package anyway, which is more or less labelled "at your own risk". :-)
I am mostly focusing on the stable package first.

Let me do the clean-up, I will post a message here when finished.

Cool. :-)

Collaborator

Jehan commented Sep 17, 2018

Is this also needed when cloning the git HEAD? In this case the commit hash is not known/predictable...

Yeah git HEAD obviously cannot be predicted. That would be a dev package anyway, which is more or less labelled "at your own risk". :-)
I am mostly focusing on the stable package first.

Let me do the clean-up, I will post a message here when finished.

Cool. :-)

@samrocketman

This comment has been minimized.

Show comment
Hide comment
@samrocketman

samrocketman Sep 17, 2018

It may be worth looking at the gimp-docker image which has installed packages for optional features.

https://gitlab.gnome.org/World/gimp-ci/docker-gimp/blob/master/debian-testing/Dockerfile

Noted as GIMP add-on dependencies.

The best thing you can do is wrap your process in a docker image like the other GIMP CI builds. It will be more clear how they work since the infrastructure will live as code. We’ll also be able to publish you docker image to Dockerhub so people can pull it down and build it themselves.

samrocketman commented Sep 17, 2018

It may be worth looking at the gimp-docker image which has installed packages for optional features.

https://gitlab.gnome.org/World/gimp-ci/docker-gimp/blob/master/debian-testing/Dockerfile

Noted as GIMP add-on dependencies.

The best thing you can do is wrap your process in a docker image like the other GIMP CI builds. It will be more clear how they work since the infrastructure will live as code. We’ll also be able to publish you docker image to Dockerhub so people can pull it down and build it themselves.

@probonopd

This comment has been minimized.

Show comment
Hide comment
@probonopd

probonopd Sep 18, 2018

Seeing you collaborate on this makes me happy. Thanks for doing it. 👍

Yeah git HEAD obviously cannot be predicted. That would be a dev package anyway, which is more or less labelled "at your own risk". :-)

Imho AppImages are especially useful for the testing of nightly or continuous builds, especially when used in conjunction with AppImageUpdate binary delta updates (just download a few MB rather the whole thing to go from build to build).

I am mostly focusing on the stable package first.

Agree, let's go step-by-step (but I'm super excited to eventually see the continuous builds, too. Should make the implement/test/feedback cycle much more "fluid").

probonopd commented Sep 18, 2018

Seeing you collaborate on this makes me happy. Thanks for doing it. 👍

Yeah git HEAD obviously cannot be predicted. That would be a dev package anyway, which is more or less labelled "at your own risk". :-)

Imho AppImages are especially useful for the testing of nightly or continuous builds, especially when used in conjunction with AppImageUpdate binary delta updates (just download a few MB rather the whole thing to go from build to build).

I am mostly focusing on the stable package first.

Agree, let's go step-by-step (but I'm super excited to eventually see the continuous builds, too. Should make the implement/test/feedback cycle much more "fluid").

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Sep 18, 2018

Owner

@samrocketman

The best thing you can do is wrap your process in a docker image like the other GIMP CI builds.

This is already the case, as you can see in the .travis.yml file: https://github.com/aferrero2707/gimp-appimage/blob/master/.travis.yml#L21
The whole building/packaging process runs in a custom Docker image, based on CentOS 7 with additional up-to-date dependencies compiled from sources, which is already available on DockerHub.
The configuration for the custom Docker image is also available on GitHub here.

This should allow for a smooth transition to any other CI environment that supports Docker images...

Owner

aferrero2707 commented Sep 18, 2018

@samrocketman

The best thing you can do is wrap your process in a docker image like the other GIMP CI builds.

This is already the case, as you can see in the .travis.yml file: https://github.com/aferrero2707/gimp-appimage/blob/master/.travis.yml#L21
The whole building/packaging process runs in a custom Docker image, based on CentOS 7 with additional up-to-date dependencies compiled from sources, which is already available on DockerHub.
The configuration for the custom Docker image is also available on GitHub here.

This should allow for a smooth transition to any other CI environment that supports Docker images...

@samrocketman

This comment has been minimized.

Show comment
Hide comment
@samrocketman

samrocketman Sep 19, 2018

like the other GIMP CI builds

I’m referring to the sources published on GitLab. The GIMP CI infrastucture can be provisioned locally on a laptop for testing (including adding official builds). It’s good that you already have the docker bits sorted out.

When I mentioned publishing to dockerhub I was referring to GIMP official dockerhub.

https://hub.docker.com/u/gimp/

samrocketman commented Sep 19, 2018

like the other GIMP CI builds

I’m referring to the sources published on GitLab. The GIMP CI infrastucture can be provisioned locally on a laptop for testing (including adding official builds). It’s good that you already have the docker bits sorted out.

When I mentioned publishing to dockerhub I was referring to GIMP official dockerhub.

https://hub.docker.com/u/gimp/

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Sep 21, 2018

Owner

@Jehan
I added most of the missing dependencies, and now GIMP seems to be quite full-featured:

Extra Binaries:
  gimp-console:              yes

Optional Features:
  Levin matting engine:      yes
  Language selection:        yes
  Vector icons:              yes
  Dr. Mingw (Win32):         no
  Relocatable bundle:        no
  Default ICC directory:     /usr/share/color/icc
  Debug console (Win32):     no
  32-bit DLL folder (Win32): no
  Detailed backtraces:       yes

Optional Plug-Ins:
  Ascii Art:                 no (AA library not found)
  Ghostscript:               yes
  Help Browser:              no (WebKit not found)
  JPEG 2000:                 yes
  MNG:                       yes
  OpenEXR:                   yes
  WebP:                      yes
  Heif:                      yes
  PDF (export):              yes
  Print:                     yes
  Python 2:                  yes
  TWAIN (Win32):             no
  Webpage:                   no (WebKit not found)
  WMF:                       yes
  X11 Mouse Cursor:          yes
  XPM:                       yes
  Email:                     needs runtime dependency: xdg-email

Optional Modules:
  ALSA (MIDI Input):         no (libasound not found or unusable)
  Linux Input:               yes (GUdev support: no (libgudev not found))
  DirectInput (Win32):       no

Tests:
  Use xvfb-run               no (not found)
  Test appdata               no (appstream-util not found)
  Test desktop file          no (desktop-file-validate not found)

I have also removed all the external dependencies from the build/packaging scripts, so they should be ready to be reviewed if you have the time.
I prefer to avoid a single, monolithic scripts, because I find it less readable. Some specific steps, like the bundling of GTK and Python, are currently put in separate scripts that are invoked by the main one.

Owner

aferrero2707 commented Sep 21, 2018

@Jehan
I added most of the missing dependencies, and now GIMP seems to be quite full-featured:

Extra Binaries:
  gimp-console:              yes

Optional Features:
  Levin matting engine:      yes
  Language selection:        yes
  Vector icons:              yes
  Dr. Mingw (Win32):         no
  Relocatable bundle:        no
  Default ICC directory:     /usr/share/color/icc
  Debug console (Win32):     no
  32-bit DLL folder (Win32): no
  Detailed backtraces:       yes

Optional Plug-Ins:
  Ascii Art:                 no (AA library not found)
  Ghostscript:               yes
  Help Browser:              no (WebKit not found)
  JPEG 2000:                 yes
  MNG:                       yes
  OpenEXR:                   yes
  WebP:                      yes
  Heif:                      yes
  PDF (export):              yes
  Print:                     yes
  Python 2:                  yes
  TWAIN (Win32):             no
  Webpage:                   no (WebKit not found)
  WMF:                       yes
  X11 Mouse Cursor:          yes
  XPM:                       yes
  Email:                     needs runtime dependency: xdg-email

Optional Modules:
  ALSA (MIDI Input):         no (libasound not found or unusable)
  Linux Input:               yes (GUdev support: no (libgudev not found))
  DirectInput (Win32):       no

Tests:
  Use xvfb-run               no (not found)
  Test appdata               no (appstream-util not found)
  Test desktop file          no (desktop-file-validate not found)

I have also removed all the external dependencies from the build/packaging scripts, so they should be ready to be reviewed if you have the time.
I prefer to avoid a single, monolithic scripts, because I find it less readable. Some specific steps, like the bundling of GTK and Python, are currently put in separate scripts that are invoked by the main one.

@aferrero2707

This comment has been minimized.

Show comment
Hide comment
@aferrero2707

aferrero2707 Oct 9, 2018

Owner

@Jehan
I recently did some further cleanup of the build scripts. The main building/packaging script is now called mkappimage.sh, which in turn calls one script for building the sources (build-appimage.sh) and one for generating the AppImage bundle (package-appimage.sh).

How would you suggest to proceed? I really would like to see this making its way upstream, if it is considered robust and reliable enough...

Owner

aferrero2707 commented Oct 9, 2018

@Jehan
I recently did some further cleanup of the build scripts. The main building/packaging script is now called mkappimage.sh, which in turn calls one script for building the sources (build-appimage.sh) and one for generating the AppImage bundle (package-appimage.sh).

How would you suggest to proceed? I really would like to see this making its way upstream, if it is considered robust and reliable enough...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment