CI: Use PR fork of libmypaint

[odysseywestra]

Description of the problem

I'm noticing that some of the builds are failing due to them not using the Contributor's Fork of libmypaint. Travis-CI and related Continuous Intergration(CI) tools should use the fork first. If the fork of libmypaint is not available, the CI tools should default to upstream.

This can also benefit the use case of Pull Requests for libmypaint as well. Since commit 2612c4a, MyPaint itself has had the ability to generate Appimages on Travis-CI and upload them to transfer.sh. This allows Testers to download the Appimage and review the Pull Request without having to build from source. This functionality could expand to test builds for windows in the future well.

So in this use case, contributors who make a pull request for libmypaint, could make a [WIP] Pull Request on MyPaint itself. This would allow CI tools would build with the Contributor's fork of libmypaint, run tests and transfer builds to transfer.sh so user can download and test the builds.

Affected Pull Requests

@briend - #731 with mypaint/libmypaint#70
@ShadowKyogre - #720 with mypaint/libmypaint#57

Steps to reproduce

1. Create pull request on mypaint/libmypaint with changes to libmypaint
2. Create second pull request on mypaint/mypaint with changes required by libmypaint
3. CI Tools will fail to build on mypaint/mypaint.

Backtraces or error messages

See CI logs from #731

[achadwick]

This used to be solvable with submodules and relative references, amusingly. Although doing a squash-merge would still make a mockery of the contributor's refs, meaning a manual fixup process.

There are several practical challenges to doing this:

• We cannot predict what the contributor's branches are going to be called
  • Does their mypaint branch "something1" require their libmypaint branch "something2"?
  • Or just their libmypaint master?
• We cannot guarantee that their libmypaint master branch will work with their mypaint working branch.
• I prefer just saying "and mypaint will build your libmypaint fork's master if it exists" though.
  • Reason: it's simple to explain. If they get the arrangement of their branches wrong, it's a manual process for redoing the checking builds.
  • Just like now, in fact...
• How to convert the checked-out repo's absolute remote URI for the contributor's mypaint fork into an absolute ref for any libmypaint fork they might happen to have
  • I suppose you start by looking at the output of git remote
  • There is no way of predicting what the CI box's remote is called. Is it always origin? Doesn't have to be.

Not insurmountable, though!

[odysseywestra]

What about using git subtrees?

https://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree/

https://developer.atlassian.com/blog/2015/05/the-power-of-git-subtree/

[odysseywestra]

Nvm, that really won't help.. in our use case subtrees will just create a bigger mess than we need.

Sadly after all the research I did on this, submodules are technically the only way to solve this. Unless I'm missing something.

Something we don't want to use.

It's either that or just have the user manually specify which repository to pull from in the .yaml in a single commit and then revert that commit when it comes to merging in the pull request.

[odysseywestra]

I have an idea.

Instead of the user pull requesting into libmypaint master, what about having them pull request and merge into a testing or unstable branch for example? That way we will know what branch it is in and can point the CI to the correct branch. It may add an extra step in the process, but at least we can have more developers working on a testing branch if they are part of the developers team.

Basically we would be changing our work flow from fork and pull request to branching and pull request.

Okay enough of hair brain ideas from me. Off to bed.

[odysseywestra]

Sorry pressed wrong button.

[odysseywestra]

I guess a good way to combat this would be to adopt git flow style approach. That way we know what the branches names are and keep them the same in both repositories.

The reason I say this is because of this: https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions

[jtojnar]

To sum up the proposed options (adding unmentioned monorepo for completeness):

• add libmypaint as a submodule/subtree to mypaint, and temporarily retarget it to a fork in PR branch
  • annoying: submodule needs to be updated everytime libmypaint is updated
• temporarily copy libmypaint to mypaint repo
  • even more annoying: same as above and we cannot even use git to handle the update
• require merging changes into testing branch in the upstream libmypaint repo
  • slow iteration: requires someone from the project merging
  • multiple people can step on each other toes when developing simultaneously
• merge mypaint, libmypaint and ¿mypaint-brushes? into a single big monorepo
  • best developer experience, easy to run integration CI pipeline in both directions
  • requires users to clone larger git repos (or use subtree clones which is not well known, or use tarballs)
  • since git tags are repo-wide, independent evolution is fuzzy
• test against libmypaint from matching branch in same user’s fork
  • this could work but we would need to check if the branch exists
  • requires discipline since GitHub does not allow renaming branches for open PRs
• require temporarily changing the libmypaint repo URL in a config file
  • I like this the most, it is explicit and understandable

---

If we switch MyPaint to Meson, I think the winning combination might be using a Meson subproject for the dependencies. Specifically a wrap file like this one.

Subprojects allow using dependencies from the system manager while transparently falling back to cloning a git repo when not present. We can specify a branch in the wrap file and if a developer needs a change from their beanch, they could temporarily override it. Delegating it to Meson would allow a single place for the change both in CI and locally for users/developers.

Downside, like with the explicit clone-ing we do now, is that CI runs would not be reproducible. (When we restart the action, someone might have pushed to the libmypaint repo in the meanwhile.) But that is not solvable without pinning like submodules do (which is annoying to manage).

[odysseywestra]

Yeah, the third option would be what I was looking for. That was my original intention of merging initial PRs into Feature Branches. The feature branch can be created after the PR is submitted and the PR can be edited to point to the correct branch after it's created. That will give those in the Core team to prioritize what parts are fully merged into the main branch if they want to chime in. So basically my job todo atm.

When MyPaint was built with SCONS, submodules were used which caused a lot of confusion among developers and thus breaking the build chain. So when we moved to distutils and setup.py, we made it a goal to do away with submodules. Also, we can't merge all three repos into a single repo since other projects use libmypaint(gimp from what I recall) and specifically asked for libmypaint and the brushes to be separated from MyPaint itself.

Meson subproject seems like a good option. Like I said in #1229, as long as we keep it mostly the same process as we have it now with the current build system, that will help foster consistency while also adopting new standards. Yes, we would need to make sure the CI tools use the same git branch in both projects. That will probably be a challenge but should be easier to do since we would be mainly keeping everything within the repos and not using the CI tools except for Hound to deal with the initial PRs. I don't think we need to worry about the brushes since they are mainly config files and modular in a sense.

Hopefully what I'm saying makes sense. I'm mainly just taking what I've observed over time with MyPaint and making sure we don't fall back to methods that didn't work for it when I got involved.

Edit: meant to say third option...

[odysseywestra]

I am working on a new build script that should address this. It works with forks and branches, but I haven't tested it with pull requests yet.

https://github.com/odysseywestra/mypaint/tree/github-actions-script-build

Still needs a lot of work, but I'm making progress.