Travis Testing with Dependent PRs

[ekilmer]

I think it would be nice to have a way to test multiple PRs that are dependent on each other. For instance, I submitted 1 2 3 PRs for a new feature's logic, testing, and binary for testing, respectively. However, they won't pass Travis testing until they are either merged or a configuration is changed to checkout the respective branch.

I did name the branches the same in each component, so it would be nice if there was way for Travis to automatically pull or checkout the dependent branches in each of the components for testing based on the branch name.

Obviously if two branches are named the same (think master) in different PRs that aren't related, then some errors may result. Contributors would also have to name their branches the same when submitting PRs that depend on each other. Another potential issue would be the age of the PRs and how up to date they are with the current revisions of each component, but this might be easily solved by updating the PR with current component's master. The order in which a PR is submitted might also mess things up, but that job could be manually restarted.

I tried looking for an existing solution from other projects, but I'm not very familiar with Travis or its configuration. I found this article that might be helpful in determining the name of the branch in the PR which could then be used to checkout the respective dependant’s branch : https://graysonkoonce.com/getting-the-current-branch-name-during-a-pull-request-in-travis-ci/

Maybe I don't quite understand how Travis works or this is too complicated or there are other issues that I haven't thought about. I'd be interested in your thoughts.

Thank you!

[rhelmot]

We looked into doing this. The real issue is with how github handles merge requests, i.e. it pre-merges everything and puts the result on a ref, and this is what is used for testing. Correct testing would require not only identifying the alternate repos which needed to be pulled in, but also identifying the alternate pull request number and grabbing the corresponding premerged ref.

It's. Doable, but it would require installing the github API and doing some really horrible stuff I really really do not want to do ever ever ever. In the meantime, prsync.py to do local testing is enough. There's nothing sacred about travis.

[ekilmer]

Sounds good! I should really use the tools in angr-dev to set up my dev environment better... Thanks again.

[zardus]

For the record, if someone did extend the test stuff with github API magic, that'd be great and a really welcome PR. The less work we have to do to run CI, the better :-)