External trigger: Integration test before release?

[pganssle]

I'm not really sure how to read Azure pipeline configurations, but it seems like @gaborbernat (thanks for setting this up, by the way!) has set it up to build on PR and also daily, which I think is a good cadence. However, I'm wondering if we want to (and also if it's even possible to) try and add some sort of hook that would trigger either on demand or one that can be added in to the repos under test.

The reason I'm thinking this might be useful/necessary is that while most projects would not necessarily want to trigger integration test builds on every PR, generally speaking right before a release (particularly an urgent bugfix), you end up tweaking and/or merging a whole bunch of stuff that you want to get into the release. It would be nice to be able to run the integration test suite once you have a release candidate ready but before a release - seems like "right before a release" would be a good time to run the full integration test suite.

I don't have a great sense of how to actually accomplish this. Some options I'm thinking:

1. Integration tests run whenever a new rc release on PyPI is cut (not sure how to register that as an event)
2. This project is made "portable" and the individual repos can check it out in their own CI on whatever schedule they want.
3. This repo somehow watches the project repos for a specific tag or event, or responds to some event that can be kicked off in the project repo CI.

None of these seem great or particularly easy to accomplish. I imagine for now if you want to trigger an integration test build, you can just make a trivial PR to the repo, or each project can make a single trivial PR and whenever they want a build they just close and re-open it. Still, I thought I'd throw it out there to see if 1. this would be useful to other people and 2. if anyone has ideas on how to accomplish it.

[gaborbernat]

@pganssle anyone (with the right usage rights) can kick of manually a one-off run. I imagine we just permission library maintainers to be able to kick off builds manually. Before they want to cut a release.

[gaborbernat]

@zooba has the rights, so just send him your Azure account e-mail addresses and he can add you to the organization.

[gaborbernat]

at https://dev.azure.com/pypa/integration-test/_build?definitionId=15 once you're permissioned:

[zooba]

@gaborbernat should also have the ability to add people. Otherwise I'll take a look when I get into the office today (which, since Seattle has almost an entire inch of snow today, is not guaranteed 😁 )

[techalchemy]

If you need to add people, the org was created from my account and I can add people. I mentioned in the discuss thread that pipenv has pretty substantial integration tests in place that I can help bring over here but it isn't clear whether anyone is interested in that, so for now I guess you can let me know if you need anything modified about the org

[gaborbernat]

We definitely need maintainers to spend some time setting up tests. I know we said we don't want to tie it to tox but I would prefer to have the test easily reproducible locally. As such I would prefer to have it as such. I've tested waters with https://github.com/tox-dev/integration-test, and I would imagine having something similar. Are we in agreement?

PS. The way I wrote that checks out projects via SCM locally and even allows to create fixes in-line, and then one can run it locally to see the test pass. Furthermore, devpi setup and wheel provision are pytest fixtures allowing to be very modular.