get-pip needs some tests

[pfmoore]

We need to be testing get-pip to ensure that pip changes don't break it. With the pip 10 release, the following issues weren't caught:

1. The change to pip's internals 😞.
2. The change to warn if Windows users use "pip install -U pip" (we need to test on Windows).
3. The change to requests where they no longer ship cacert.pem.

[pradyunsg]

Quoting myself from IRC:

How does a "cron job" on Travis CI for the get-pip repo to run some basic sanity checks? I guess it could check against pip's master periodically that way and we'd have something to use as a tester after actually generating it.

Thoughts?

[pfmoore]

We'd also need something on appveyor to cover the Windows one I mentioned. Not sure if appveyor also has cron jobs, but if so then yes. We'd need to work out how such a job would trigger a notification, too...

An alternative would be to add a "regenerate and run get-pip" step to the pip test suite. But that may be complicated.

[pfmoore]

Step 1 should probably simply be to write some scripts that test get-pip. Even if they have to be run manually in the first instance, we can work on integrating them into CI once we have them :-)

[dstufft]

I normally manually test them when I regenerate them, which is not great.

[hugovk]

If you are the maintainer of OSS project and your CI workflow requires scheduled builds contact AppVeyor team and briefly describe your needs to request this feature.

https://www.appveyor.com/blog/2017/11/08/sheduled-builds-for-free-accounts/

https://www.appveyor.com/docs/build-configuration/#scheduled-builds

[pfmoore]

Some further notes, on what's needed here.

1. Most of the changes that actually break get-pip.py are changes to pip. We do not want to encounter those issues when we build get-pip.py during a release. So a test done here has a number of issues: (a) it flags up the problem way too late, and (b) it doesn't pinpoint which pip change broke get-pip.py.
2. Even generate.installer only builds get-pip.py from released versions on PyPI. What is actually wanted from a test is a build of get-pip from a local version of pip.
3. We should think carefully about what versions of Python we test on. The copies of get-pip.py for Python 2.6 and 3.2 are static, in terms of not changing when new pip versions get released, so we shouldn't test those versions against pip changes.
4. Having automated tests that can be run when building a release, basically automating what @dstufft described above, would be good. See [WIP] Add a very basic test suite #31 for my initial take on that. Putting such tests into CI is quite probably not the right thing to do (I doubt it would do any actual harm, but I'm not sure it would help much, either).

For the record, just because I created the issue here didn't mean that I felt that CI on this repository was the answer - and the above comments explain why.

My plan is as follows:

1. Get the tests in [WIP] Add a very basic test suite #31 tidied up and maybe extended a little. Merge that to provide a standard, but manual, way of checking that "all is OK".
2. Look at adding a functional test to pip's test suite that creates a get-pip.py and tests that it works.
3. Consider whether adding the tests from (1) should be added to Travis and Appveyor.

[webknjaz]

We can do (2) and run that specific test on daily basis with travis.

[webknjaz]

(4) As an end-user I would like to see some proof that it's not broken right in my browser w/o needing to understand how to set up env locally and run those tests.

[pfmoore]

So your (4) for me counts as a vote in favour of my (3).

We can do (2) and run that specific test on daily basis with travis.

Pip's tests run on every commit and PR. No need for daily runs as well.

[pradyunsg]

pip's tests also run everyday - a "cron" job on Travis.

So, I feel we should have it here too. It's just a switch in the UI so, that's not anything major IMO.

[webknjaz]

@pfmoore @pradyunsg also note that with Build Stages feature there's an if-clause support on stage/job definition levels, which allows to run completely different set of jobs/stages depending on whether it's normal build or it's cron/API/PR-triggered.

[pradyunsg]

#37 is something that tells me that we need to do this ASAP. :)

[pradyunsg]

I think this is resolved with #85.