A new and terrifying age of continuous deployment is upon us #537

DRMacIver · 2017-04-20T11:54:31Z

This pull request closes #408.

I am operating on the thesis that if something is worth doing then it's worth dialling up to 11, releasing the hounds, and then nuking it from orbit (after ensuring the hounds are safely out of the blast radius. Hypothesis does not condone cruelty to animals): All merges to master that are green on Travis (it does not check the other CI, but the other CI had to be green to merge in the first place so I think that's OK) and have any source changes from the last version will be immediately tagged for release and uploaded to pypi.

This PR contains two necessary knock on changes:

It adds some build steps that check that you have updated the version to something that is not tagged, and checks that both the the current date and the current version are present in the changelog (any checking beyond that is up to the reviewer).
It updates the changelog and version to do a release of 3.7.1 today.

I think these changes do not make logical sense to separate (arguably the 3.7.1 change does, but I'd like to actually see this working in practice as soon as we merge it). I am happy to accept arguments to the contrary.

Why do this?

Release automation is an intrinsic good - less scope for me to get things wrong.
This will force us to be much better about changelog management than we have currently been.
I have complete faith in our CI and review processes stopping anything that shouldn't be released making it into master. Therefore anything that is in master and is unreleased is basically awesome stuff that we are withholding from users. That seems like a thing we'd like to avoid.

On a person level, the other motivation for doing this is that I used to be really good about releasing early and releasing often, and due to a variety of reasons have stopped being so. This makes me sad, so I'd like to get back to it, but good intentions are rarely enough to do better: The way to do better is instead to bake it into the process.

A knock-on effect is that depending on the merge order, either #527 needs to be updated to talk about changelogs and versioning for all functionality change or this needs to be updated to include changes to the review list. My vote would be to merge this first and update #527.

Revert "Update version and changelog" This reverts commit 98a3bae.

DRMacIver · 2017-04-20T12:10:00Z

Note that the Travis build is currently failing because I wanted to test the jobs that check the changelog and version (which was good, because I found a bug in them). I'll unrevert the commit updating the changelog and version once I've seen them both failing correctly.

alexwlchan

I am operating on the thesis that if something is worth doing then it's worth dialling up to 11, releasing the hounds, and then nuking it from orbit (after ensuring the hounds are safely out of the blast radius. Hypothesis does not condone cruelty to animals):

Mentions of nukes in my inbox didn’t cause a blood pressure spike, nope.

(😉)

All merges to master that are green on Travis (it does not check the other CI, but the other CI had to be green to merge in the first place so I think that's OK) and have any source changes from the last version will be immediately tagged for release and uploaded to pypi.

Yep, sounds sensible. Patch looks good to me.

It adds some build steps that check that you have updated the version to something that is not tagged, and checks that both the the current date and the current version are present in the changelog (any checking beyond that is up to the reviewer).

👍

It updates the changelog and version to do a release of 3.7.1 today.

Where?

This will force us to be much better about changelog management than we have currently been.

Agreed. We should have a line in #527 about every PR coming with a changelog.

alexwlchan · 2017-04-20T11:59:40Z

scripts/check-changelog.py

+import os
+import sys
+
+sys.path.append(os.path.dirname(__file__)) # noqa


nit: two spaces before inline comment

alexwlchan · 2017-04-20T12:03:13Z

scripts/hypothesistooling.py

+        return i.read()
+
+
+DEVNULL = open(os.devnull)


Alternatively, subprocess.DEVNULL?

Huh. I failed to find that.

DRMacIver · 2017-04-20T12:20:09Z

Where?

In the bit that is currently reverted to test the build jobs making sure I do that.

DRMacIver · 2017-04-20T12:28:48Z

Spaces fixed, subprocess.DEVNULL used, and the version bump and changelog now added back in. I think that's everything?

alexwlchan

Where?

In the bit that is currently reverted to test the build jobs making sure I do that.

Oops, missed that. LGTM now, just one suggestion.

alexwlchan · 2017-04-20T12:29:48Z

docs/changes.rst

+  improves the performance of shrinking failing examples by allowing it to
+  skip some shrinks that it can determine can't possibly work.
+* Hypothesis will no longer seed the global random arbitrarily unless you have
+  asked it to using random\_module.


Worth a link to the docs explaining how to do that?

Zac-HD

A new and terrifying age of continuous deployment is upon us

You're not kidding!

I like the concept, but it's going to lead to an extremely noisy version history where there are a million entries and most change hardly anything. As an example, there are so many tags that readthedocs just omits some (click v: latest in the sidebar and look for 3.x docs). Suggestions:

Wait some number of days (I like 14; it's a fast but steady cadence) from the previous tag before cutting a new release.
There's a timing problem if I put a date in the changelog and the build runs a few minutes later (on the next UTC date). Perhaps compare the message to the commit timestamp, which is easier to set at the right time?

froztbyte · 2017-04-20T12:25:23Z

scripts/check-changelog.py

+
+    when = datetime.utcnow().strftime("%Y-%m-%d")
+
+    if when not in changelog:


potential failure condition: what do we want to happen if the CI run happens "just" after crossing a day boundary?

I've added some wiggle room so it allows any date within +/-1 hour of now.

froztbyte · 2017-04-20T12:27:54Z

scripts/hypothesistooling.py

+
+def current_branch():
+    return subprocess.check_output([
+        "git", "rev-parse", "--abbrev-ref", "HEAD"


Is this for use on a local/dev environment, or is it a more general component?

If more general (incl. Travis), is git(1) guaranteed to be present?

It's entirely for use on Travis, where git is guaranteed to be present.

froztbyte · 2017-04-20T12:35:17Z

scripts/hypothesistooling.py

+    return set(result)
+
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))


this is a nasty failure if we ever move the scripts directory

git rev-parse --show-toplevel may be a better fit, and the following should be enough to get it

ROOT = subprocess.check_output(["git", "rev-parse", "--show-toplevel"]).decode('ascii').strip()

I think it's OK, because if we move scripts we'll fail at reading version.py later, but I'm happy to change it.

froztbyte · 2017-04-20T12:36:36Z

scripts/hypothesistooling.py

+def latest_version():
+    versions = []
+
+    for t in tags():


possibly undocumented assumption about tags structure

I've added a comment explaining it.

froztbyte · 2017-04-20T12:37:03Z

scripts/hypothesistooling.py

+
+def create_tag():
+    assert __version__ not in tags()
+    git("config", "user.name", "Travis CI on behalf of David R. MacIver")


gnnnngnnngh guess we'll have to wait for #535

to clarify: these feel magic strings instead of actual config properties

I don't think this is related to Hypothesis itself's settings, so #535 should be irrelevant.

I'm not intrinsically opposed to moving this out into something more configgy, but the entire build is very specialised and mostly defined in code, so I'm also not sure I see there being much benefit in doing so.

froztbyte · 2017-04-20T12:38:25Z

meta: as per IRC, the PR description is a good attempt towards that discussed in #505, 👍 for that

DRMacIver · 2017-04-20T13:11:11Z

I like the concept, but it's going to lead to an extremely noisy version history where there are a million entries and most change hardly anything.

I don't really see this as a problem. Having more fine grained versioning means that it's easier to localise changes - the CHANGELOG is much the same either way, as it has the same number of entries just somewhat more versions.

As an example, there are so many tags that readthedocs just omits some (click v: latest in the sidebar and look for 3.x docs)

I don't really care about readthedocs's tagging TBH. I'd much rather move the documentation of readthedocs entirely (on my long list of things I want to but can't quite be bothered to do) than adapt to its quirks.

In general past experience suggests that frequent releases are very positively received, especially with bug fixes. I can see some merit in staggering minor releases to not go out so often, but I think having bug fixes and general improvements go out immediately is a fairly big win, and I think I'd like to try doing feature releases with that frequency too.

If it proves annoying in practice then I think the thing to do will be to build a workflow where master is still whatever is released, but we have a separate branch for the upcoming minor release. We'll want to do that for major releases anyway when we finally have another one.

DRMacIver · 2017-04-20T13:14:27Z

@froztbyte I believe I've addressed all your changes.

froztbyte · 2017-04-20T13:20:37Z

@DRMacIver indeed you have, +1'd

DRMacIver · 2017-04-20T13:23:19Z

Aside note: I am moderately confident that this will work but it is very hard to test all of the moving pieces without actually running it. A certain amount of frantically fixing things in post might be required once this is merged.

froztbyte · 2017-04-20T13:25:11Z

I've had that with CI changes in general: each major change is usually followed by a small flurry of successive micro-fixes for weird edge cases etc. Seems fine enough.

DRMacIver · 2017-04-20T14:04:08Z

@Zac-HD can we talk you into withdrawing your objections? :-) I'd quite like to merge this sooner rather than later.

DRMacIver · 2017-04-20T14:17:58Z

I have just looked up the time where he is, and I assume Zac is asleep. I'm going to dismiss the review, but we should think about what to do here more generally in light of #527.

Have acceptance from two other maintainers. Fundamental difference of opinion as to whether this is a good idea, but otherwise all changes have been addressed.

DRMacIver · 2017-04-20T14:24:53Z

Right... Hold on to your hats, and lets see how this goes. :-)

DRMacIver · 2017-04-20T14:32:45Z

How it went was that Travis got very confused with me cancelling a bunch of build PRs and seems to have got stuck. Sigh. Going to repeatedly mash cancel, wait 5 minutes, and then try restarting master.

alexwlchan · 2017-04-20T14:35:44Z

So, err… the build just got cancelled.

@DRMacIver Was that you, or did I inadvertently cancel the wrong build? (I’m doing Travis stuff with the builds at work, I could quite believe I hit the wrong button. If so, apologies.)

DRMacIver · 2017-04-20T14:37:02Z

That was me. It got stuck - no other builds were running but the build was just sitting there hanging waiting to start, so I cancelled it and then restarted.

Cleverly I seem to have figured out a way to bypass the parallelism limits in Travis by doing that. I wish I knew how.

DRMacIver · 2017-04-20T14:40:53Z

So I wonder where we see the post build step running. It's not showing up anywhere in the matrix.

DRMacIver · 2017-04-20T14:46:35Z

Well, that hasn't worked, and I'm getting zero indications as to why.

alexwlchan · 2017-04-20T14:49:51Z

This, from one of the check-django tasks, looks a bit suspicious:

/home/travis/build.sh: line 703: travis_run_after_success: command not found
/home/travis/build.sh: line 704: travis_run_after_failure: command not found
/home/travis/build.sh: line 705: travis_run_after_script: command not found
/home/travis/build.sh: line 706: travis_run_finish: command not found
Done. Your build exited with 0.
uploading archive
/home/travis/build.sh: line 703: travis_run_after_success: command not found
/home/travis/build.sh: line 704: travis_run_after_failure: command not found
/home/travis/build.sh: line 705: travis_run_after_script: command not found
/home/travis/build.sh: line 706: travis_run_finish: command not found

DRMacIver · 2017-04-20T14:50:35Z

That does look very suspicious.

DRMacIver · 2017-04-20T14:56:07Z

Ugh, right, I see what's going on. The condition thing isn't a script, it's one of bash's weird conditional expressions. BRB, reading up on how to bash.

Zac-HD · 2017-04-20T23:34:41Z

Further comments from my review: it was perfectly appropriate to dismiss my review, and reading the thread I'd withdrawn my objections before this point - the difference of opinion is only in degree 😉

We will need to be careful about semver though - any new (non-internal) features or deprecation warnings will require a minor version bump, and we shouldn't bump the minor version without any. Maybe this can be tested by running the old cover suite against the new code (and fail if a patch release raises a deprecation warning), and the new suite against the old public API (and fail if minor releases don't attempt to use new code paths)? I'm not sure how well it would work but given the sheer number of releases we're looking at, things will eventually slip through if we don't have an automatic check.

alexwlchan · 2017-04-21T06:26:10Z

We will need to be careful about semver though - any new (non-internal) features or deprecation warnings will require a minor version bump, and we shouldn't bump the minor version without any.

I don’t agree. Semver guarantees that if there are public API changes, the minor version will change, but not the other way around. I don’t think there are any problems with bumping the minor version for, say, modifications to the engine code.

Quoting from the semver v2.0 guidelines on minor versions:

Minor version Y (x.Y.z | x > 0) MUST be incremented if new, backwards compatible functionality is introduced to the public API. It MUST be incremented if any public API functionality is marked as deprecated. It MAY be incremented if substantial new functionality or improvements are introduced within the private code. It MAY include patch level changes. Patch version MUST be reset to 0 when minor version is incremented.

DRMacIver · 2017-04-21T06:32:17Z

Yeah, I've occasionally updated the minor version for changes that I thought were too large for a patch version even if they were technically not changing the public API. I think it's a reasonable thing to do.

I do think we shouldn't do it too often, as the version numbers will get a bit silly even without, but it's an option to bear in mind.

For now I'm happy with just checking that the versioning is sensible in review. The first time we experience a problem as a result of that (either a bad version number or just finding it really annoying) we can look into automation.

Zac-HD · 2017-04-21T07:36:05Z

No problem, YAGNI it is.

DRMacIver added 6 commits April 20, 2017 12:07

Initial travis config changes

a5d5d52

Update version and changelog

98a3bae

Check the changelog and tags as build jobs

a2e16ba

Actually do the deploy.

9666574

Temporary revert so we can see the build jobs fail

47465c1

Revert "Update version and changelog" This reverts commit 98a3bae.

Actually exit on failure

7b3932a

alexwlchan requested changes Apr 20, 2017

View reviewed changes

DRMacIver added 3 commits April 20, 2017 13:21

Update inline space

716e263

Use subprocess.DEVNULL

f1df5e3

Actually update changelog and version.

882a1f0

alexwlchan approved these changes Apr 20, 2017

View reviewed changes

Zac-HD previously requested changes Apr 20, 2017

View reviewed changes

froztbyte requested changes Apr 20, 2017

View reviewed changes

DRMacIver added 4 commits April 20, 2017 13:56

add link to random_module

c2ff343

Allow some wiggle room on dates

80bb25b

Correct wiggle room logic

f5547fa

Use git to find ROOT

1a99879

Clarify tag version parsing

7cf6d31

froztbyte approved these changes Apr 20, 2017

View reviewed changes

DRMacIver mentioned this pull request Apr 20, 2017

We should have an explicit review checklist #505

Closed

DRMacIver merged commit cf729a2 into master Apr 20, 2017

alexwlchan deleted the DRMacIver/release-automation branch April 20, 2017 14:25

DRMacIver mentioned this pull request Apr 20, 2017

Write our own deploy orchestration code #540

Merged

froztbyte mentioned this pull request Apr 20, 2017

Add benchmarking build step #528

Merged

Zac-HD mentioned this pull request Apr 21, 2017

We should document plans for future releases #532

Closed

Zac-HD mentioned this pull request Jul 6, 2018

Better tooling for releases and development python-trio/trio#220

Open


		when = datetime.utcnow().strftime("%Y-%m-%d")

		if when not in changelog:

		return set(result)


		ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

A new and terrifying age of continuous deployment is upon us #537

A new and terrifying age of continuous deployment is upon us #537

Conversation

DRMacIver commented Apr 20, 2017

DRMacIver commented Apr 20, 2017

alexwlchan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DRMacIver commented Apr 20, 2017

DRMacIver commented Apr 20, 2017

alexwlchan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zac-HD left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

froztbyte commented Apr 20, 2017

DRMacIver commented Apr 20, 2017

DRMacIver commented Apr 20, 2017

froztbyte commented Apr 20, 2017

DRMacIver commented Apr 20, 2017

froztbyte commented Apr 20, 2017

DRMacIver commented Apr 20, 2017

DRMacIver commented Apr 20, 2017

DRMacIver commented Apr 20, 2017

DRMacIver commented Apr 20, 2017

alexwlchan commented Apr 20, 2017

DRMacIver commented Apr 20, 2017

DRMacIver commented Apr 20, 2017

DRMacIver commented Apr 20, 2017

alexwlchan commented Apr 20, 2017

DRMacIver commented Apr 20, 2017

DRMacIver commented Apr 20, 2017

Zac-HD commented Apr 20, 2017

alexwlchan commented Apr 21, 2017

DRMacIver commented Apr 21, 2017

Zac-HD commented Apr 21, 2017