Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The "we're dropping Python 2" checklist #2218

Open
Zac-HD opened this issue Nov 21, 2019 · 17 comments
Open

The "we're dropping Python 2" checklist #2218

Zac-HD opened this issue Nov 21, 2019 · 17 comments
Assignees
Labels

Comments

@Zac-HD
Copy link
Member

@Zac-HD Zac-HD commented Nov 21, 2019

This post is a living document, and regularly edited to reflect our latest plans.

Per our compatibility policy, Hypothesis will drop Python 2 support soon after upstream support ends on 2020-01-01 (countdown here). This issue exists to describe, discuss, and track progress on all the improvements that are possible once we go Python-3-only.

First,

  • #2282: release Hypothesis 5.0 without Python 2 support on Jan 1st.
    That PR has its own checklist, with minimal changes to get to the new public API.

Once Hypothesis 5.0 is released, I'll open a second PR with the large tool-assisted changes. This will be a very large diff, but quite mechanical and not too hard to review. IMO it's best to get this in as a single hard-changeover, to minimise the merge conflicts and churn it could cause for other PRs.

  • #2299: fully-automated removal of py2 compat code, with some minor manual fixes.
    That PR also has its own checklist, and has no changes to the public API.

That's a +1,020 −2,461 line PR, which is quite enough to review even as obvious mechanical fixes. Once that's in, we have the manual fixes - the bulk of which are "shrink compat.py":

  • #2301: replacing compatibility shims and imports with native Python 3 code.

Finally, I'll:

  • #2309: convert type comments to annotations, fix conditional imports of the typing module, and fix bad annotations

Subsequent changes - of which there are many! - can be done in smaller and mostly independent chunks. This reduces the size of merge conflicts, facilitates code review, and ensures that we can mix feature development, unrelated bugfixes, and refactoring work as they appeal to each contributor 😄

  • Finish refactoring stateful testing (initial PR just makes GenericStateMachine a private API)
  • Consider replacing uses of inspect.getfullargspec with inspect.signature
  • Consider specifying how to write short bytestrings in our project style guide
    e.g. preferring b"\x00" over any of b"\0", bytes(1), or bytes([0]).
  • Move remaining functions from compat.py to live with related code instead
  • Misc. cleanups, i.e. #2315

I'm sure there are many things I'm missing - if you think of one that doesn't already have its own issue or pull request, please comment below and I'll update the list! My goal here is to assemble a complete checklist of refactorings and upgrades that we probably want to do; so interpret these as options rather than requirements.


Special bonus checklist: things to revisit when we drop Python 3.5 later this year!

  • Use com2ann to convert type-hints on variables to annotations
  • Check git grep -nE "# pragma: no (branch|cover)|[pP]y(thon)? ?3|sys\.version" to see all of the locations we've tagged for version compatibility reasons (and a few false-positives)
  • Generally clean up a lot of the from_type() backend
  • Bump target versions for black and pyupgrade
  • Consider converting from percent-formatting to f-strings
@Zac-HD Zac-HD added the meta label Nov 21, 2019
@Zac-HD Zac-HD self-assigned this Nov 21, 2019
@aarchiba

This comment has been minimized.

Copy link
Contributor

@aarchiba aarchiba commented Nov 23, 2019

Is python 3.5 still desired? If not updating to f-strings can tidy the code substantially.

@Zac-HD

This comment has been minimized.

Copy link
Member Author

@Zac-HD Zac-HD commented Nov 24, 2019

Our compatibility policy is "until end of upstream support", so we'll be keeping 3.5 until next September. I'm definitely looking forward to a 3.6+ codebase though!

(strictly speaking our policy only says "Hypothesis is ...", not "will remain" - so we're in the clear to drop it early if we choose, as proposed below)

@DRMacIver

This comment has been minimized.

Copy link
Member

@DRMacIver DRMacIver commented Nov 24, 2019

Our compatibility policy is "until end of upstream support"

though personally I'm keen to be more aggressive about this, and dropping 3.5 early is a pretty safe choice IMO.

@Zac-HD

This comment has been minimized.

Copy link
Member Author

@Zac-HD Zac-HD commented Nov 24, 2019

I've been thinking about this too - the only reason I'm keen to keep compatibility for 3.5 given the relatively low usage is to avoid blocking users from upgrading to the latest Hypothesis. (honestly this shouldn't be an issue anyway, but it is anyway for some) Simplicity of communication is nice, but "3.6+" is even easier to say! I think a reasonable pathway would be to:

  1. drop Python 2 at the new year and rip out the compatibility stuff shortly afterwards,
  2. release Hypothesis 5.0 to clear out the deprecated stuff, and
  3. release a minor version after that which drops Python 3.5 support - aiming for early Feb.

End result, 3.5 users have ~8 months on "modern" Hypothesis but without any extra features / bugfixes / whatever else we do over that timeframe; and we have a nicer development experience. If any users care a lot, I'm happy to do paid backports in an isolated branch!

Sound good?

@DRMacIver

This comment has been minimized.

Copy link
Member

@DRMacIver DRMacIver commented Dec 2, 2019

Modify our build system to auto-publish the release at exactly midnight (@DRMacIver)

Ha! You're on the ball. I was just coming over to this issue to add that.

@DRMacIver

This comment has been minimized.

Copy link
Member

@DRMacIver DRMacIver commented Dec 2, 2019

This will be Hypothesis 4.XX.0, i.e. a minor version.

Wait, just rereading this... I missed this the first time. Why will this be a minor version? I don't think we should drop Python 2 in a minor version.

If the reason is that we don't want to do both turn deprecations into errors and drop Python 2 support in the same release (which I'd be fine with us doing but I can see the argument against) I'd rather us do two major versions in short succession (it wouldn't be the first time!) than drop Python 2 support in a minor version.

@Zac-HD

This comment has been minimized.

Copy link
Member Author

@Zac-HD Zac-HD commented Dec 2, 2019

I'm fine with doing 5.0 at the stroke of midnight - it shouldn't actually be hard to manage, so long as we push the refactoring off to follow-up patches (which was already the plan). Top post updated.

I've got that week off and I'm happy to write delete a lot of code, but you-as-codeowner will be doing a lot of reviews...

@auvipy

This comment has been minimized.

Copy link

@auvipy auvipy commented Dec 16, 2019

If you don't mind, I want to help on this beside django and drf.

@Zac-HD

This comment has been minimized.

Copy link
Member Author

@Zac-HD Zac-HD commented Dec 16, 2019

I don't mind at all - I'll write the initial set of three or four big changes (I've already scripted most of it), but after that I'd be happy for you to write some of the more contained ones in the first week of January. Review comments are also helpful!

@rsokl

This comment has been minimized.

Copy link
Contributor

@rsokl rsokl commented Dec 17, 2019

I will happily help shepherd the extra.numpy material along for this triumphant journey

@Zac-HD

This comment has been minimized.

Copy link
Member Author

@Zac-HD Zac-HD commented Dec 18, 2019

I've re-organised the post-5.0 checklist into two parts - one large but fairly mechanical PR that I'll aim to get out in the first day or two of January, and a second part with smaller changes which more of us can work on concurrently 😄

@auvipy

This comment has been minimized.

Copy link

@auvipy auvipy commented Dec 18, 2019

can you try pyupgrade?

@Zac-HD

This comment has been minimized.

Copy link
Member Author

@Zac-HD Zac-HD commented Dec 18, 2019

pyupgrade is a standard part of our formatting task, so we're already using it!

If you mean "to remove the if PY2: blocks, I've experimented on top of my drop-py2 branch and it works pretty well - as with most of the ambitious formatting tools it does 80-90% of the work and then I'll do some manual cleanup to make it look like the code has always been that way 😄

@Zac-HD

This comment has been minimized.

Copy link
Member Author

@Zac-HD Zac-HD commented Dec 29, 2019

As of the current PR, plus drafts linked in the top comment:

Showing 361 changed files with 1,867 additions and 4,705 deletions.

My condolences thanks to everyone involved in review! 😂

I'm already enjoying life without awkward compat shims - it makes a surprisingly big difference. Or perhaps not so surprising; pygount suggests that I've deleted a net 6% of the library even before considering tests... and the changed portion is generally much nicer to work with.

@tomato42

This comment has been minimized.

Copy link

@tomato42 tomato42 commented Jan 3, 2020

Our compatibility policy is "until end of upstream support", so we'll be keeping 3.5 until next September. I'm definitely looking forward to a 3.6+ codebase though!

Could I ask for a major version bump then? It makes version pinning clearer...

/me is still running hypothesis on python 2.6 and 3.3

@Zac-HD

This comment has been minimized.

Copy link
Member Author

@Zac-HD Zac-HD commented Jan 9, 2020

Surely you already have to pin to a much older version though?

@tomato42

This comment has been minimized.

Copy link

@tomato42 tomato42 commented Jan 9, 2020

Surely you already have to pin to a much older version though?

yes, that's why I'm asking for making it easier/more obvious to know which version to pin to (I don't pin all dependencies, only the ones I have to)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.