Drop support for Python 2

[PCManticore]

Recently Django 2.0 was released just with Python 3 support. I think we can drop Python 2 support as well, especially since it is planned to do so for Python 3.7, since afterwards some parts of astroid's inference engine will stop working due to the way we propagate context outside of generators.

I think we could try the following plan:

• release 1.8 this year. It's going to be the last release with Python 2 support, which means we're going to support bug fixes to it for a little while (let's say until 2020, since I imagine 2.7 it's not going to disappear soon)
• master becomes 2.0 and we start removing support for Python 2. We could consider if we need to drop support for analyzing Python 2 code altogether. First and foremost using new features from 3+ would be ideal. We could still keep support for analyzing Python 2 code for a little while.
• astroid's master becomes 2.0 as well and we do the same thing over there.

What do you think about this @rogalski @hippo91 and the rest of the folks?

[PCManticore]

cc @AWhetter :)

[rogalski]

Looks good to me. Especially part about not removing py2.x related code in a hurry. Keeing it will help with backporting any fixes to 1.8 branch. Do we agree that having backport for 1.8.x ready is mandatory for merging any new bug fix pull request? Dnia 11.12.2017 o godz. 12:20 Claudiu Popa <notifications@github.com> napisał(a):

…

cc @AWhetter :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[PCManticore]

@rogalski Yeah, backporting bug fixes sounds good to me. We'll most likely won't backport features (e.g new checks or flags).

[AWhetter]

This sounds good to me. I definitely think it's important to keep an LTS, Python 2 compatible version going alongside the Python 3 version!
I also like the sound of making backports for bug fixes mandatory, although I don't know how practical this will be for large bug fixes but we can make exceptions if that ever happens.

Looks like I need to knuckle down and get per directory configuration finished! I'm hoping I can get it done for v2.0.

[hippo91]

I agree with all of you and i also think it is very important to keep the ability to analyze python2 code even if it is with python3 interpreter. There are still projects using python2.7 and some of them will keep using it for a while...

[hugovk]

Here's the pip installs for pylint from PyPI for the last month (via pypinfo --percent --pip --markdown pylint pyversion)

python_version	percent	download_count
2.7	59.4%	384,749
3.6	20.7%	133,865
3.5	12.0%	77,575
3.4	6.6%	42,701
2.6	0.8%	5,312
3.7	0.3%	1,672
3.3	0.2%	1,527
3.2	0.0%	20
None	0.0%	4

[brycepg]

We use a lot of hardcoded version checks in both astroid and pylint. These checks will need to become dynamic if we are going to allow python 3 pylint to parse python2 code properly (that is I don't think it will happen without a gigantic amount of effort).

[PCManticore]

That's true, @brycepg, there is a caveat that comes with Python 2 support in pylint 2.0 that wasn't mentioned: pylint is tied to its underlying AST parser. That is, we can still analyze Python 2 code as long as it doesn't use Python 2 specific syntax, so projects that rely on six to offer a Python 2/3 compatible code base could still be analysed by Pylint. Because of this, I'm not proposing to remove just yet Python 2 specific checks (but we can remove Python 2 specific syntax error checks, for instance for print-statement, there is no way to get a Print statement in a Python 3 AST). We should also remove if guards for Python 2 specific branches, e.g if the code assumes that we run under Python 2, then it can be safely removed, since it will never happen under Pylint 2.0. So to summarize, we should:

• remove if guards for Python 2
• Python 2 specific syntax checks or syntax compatibilities in the AST (so any checks that checks for print-statement etc as well as visit_print & friends in astroid/pylint)

[gpshead]

I do not consider this an enhancement so long as Python 2 is a supported CPython runtime. The world is full of Python 2 code. This means there can be no new features added to pylint to help out the bulk of the worlds code bases (including lint features aimed at helping port to Python 3!) without someone creating a fork of the pylint project off of the 1.8 branch. :(

At which point you've got two pylints, with diverging feature sets, checker behaviors, checker names, all sharing a common # pylint: disable= namespace and common config files yet not coordinated.

Please do not do this yet. Python 2 is quite relevant to the world at large. Lets keep pylint relevant.

I fully support Django and NumPy moving forwards, but source code analysis tooling is a different matter as people use tools to improve their development practices no matter what language version their code is in.

If you do this, you need regression tests to guarantee that "2 and 3" single source compatible code analyzed by pylint 1.8 on Python 2 does not have different warning behavior when analyzed by pylint 2.x+ running under Python 3? One can imagine a hellish situation where it becomes impossible to write lint clean code that passes both as implementations diverge.

[PCManticore]

@gpshead Thanks for the input. I understand this is a bit sensible, so I wanted to have some time to think thoroughly with regard to your comment.

One thing to start the discussion is that this decision wasn't necessarily motivated by a personal choice. Since PEP 479 changed the way StopIteration are handled inside generators, we found ourselves in a bit of a conundrum with regard to astroid, which uses StopIteration across the inference parts for propagating additional context out of the inference loop. The solutions we found meant either using exec() to generate different code for different versions, or changing the inference to work in some other way so it wouldn't rely on StopIteration, or completely dropping support for Python <3.3. Given this choice, we thought by dropping Python 2 support that we could leverage other nice things that could improve the code quality of the project: type annotations,
we could remove tons of compatiblity shims, yield from. On the personal side, I think all of the current Pylint & Astroid committers would also prefer to write the code in Python 3, since none of us is paid to work on this project.

That being said, I understand from where you're coming from, but at the same time I feel there is some FUD in your comment that somewhat feels aggressive. It's the second time this week that I've heard a similar comment from a pylint user, where either we do a disservice to the community at large or we're not going to be relevant just because we don't want to support an old version of Python. There is no financial incentive that motivates the Pylint committers to do their work,
so where this don't do this or you're getting irrelevant is coming from? And it's somewhat disingenuos to let other tools be Python 3 only, numpy and django as you mentioned, but when we talk about source analysis tools we suddenly need to support any version of Python.

But now after seeing your point, I admit it would be drastic in a way to stop supporting the old Python 2 codebases by this 1.8 - 2.0 dichotomy, where 1.8 is frozen with regard to features. I think it is possible to find a middle ground. The initial proposal was for 1.8 to be bugfixes only, but I think it could be possible to spin a another 1.9 branch/release, that will keep working on Python 2. This one should receive bug fixes as mentioned, but extra I would say that it could also receive
Python 3 porting checkers, since this is one of the main drives towards Python 3 adoption that pylint could offer. Other features will still be Pylint 2.0 only ( so flow understanding, type annotations support, config files per directories etc.)
With this in mind, would a 1.9 version, that has support until 2020 and that receives bug fixes + Py3k porting checks, be an alternative middle ground that you'd be happy with? I understand that having two separate linter versions means extra hassle as you mentioned, that they will come with diverging feature sets. If we can, we'll try to also backport some other new checks, as long as they don't depend on refactorings/features that are unavailable in a way to 1.9. So in the end, we aim for bug fixes + py3k portings checks + any new check that is somewhat easily backportable to 1.9,
but no new major features as those aforementioned.

Let me know what you think. I hope that we can find a middle ground that could work for all of us.

[gpshead]

Apologies. Sorry for coming off so aggressive there. That was uncalled for frustration on my part after I did a git pull and found master no longer supporting Python 2. :(

Thanks for explaining your reasoning on the implementation side!

I don't actually know if there are going to be features worthwhile adding that deserve more than the 1.8 bugfix only branch so I wouldn't bother planning a 1.9 unless a set of desired new features for use on Python 2 is actually found.

As for why I complain about this for pylint but not for the numpys and djangos of the world... major libraries and frameworks used within applications are developer leaders, their choice to drop Python 2 helps pull users who have been kicking and screaming forward. Good! I do not expect people view development time tool such as pylint the same way (it has a behind the scenes role - people don't think about it - it's effectively a "compile" time step). That said, people not thinking about it suggests that in the short term they won't even notice. So long as they get 1.8 vs 2.x depending on which python version they are using (which should be automatic thanks to pip and metadata), they'll be happy enough.

[mcepl]

I would like also my support for keeping Python 2 at least until EOS of 2.7 (unfortunately, py2k is still widely used for in-house code), but pain could me greatly decreased, if pylint could recognize whether it analyses py2k or py3k code. Even just checking for python3 string in shebang could help. Then we could avoid nonsenses like "invalid-encoded-data" for py3k scripts, and pain would be tolerable, I hope.

[PCManticore]

Closing this as the support was already removed. To summarise and to provide some extra context:

• Pylint 2.0 is going to run only on Python 3. That is, it's running environment will need to be Python 3, but it will still analyze Python 2 files. Some checks might not work, for instance those that assumed a particular Python 2 syntax, but for the majority, we'll still be able to emit them. Also we'll add a flag to pylint (Replace sys.version_info checks for Python 2 compatibility with another way of detecting the file environment #2070), through which one can say that wants to analyze Python 2 files (something along the lines of pylint -2 ...)
• Pylint 1.8/1.9 will support Python 2 until 2020. We'll probably release 1.9 instead soonish, the branch is up and has a couple of new Python 2/3 porting checks, we'll probably release it these days at PyCon.
• Most likely the --py3k flag for Python 2/3 compat checks will also run on Python 3. Currently it is a no-op on Python 3, but for the sake of helping the community to port, it probably should be enabled as well.
• I added some notes in the documentation as well. There shouldn't be too much disruption, as long as pip install gets the right version depending on the running Python, and with the -2 flag we'll still be able to analyze Python 2 files.

Let me know if you have any other concerns regarding this change!