Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop Python 2 and 3.4 support #229

Merged
merged 5 commits into from Jul 2, 2021
Merged

Conversation

peternowee
Copy link
Member

@peternowee peternowee commented May 11, 2020

UPDATE 2021-06-25: Updated description in my comment of 2021-06-25 below.

Original description (outdated)

1st commit: Drop Python 2 and <=3.4 support, add 3.8 support

Why support for Python 3.5 is not dropped yet:

  • We do not urgently need Python 3.6+ specific functionality.
  • Python 3.5 is still receiving security fixes until 2020-09-13.[1] [2]
  • Recent PyPI download statistics for pydot show that 20 to 40% of the users are still on Python 3.5.

For further details, see pydot/pydot#171.

2nd commit: Drop Travis CI tests Python 2, 3.4, add Python 3.8

Also removes the workaround for the Python 3.7 test environment added by commit eb7aef1. It is no longer needed by travis-ci.com.

3rd commit: Change AppVeyor tests to Python 3.5 and 3.8

Drop the test for Python 2 now we no longer support that.

Also:

  • Explicitely choose the image to be used (Visual Studio 2019), as the default image (Visual Studio 2015) does not have Python 3.8 in Miniconda.
  • Add fast_finish option to prevent running subsequent jobs if the first job fails.
  • Only test the oldest and the newest supported Python versions (currently Python 3.5 and 3.8). Intermediate versions are tested in Travis CI already.
  • Correct a typo in the init that prevented correct printing of environment variables.

Additional remarks:

  • This PR targets a future major release of pydot.
  • Earlier discussion was in issue Exception on parsing dot file #171.
  • No code changes yet. This is just to declare that Python 2 compatibility will break soon, because of other changes that I am working on. I do not have time for a cleanup of Python 2-supporting code right now and I also do not think that is necessary for this to merge.
  • All tests pass on my laptop (Debian buster 10.4, Linux 4.19.118-2 amd64, Python 3.7.3, Graphviz 2.40.1 (20161225.0304), pyparsing 2.4.7).
  • Travis CI runs all the test environments 3.5 through 3.8 and all of them succeed when run together with Fix multi.dot Graphviz regression test #211.
  • AppVeyor starts the tests for 3.5, and also for 3.8 when fast_finish is off, but both still fail because of issue Inconsistent GraphViz output on AppVeyor/Windows/Anaconda #213.
  • Let me know if you start merging PRs and want me to rebase this branch or if you need help determining in which order to merge all the open PRs.
  • Reviews welcome!

CC: @ankostis

@ankostis
Copy link

ankostis commented May 18, 2020

Just noting that 2019 Python's developers survey just came out and Py3.5 usage is really low, 4% of all PY3. Actually 90% comprises just 3.6 + 3.7 + 3.8.
So i would say that relying on deterministic dictionary order & f-strings may well worth losing 5% of user-base for all new releases after 2020.

https://www.jetbrains.com/lp/python-developers-survey-2019/

@peternowee
Copy link
Member Author

Just noting that 2019 Python's developers survey just came out and Py3.5 usage is really low, 4% of all PY3. Actually 90% comprises just 3.6 + 3.7 + 3.8.

Wow, that is a big difference with the pydot download stats. I just had another look at them and they still show usage of Python 3.5 around 25%. And that is of Python 2+3 overall, so it is even higher when viewed as a proportion of PY3 only: (25%/80%=) 31% of PY3, vs. the 4% reported by the JetBrains survey.

Some possible explanations I can think of right now:

  • The JetBrains survey may be selecting more for frequent or experienced Python developers, while the download statistics also include occasional or unaware Python users, not necessarily developers.
  • The JetBrains survey was conducted in October and November 2019. The download statistics of end of November 2019 also show a lower Python 3.5 usage than now: around 7%, translating into (7%/40%=) 17.5% of PY3. However, in the first quarter of 2020, the download statistics show a large migration from Python 2 to Python 3 that disproportionately benefited Python 3.5. As I mentioned earlier, probably those are users of distributions shipping Python 2.7 and 3.5 such as Debian 9 and Ubuntu 16.04. The effect of this will be less for the JetBrains sample, because most of those respondents (90%) were on Python 3 already in Oct/Nov 2019.
  • The download statistics may include more duplicate responses, e.g. CI VMs that are repeatedly destroyed and re-provisioned, and maybe that happens more often with older setups than with newer ones. Really just speculating here, though.

So i would say that relying on deterministic dictionary order & f-strings may well worth losing 5% of user-base for all new releases after 2020.

I am not so sure if I would easily give up 5% of our user base just for some new features for us pydot developers. Anyway, luckily, you add "for all new releases after 2020", so that means at least the official Python support for PY3.5 has also been already dropped (2020-09-13), and as I said earlier, I am willing to reconsider then. I assume that by the end of 2020, the pydot download statistics will also have shown a big drop in PY3.5. But currently, it is only May 2020 and I am still hoping this PR gets merged earlier than September or the end of 2020.

Also again, I am willing to reconsider earlier if there is a need for it, but then again I would like to know if you use PY3.6-specific functionality in your open PR now, or need it for something else you are working on?

@ankostis
Copy link

No, no , it's nothing to do with my functionality needs.
It's just for easing the pydot developer - i know that it's fun to program with compile-time checked f-strings

peternowee referenced this pull request in Ashafix/pydot May 21, 2020
@peternowee peternowee added this to the 1.5.0 or 2.0.0 milestone Aug 29, 2020
peternowee added a commit to peternowee/pydot that referenced this pull request Jun 24, 2021
This starts a series of commits to drop support for Python 2 and 3.4 as
discussed in [pydot#171][171] and [pydot#229][229].

| Python version | Python EOL | PyPI 2020-05-02 | PyPI 2021-06-21 |
|----------------|------------|-----------------|-----------------|
| Python 2.7     | 2020-01-01 |             26% |             12% |
| Python 3.4     | 2019-03-18 |              0% |              0% |
| Python 3.5     | 2020-09-30 |             27% |              3% |
| Python 3.6     | 2021-12    |             21% |             14% |
| Python 3.7     | 2023-06    |             23% |             51% |
| Python 3.8     | 2024-10    |              3% |             13% |
| Python 3.9     | 2025-10    |              0% |              6% |
| Python 3.10    | 2026-10    |               - |              0% |

EOL : End of life, from End of security support on [Wikipedia][1].
PyPI: Python Package Index statistics for pydot from [PyPIstats.org][2].

Without support for Python 2, wheel distributions of pydot can [no
longer be marked as "universal"][3], so removing that from `setup.cfg`.

**USER FEEDBACK REQUESTED**
We are considering if pydot 2.0 should drop support for Python 3.5
and 3.6 as well. If this would affect you, please leave a comment in
[pydot#268][268].

[1]: https://en.wikipedia.org/w/index.php?title=History_of_Python&oldid=1022680403#Table_of_versions
[2]: https://pypistats.org/packages/pydot
[3]: https://github.com/pypa/packaging.python.org/blob/44313e5db4d729eede0bcb91c08d6ec93e89c5c8/source/guides/dropping-older-python-versions.rst#dealing-with-the-universal-wheels
[171]: pydot#171 (comment)
[229]: pydot#229
[268]: pydot#268
This starts a series of commits to drop support for Python 2 and 3.4 as
discussed in [pydot#171][171] and [pydot#229][229].

| Python version | Python EOL | PyPI 2020-05-02 | PyPI 2021-06-21 |
|----------------|------------|-----------------|-----------------|
| Python 2.7     | 2020-01-01 |             26% |             12% |
| Python 3.4     | 2019-03-18 |              0% |              0% |
| Python 3.5     | 2020-09-30 |             27% |              3% |
| Python 3.6     | 2021-12    |             21% |             14% |
| Python 3.7     | 2023-06    |             23% |             51% |
| Python 3.8     | 2024-10    |              3% |             13% |
| Python 3.9     | 2025-10    |              0% |              6% |
| Python 3.10    | 2026-10    |               - |              0% |

EOL : End of life, from End of security support on [Wikipedia][1].
PyPI: Python Package Index statistics for pydot from [PyPIstats.org][2].

Without support for Python 2, wheel distributions of pydot can [no
longer be marked as "universal"][3], so removing that from `setup.cfg`.

**USER FEEDBACK REQUESTED**
We are considering if pydot 2.0 should drop support for Python 3.5
and 3.6 as well. If this would affect you, please leave a comment in
[pydot#268][268].

[1]: https://en.wikipedia.org/w/index.php?title=History_of_Python&oldid=1022680403#Table_of_versions
[2]: https://pypistats.org/packages/pydot
[3]: https://github.com/pypa/packaging.python.org/blob/44313e5db4d729eede0bcb91c08d6ec93e89c5c8/source/guides/dropping-older-python-versions.rst#dealing-with-the-universal-wheels
[171]: pydot#171 (comment)
[229]: pydot#229
[268]: pydot#268
Specifically for AppVeyor:
- Explicitely choose the image to be used (Visual Studio 2019), as
  the default image (Visual Studio 2015) does not have Python 3.8 *in
  Miniconda*. There is no image with Python 3.9 in Miniconda available
  yet. [1]
- Add `fast_finish` option [2] to prevent running subsequent jobs if
  the first job fails.
- Normally only test the oldest and the newest Python versions pydot
  supports, which are currently Python 3.5 (EOL) and 3.9. However, now
  using 3.6 instead of 3.5, because AppVeyor fails to install pyparsing
  in 3.5, and using 3.8 instead of 3.9 because 3.9 is not available on
  AppVeyor yet (see above).
- Correct a typo in the `init` that prevented correct printing of
  environment variables.

[1]: https://www.appveyor.com/docs/windows-images-software/#miniconda
[2]: https://www.appveyor.com/docs/build-configuration/#failing-strategy
`dot_parser.py` currently has 6 functions that still use `str` as a
parameter name. This will cause conflicts if inside the function body
a reference to Python's built-in [`str` type][1] needs to be made.

Specifically, as part of the clean up of old Python 2 code, the next
commit will introduce direct usage of the built-in `str` in two of
these functions. The concurrent use of `str` as a function parameter
name would then lead to name collisions and errors such as:

      File "dot_parser.py", line 349, in push_edge_stmt
        elif isinstance(toks[2][0], str):
    TypeError: isinstance() arg 2 must be a type or tuple of types

Commit 842173c already renamed the parameter for one particular
function from `str` to `s` in 2008. This commit will now do the same
for the remaining functions.

Further notes:
- None of the affected functions actually use the parameter (`str`, now
  `s`) in their function bodies.
- The functions themselves are passed to `pyparsing.setParseAction()`
  as callback functions. The docstring of that method also uses the
  parameter name `s` in the example callback signature. However, the
  actual parameter name is irrelevant for pyparsing (2.4.7), as it uses
  positional arguments when actually calling the callback.
- The affected functions contain 3 format strings that use `{s}` as a
  replacement field name. Other than perhaps being slightly disturbing
  to the human reader, these do not actually collide with the function
  parameters named `s`, so leaving them as-is for now. A future change
  to f-strings would be a better way to resolve the confusion.

[1]: https://docs.python.org/3/library/stdtypes.html#str
Just a very basic cleanup, made with some help from 2to3.

Includes the removal of intermediairy variable `str_type`, which held
the string type for either Python 2 or 3, replacing it with the string
type for Python 3 `str`. The previous commit prepared for this by
removing the conflicting uses of the name `str` as function parameter
names.
@peternowee
Copy link
Member Author

Rebased to master.

1st commit: Drop Python 2 and 3.4: Update Metadata, ChangeLog

This starts a series of commits to drop support for Python 2 and 3.4 as discussed in pydot/pydot#171 and #229.

Python version Python EOL PyPI 2020-05-02 PyPI 2021-06-21
Python 2.7 2020-01-01 26% 12%
Python 3.4 2019-03-18 0% 0%
Python 3.5 2020-09-30 27% 3%
Python 3.6 2021-12 21% 14%
Python 3.7 2023-06 23% 51%
Python 3.8 2024-10 3% 13%
Python 3.9 2025-10 0% 6%
Python 3.10 2026-10 - 0%

EOL : End of life, from End of security support on Wikipedia.
PyPI: Python Package Index statistics for pydot from PyPIstats.org.

Without support for Python 2, wheel distributions of pydot can no longer be marked as "universal", so removing that from setup.cfg.

USER FEEDBACK REQUESTED
We are considering if pydot 2.0 should drop support for Python 3.5 and 3.6 as well. If this would affect you, please leave a comment in #268.

2nd commit: Drop Python 2 and 3.4: Update CI configuration

Specifically for AppVeyor:

  • Explicitely choose the image to be used (Visual Studio 2019), as the default image (Visual Studio 2015) does not have Python 3.8 in Miniconda. There is no image with Python 3.9 in Miniconda available yet.
  • Add fast_finish option to prevent running subsequent jobs if the first job fails.
  • Normally only test the oldest and the newest Python versions pydot supports, which are currently Python 3.5 (EOL) and 3.9. However, now using 3.6 instead of 3.5, because AppVeyor fails to install pyparsing in 3.5, and using 3.8 instead of 3.9 because 3.9 is not available on AppVeyor yet (see above).
  • Correct a typo in the init that prevented correct printing of environment variables.

3rd commit: Stop using str as a function parameter name

dot_parser.py currently has 6 functions that still use str as a parameter name. This will cause conflicts if inside the function body a reference to Python's built-in str type needs to be made.

Specifically, as part of the clean up of old Python 2 code, the next commit will introduce direct usage of the built-in str in two of these functions. The concurrent use of str as a function parameter name would then lead to name collisions and errors such as:

  File "dot_parser.py", line 349, in push_edge_stmt
    elif isinstance(toks[2][0], str):
TypeError: isinstance() arg 2 must be a type or tuple of types

Commit 842173c already renamed the parameter for one particular function from str to s in 2008. This commit will now do the same for the remaining functions.

Further notes:

  • None of the affected functions actually use the parameter (str, now s) in their function bodies.
  • The functions themselves are passed to pyparsing.setParseAction() as callback functions. The docstring of that method also uses the parameter name s in the example callback signature. However, the actual parameter name is irrelevant for pyparsing (2.4.7), as it uses positional arguments when actually calling the callback.
  • The affected functions contain 3 format strings that use {s} as a replacement field name. Other than perhaps being slightly disturbing to the human reader, these do not actually collide with the function parameters named s, so leaving them as-is for now. A future change to f-strings would be a better way to resolve the confusion.

4th commit: Drop Python 2 and 3.4: Clean up PY2-specific code

Just a very basic cleanup, made with some help from 2to3.

Includes the removal of intermediairy variable str_type, which held the string type for either Python 2 or 3, replacing it with the string type for Python 3 str. The previous commit prepared for this by removing the conflicting uses of the name str as function parameter names.

5th commit: Drop Python 2 and 3.4: Bump black target-version

(No further details.)

Additional remarks:

  • Rebased to master. Changes compared to before the rebase:
    • Support for Python 3.8 and 3.9 was declared in master branch, commit 23027b3b already.
    • Not testing all intermediate Python versions in CI anymore in master branch since commit 682edb7b.
    • Drop universal=1 from setup.cfg. See 1st commit.
    • Added a basic cleanup of Python 2-supporting code, which also involved changing some function signatures in dot_parser.py. See 3rd and 4th commit.
    • Added configuration update for the Black code formatter, in master since commit 65988af0. See 5th commit.
  • All tests pass on my laptop (Debian buster 10.10, Linux 4.19.194-1 amd64, Python 3.7.3, Graphviz 2.40.1 (20161225.0304), pyparsing 2.4.7).
  • AppVeyor still fails because of issue Inconsistent GraphViz output on AppVeyor/Windows/Anaconda #213. Help welcome.
  • I am still hesitant about dropping support for Python 3.5. I opened Need for pydot 2.0 to support Python 3.5, 3.6? #268 to ask users on Python 3.5 or 3.6 to step forward. I think this PR does not have to wait for that and any dropping of Python 3.5 can go in a separate PR later.
  • Reviews/comments welcome. If nothing comes up, I plan to merge this 1 week from now.

@peternowee peternowee changed the title Drop Python 2 and <=3.4 support, add 3.8 support Drop Python 2 and 3.4 support Jun 25, 2021
@peternowee peternowee merged commit 4d45fff into pydot:master Jul 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants