Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

check-python-versions --add 3.8 --drop 3.4 to update packages automatically. #3

Merged
merged 76 commits into from Apr 18, 2019

Conversation

mgedmin
Copy link
Owner

@mgedmin mgedmin commented Apr 12, 2019

Updating all the files by hand is boring, especially when there are many packages (see eg. zopefoundation/meta#1).

This is a proof-of-concept that can update the files if they're formatted nicely enough. Currently it can handle all of my personal projects (https://projects.gedmin.as). I intend to make it handle zopefoundation too, and then see if people report bugs.

For completeness this needs

  • test coverage
  • documentation update

Optionally:

  • suppress repeated warnings about the same file
  • code to update CHANGES.rst

I'm not going to do these at this time:

  • perhaps a command-line option --no-confirm or --noninteractive (do not ask, just write out the changes)
  • what if setup(..., classifiers=[...]) i.e. the closing ) is on the same line as the closing ]? do I care?
  • what if setup(classifiers=[...]) i.e. the keyword arg is on the same line as the opening (? do I care?
  • can classifiers be a tuple? tripple-quoted string? do I want to deal with that? (probably not)
  • support adding/dropping pypy/pypy3

Already done:

  • code to update setup.py (both classifiers and python_requires)
  • code to update tox.ini
  • code to update .travis.yml
  • code to update appevyor.yml
  • code to update manylinux-install.sh
  • command line options to --add, --drop or --update versions wholesale
  • perhaps command-line options to --dry-run (show diff, make no changes)
  • fix it so --add 3.7 doesn't drop unreleased upstream versions like 3.8 and things like pypy
  • handle classifiers=[] like in zopefoundation/z3c.schema2xml
  • decide if it makes sense to add classifiers where there were no classifiers before (probably not)
  • if we add dist: xenial, then pypy/pypy3 will no longer work -> rewrite to pypy2.7-6.0.0 and pypy3.5-6.0.0
  • do not insist on 2-space indents inside matrix: in appveryor.yml

Known problems from testing ./check-python-versions --dry-run --drop -3.4 ~/src/zopefoundation/* (and then searching for 3.4 in the output, meaning the drop didn't succeed -- this will miss other kinds of problems with the diff):

  • zope.app.apidoc's .travis.yml (uses jobs instead of python)
  • zope.component's tox.ini (multiline value preceded by an unindented comment)
  • zope.dottedname's setup.py (spaces around =)
  • zope.event's tox.ini (multiline value preceded by an unindented comment)
  • zope.hookable's .travis.yml (uses matrix instead of python)
  • zope.index's .travis.yml (jobs start with - os: linux instead of - python: ...)
  • zope.interface's .travis.yml (jobs start with - os: linux instead of - python: ...)
  • zope.location's tox.ini (multiline value preceded by an unindented comment)
  • zope.schema's tox.ini (multiline value preceded by an unindented comment)
  • z3c.csvvocabulary's setup.py (setup space ()
  • z3c.schema's setup.py
  • z3c.traverser's setup.py
  • zc.lockfile's setup.py
  • zc.zrs's setup.py (classifiers is a tripple-quoted string assigned to a global variable)
  • ZODB's .travis.yml (jobs start with - os: linux instead of - python: ...)
  • persistent's .travis.yml (jobs start with - os: linux instead of - python: ...)
  • BTrees's .travis.yml (jobs start with - os: linux instead of - python: ...)

Other problems I've noticed:

  • zope.component's .travis.yml loses a python 2.7 matrix item with env: MINIMAL=1 (haha actually due to a bug there are two top-level matrix elements so that one was dropped long ago!)
  • zope.hookable's .travis.yml loses { python: 2.7, env: PURE_PYTHON=1 } job
  • zope.testrunner's appveyor.yml loses a bunch of extra environs
  • BTree's appevyor.yml wants both 32-bit and 64-bit Pythons; we lose half

Let's pass a parser function to argparse directly so we don't have to
manually try/except etc.

(I want this because I intend to add a few additional arguments that
take version ranges.)
check-python-versions {--add|--drop|--update} VERSIONS will attempt to
update your setup.py to add/drop/set the supported version classifiers.

The update script is interactive and shows you the diff and waits for
confirmation before making any changes.
@mgedmin mgedmin added the enhancement New feature or request label Apr 12, 2019
Fixes TypeError in difflib because new_lines happened to be None because
I forgot to return a value when update_call_arg_in_source fails.
When the stdin is redirected from /dev/null, this will apply the default
action (no update).

When the stdin is a terminal, ^D will be the same as hitting Enter and
accepting the default action (no update) once.
(tox.ini still has the --fail-under=100 commented out because I intend
to do more rapid prototyping on this branch.)
Let the updaters return new file contents so we can do what we want with
them.
Support testing both 32-bit and 64-bit Pythons.
as long as they fit on one line.
"Better" means if all the build jobs are defined via matrix.include
or jobs.include, then we'll attempt to modify those.

Also handle environment.matrix better in appveyor.yml, and here "better"
means without assuming 2-space indents.
The logic here is very fragile and I think only the very last job will
be kept intact.
Lists nested more deeply could've confused the "parser".
@mgedmin mgedmin marked this pull request as ready for review April 17, 2019 19:35
@mgedmin mgedmin changed the title WIP: check-python-versions --add 3.8 --drop 3.4 to update packages automatically. check-python-versions --add 3.8 --drop 3.4 to update packages automatically. Apr 17, 2019
@mgedmin mgedmin merged commit 1ec4c00 into master Apr 18, 2019
@mgedmin mgedmin deleted the update-support branch April 18, 2019 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant