Skip to content
How many packages will PEP 594 break?
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.gitignore
README.md
go.mod
go.sum
imports.csv
imports.py
main.go
mirror.go
packages.json
search.go
stats.go

README.md

Dead Batteries

PEP 594 outlines the plan to deprecate and remove packages from the Python standard library. If accepted in its current form, PEP 594 will break 3.8% of all Python 3 packages on PyPI.

Results

As of May 2019 there are 3604 Python 3 packages on PyPI (out of 94,680) that import packages deprecated by PEP 594.

  total packages: 181225
  valid packages: 178642
python3 packages: 94680
scanned packages: 92779
 broken packages: 3604

Instructions

Before you begin, you'll need Python >= 3.6 and a recent version of Go to run the code. All commands should be run inside a cloned checkout of the this repository. First, build the binary.

go build

Next, you'll download the metadata for every package on PyPI, which will take up about ~500MB of space.

./dead-battery mirror
  • simple.html: contains a list of all known packages on PyPI
  • meta/*.json: contains package metadata for each package
  • python3-packages.json: contains a list of packages that support Python3

With a complete mirror, you're now ready to scan packages for imports of deprecated packages. Open up a new shell to install and run the Python service.

python3 -m venv venv
venv/bin/pip install flask
venv/bin/pip install gunicorn
venv/bin/pip install gunicorn[gevent]
venv/bin/gunicorn -k gevent -w 10 -b 127.0.0.1:4000 imports:app

The service exposes a HTTP/JSON interface to the ast package. It parses a given file and returns any deprecated imports as well as parsing errors.

// INPUT
{
  "path": "/path/to/python/file"
}

// OUTPUT
{ 
  "imports": {
    "imp": 2
  },
  "errors": {
    "syntax-error": 2
  }
}

With that running, you can now start the scan. On my laptop, the scan took about an hour to complete. The output is continually saved to results.json.

./dead-battery scan

Once the search process is complete, generate the package statistics.

./dead-battery stats

Two new files have been created: import.csv and packages.json. The CSV file contains the total number of imports for each deprecated standard library package. The JSON file contains a list of every package that imports one of these deprecated packages, along with a link to the package on PyPI.

A quick jump into the Python interpreter gives us the total number of packages affected by PEP 594.

>>> import json
>>> len(json.load(open('packages.json')))
3604

Methodology

Since PEP 594 only affects Python 3, I needed to filter out packages that don't support Python 3. For each package, I first looked for any classifiers with the prefix Programming Language :: Python :: 3. Next, I checked the python_version of the latest release. It's a bit messy, but the code to do so can be found here.

{
    "info": {
        "classifiers": [
            "Programming Language :: Python :: 2.7",
            "Programming Language :: Python :: 3"
        ],
        "version": "2.22.0"
    },
    "releases": {
        "2.22.0": [
            {
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "url": "https://files.pythonhosted.org/.../requests-2.22.0-py2.py3-none-any.whl"
            }
        ]
    }
}

If you know a better way to check for Python 3 compatibility, please reach out or open an issue.

You can’t perform that action at this time.