Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Dead Batteries

PEP 594 outlines the plan to deprecate and remove packages from the Python standard library. If accepted in its current form, PEP 594 will break 3.8% of all Python 3 packages on PyPI.

Results

As of May 2019 there are 3604 Python 3 packages on PyPI (out of 94,680) that import packages deprecated by PEP 594.

  total packages: 181225
  valid packages: 178642
python3 packages: 94680
scanned packages: 92779
 broken packages: 3604

Instructions

Before you begin, you'll need Python >= 3.6 and a recent version of Go to run the code. All commands should be run inside a cloned checkout of the this repository. First, build the binary.

go build

Next, you'll download the metadata for every package on PyPI, which will take up about ~500MB of space.

./dead-battery mirror
  • simple.html: contains a list of all known packages on PyPI
  • meta/*.json: contains package metadata for each package
  • python3-packages.json: contains a list of packages that support Python3

With a complete mirror, you're now ready to scan packages for imports of deprecated packages. Open up a new shell to install and run the Python service.

python3 -m venv venv
venv/bin/pip install flask
venv/bin/pip install gunicorn
venv/bin/pip install gunicorn[gevent]
venv/bin/gunicorn -k gevent -w 10 -b 127.0.0.1:4000 imports:app

The service exposes a HTTP/JSON interface to the ast package. It parses a given file and returns any deprecated imports as well as parsing errors.

// INPUT
{
  "path": "/path/to/python/file"
}

// OUTPUT
{ 
  "imports": {
    "imp": 2
  },
  "errors": {
    "syntax-error": 2
  }
}

With that running, you can now start the scan. On my laptop, the scan took about an hour to complete. The output is continually saved to results.json.

./dead-battery scan

Once the search process is complete, generate the package statistics.

./dead-battery stats

Two new files have been created: import.csv and packages.json. The CSV file contains the total number of imports for each deprecated standard library package. The JSON file contains a list of every package that imports one of these deprecated packages, along with a link to the package on PyPI.

A quick jump into the Python interpreter gives us the total number of packages affected by PEP 594.

>>> import json
>>> len(json.load(open('packages.json')))
3604

Methodology

Since PEP 594 only affects Python 3, I needed to filter out packages that don't support Python 3. For each package, I first looked for any classifiers with the prefix Programming Language :: Python :: 3. Next, I checked the python_version of the latest release. It's a bit messy, but the code to do so can be found here.

{
    "info": {
        "classifiers": [
            "Programming Language :: Python :: 2.7",
            "Programming Language :: Python :: 3"
        ],
        "version": "2.22.0"
    },
    "releases": {
        "2.22.0": [
            {
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "url": "https://files.pythonhosted.org/.../requests-2.22.0-py2.py3-none-any.whl"
            }
        ]
    }
}

If you know a better way to check for Python 3 compatibility, please reach out or open an issue.

About

How many packages will PEP 594 break?

Resources

Releases

No releases published

Packages

No packages published