Making a Python Package

Specifying how to build your python package

Python Packages

What is a "package" in Python ?

Packages, modules, imports, oh my!

Before we get started on making your own package -- let's remind ourselves about packages and modules, and importing....

Modules

A python "module" is a single namespace, with a collection of values:

functions

constants

class definitions

really any old value.

A module usually corresponds to a single file: something.py

Packages

A "package" is essentially a module, except it can have other modules (and indeed other packages) inside it.

A package usually corresponds to a directory with a file in it called __init__.py and any number of python files or other package directories:

a_package
   __init__.py
   module_a.py
   a_sub_package
     __init__.py
     module_b.py

The __init__.py can be totally empty -- or it can have arbitrary python code in it. The code will be run when the package is imported -- just like a module,

modules inside packages are not automatically imported. So, with the above structure:

import a_package

will run the code in a_package/__init__.py. Any names defined in the __init__.py will be available in:

a_package.a_name

but:

a_package.module_a

will not exist. To get submodules, you need to explicitly import them:

import a_package.module_a

https://docs.python.org/3/tutorial/modules.html#packages

The module search path

The interpreter keeps a list of all the places that it looks for modules or packages when you do an import:

import sys
for p in sys.path:
    print p

You can manipulate that list to add or remove paths to let python find modules on a new place.

And every module has a __file__ name that points to the path it lives in. This lets you add paths relative to where you are, etc.

NOTE it's usually better to use setuptools' "develop" mode instead -- see below.

Building Your Own Package

The very basics of what you need to know to make your own package.

Why Build a Package?

There are a bunch of nifty tools that help you build, install and distribute packages.

Using a well structured, standard layout for your package makes it easy to use those tools.

Even if you never want to give anyone else your code, a well structured package eases development.

What is a Package?

A collection of modules

... and the documentation
... and the tests
... and any top-level scripts
... and any data files required
... and a way to build and install it...

Python packaging tools:

The distutils:

from distutils.core import setup

Getting klunky, hard to extend, maybe destined for deprecation ...

But it gets the job done -- and it does it well for the simple cases.

setuptools: for extra features

"The Python Packaging Authority" -- PaPA

https://www.pypa.io/en/latest/

Where do I go to figure this out?

This is a really good guide:

Python Packaging User Guide:

https://packaging.python.org/

and a more detailed tutorial:

http://python-packaging.readthedocs.io/en/latest/

Follow one of them

There is a sample project here:

https://github.com/pypa/sampleproject

(this has all the complexity you might need...)

You can use this as a template for your own packages.

Here is an opinionated update -- a little more fancy, but some good ideas:

https://blog.ionelmc.ro/2014/05/25/python-packaging/

Rather than doing it by hand, you can use the nifty "cookie cutter" project:

https://cookiecutter.readthedocs.io/en/latest/

And there are a few templates that can be used with that.

The core template written by the author:

https://github.com/audreyr/cookiecutter-pypackage

And one written by the author of the opinionated blog post above:

https://github.com/ionelmc/cookiecutter-pylibrary

Either are great starting points.

Basic Package Structure:

package_name/
    bin/
    CHANGES.txt
    docs/
    LICENSE.txt
    MANIFEST.in
    README.txt
    setup.py
    package_name/
          __init__.py
          module1.py
          module2.py
          test/
              __init__.py
              test_module1.py
              test_module2.py

CHANGES.txt: log of changes with each release

LICENSE.txt: text of the license you choose (do choose one!)

MANIFEST.in: description of what non-code files to include

README.txt: description of the package -- should be written in ReST (for PyPi):

(http://docutils.sourceforge.net/rst.html)

setup.py: the script for building/installing package.

bin/: This is where you put top-level scripts

( some folks use scripts )

docs/: the documentation

package_name/: The main package -- this is where the code goes.

test/: your unit tests. Options here:

Put it inside the package -- supports :

$ pip install package_name
>> import package_name.test
>> package_name.test.runall()

Or keep it at the top level.

Some notes on that:

Where to put Tests

The `setup.py` File

Your setup.py file is what describes your package, and tells setuptools how to package, build and install it

It is python code, so you can add anything custom you need to it

But in the simple case, it is essentially declarative.

http://docs.python.org/3/distutils/

What Does `setup.py` Do?

Version & package metadata
List of packages to include
List of other files to include
Lists of dependencies
Lists of extensions to be compiled

An example `setup.py`:

from setuptools import setup

setup(
  name='PackageName',
  version='0.1.0',
  author='An Awesome Coder',
  author_email='aac@example.com',
  packages=['package_name', 'package_name.test'],
  scripts=['bin/script1','bin/script2'],
  url='http://pypi.python.org/pypi/PackageName/',
  license='LICENSE.txt',
  description='An awesome package that does something',
  long_description=open('README.txt').read(),
  install_requires=[
      "Django >= 1.1.1",
      "pytest",
  ],

)

`setup.cfg`

NOTE: this is usually a pretty advanced option -- simple packages don't need this.

setup.cfg provides a way to give the end user some ability to customize the install

It's an ini style file:

[command]
option=value
...

simple to read and write.

command is one of the Distutils commands (e.g. build_py, install)

option is one of the options that command supports.

Note that an option spelled --foo-bar on the command-line is spelled ffoo_bar in configuration files.

Running `setup.py`

With a setup.py script defined, setuptools can do a lot:

builds a source distribution (a tar archive of all the files needed to build and install the package):
```
python setup.py sdist
```
builds wheels:
```
./setup.py bdist_wheel
```

(you need the wheel package for this to work: pip install wheel)

build from source:
```
python setup.py build
```
and install:
```
python setup.py install
```
install in "develop" or "editable" mode:
```
python setup.py develop
```

or:

pip install .

setuptools

setuptools is an extension to distutils that provides a number of extensions:

from setuptools import setup

superset of the distutils setup

This buys you a bunch of additional functionality:

auto-finding packages

better script installation

resource (non-code files) management

develop mode

a LOT more

http://pythonhosted.org//setuptools/

Under Development

Develop mode is really, really nice:

$ python setup.py develop

or:

$ pip install -e ./

(the e stands for "editable" -- it is the same thing)

It puts a links into the python installation to your code, so that your package is installed, but any changes will immediately take effect.

This way all your test code, and client code, etc, can all import your package the usual way.

No sys.path hacking

Good idea to use it for anything more than a single file project.

Install	Development Install
Copies package into site-packages	Adds a `.pth` file to site-packages, pointed at package source root
Used when creating conda packages	Used when developing software locally
Normal priority in sys.path	End of `sys.path` (only found if nothing else comes first)

https://grahamwideman.wikispaces.com/Python-+site-package+dirs+and+.pth+files

Getting Started With a New Package

For anything but a single-file script (and maybe even then):

Create the basic package structure
Write a setup.py
pip install -e .
Put some tests in package/test
pytest in the test dir, or pytest --pyargs package_name

or use "Cookie Cutter":

https://cookiecutter.readthedocs.io/en/latest/

Exercise: A Small Example Package

Create a small package
- package structure
- setup.py
- python setup.py develop
- at least one working test

Start with the silly code in:

capitalize.zip <examples/capitalize.zip>

Let’s Make a Package

mypkg/
    __init__.py
    subpkg/
        __init__.py
        a.py

Windows:

mkdir mypkg/subpkg

echo. > mypkg/__init__.py

echo . > mypkg/subpkg/__init__.py

echo . > mypkg/subpkg/a.py

Mac/Linux:

mkdir -p mypkg/subpkg

touch mypkg/__init__.py

touch mypkg/subpkg/__init__.py

touch mypkg/subpkg/a.py

Let’s Write a `setup.py`

#!/usr/bin/env python

from setuptools import setup

setup(name='mypkg',
      version='1.0',
      # list folders, not files
      packages=['mypkg', 'mypkg.subpkg'],
      )

(remember that a "package" is a folder with a __init__.py__ file)

Try installing your package

cd mypkg-src

python setup.py install

python -c “import mypkg.subpkg.a”

Go look in your site-packages folder

Making Packages the Easy Way

github.com/audreyr/cookiecutter

conda install -c conda-forge cookiecutter

or

pip install  cookiecutter

No time for that now :-(

Handling Requirements

Only the simplest of packages need only the Python standard library.

Requirements in `setup.py`

#!/usr/bin/env python
from distutils.core import setup

setup(name='mypkg',
      version='1.0',
      # list folders, not files
      packages=['mypkg', 'mypkg.subpkg'],
      install_requires=['click'],
      )

Requirements in `requirements.txt`

Common Mistake:

requirements.txt often from pip freeze
Pinned way too tightly. OK for env creation, bad for packaging.

Donald Stufft (PyPA): Abstract vs. Concrete dependencies

Requirements in `setup.cfg` (ideal)

[metadata]
name = my_package
version = attr:
src.VERSION

[options]
packages = find:
install_requires = click

Parse-able without execution, unlike setup.py

configuring setup using setup cfg files

Exercise

Fill in the missing pieces in a setup.py for a sample package
Do a development install for the package

Break time!

Up next: producing redistributable artifacts

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

setup_py.rst

setup_py.rst

Making a Python Package

Python Packages

Packages, modules, imports, oh my!

The module search path

Building Your Own Package

Why Build a Package?

What is a Package?

Python packaging tools:

Where do I go to figure this out?

Basic Package Structure:

The `setup.py` File

What Does `setup.py` Do?

An example `setup.py`:

`setup.cfg`

Running `setup.py`

setuptools

Under Development

Getting Started With a New Package

Exercise: A Small Example Package

Let’s Make a Package

Let’s Write a `setup.py`

Try installing your package

Making Packages the Easy Way

Handling Requirements

Requirements in `setup.py`

Requirements in `requirements.txt`

Requirements in `setup.cfg` (ideal)

Exercise

Break time!

Files

setup_py.rst

Latest commit

History

setup_py.rst

File metadata and controls

Making a Python Package

Python Packages

Packages, modules, imports, oh my!

The module search path

Building Your Own Package

Why Build a Package?

What is a Package?

Python packaging tools:

Where do I go to figure this out?

Basic Package Structure:

The setup.py File

What Does setup.py Do?

An example setup.py:

setup.cfg

Running setup.py

setuptools

Under Development

Getting Started With a New Package

Exercise: A Small Example Package

Let’s Make a Package

Let’s Write a setup.py

Try installing your package

Making Packages the Easy Way

Handling Requirements

Requirements in setup.py

Requirements in requirements.txt

Requirements in setup.cfg (ideal)

Exercise

Break time!

The `setup.py` File

What Does `setup.py` Do?

An example `setup.py`:

`setup.cfg`

Running `setup.py`

Let’s Write a `setup.py`

Requirements in `setup.py`

Requirements in `requirements.txt`

Requirements in `setup.cfg` (ideal)