Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: migrate to pyproject.toml #1068

Merged
merged 49 commits into from
Jan 16, 2024
Merged
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
47f8537
Remove obsoleted files.
tonyandrewmeyer Nov 16, 2023
c942cc7
Organise the build files together and ensure that version.py doesn't …
tonyandrewmeyer Nov 16, 2023
a160ce8
Migrate data from setup.py.
tonyandrewmeyer Nov 16, 2023
a0feaeb
Exclude some files from the sdist.
tonyandrewmeyer Nov 16, 2023
0d18f51
Update the dependency location.
tonyandrewmeyer Nov 16, 2023
0b8f710
Remove auto-generated file.
tonyandrewmeyer Nov 16, 2023
b58abed
Update with email address.
tonyandrewmeyer Nov 16, 2023
edca060
Move the requirements into pyproject.toml.
tonyandrewmeyer Nov 16, 2023
9fedd2e
Remove outdated tests.
tonyandrewmeyer Nov 16, 2023
66b8057
Use build to create the distribution package.
tonyandrewmeyer Nov 16, 2023
96d0c39
Update build command.
tonyandrewmeyer Nov 16, 2023
183ff1a
Add pinned versions of requirements files.
tonyandrewmeyer Nov 16, 2023
3790aa4
Add dev dependencies.
tonyandrewmeyer Nov 16, 2023
54a471e
Add type hint now that the default value is gone.
tonyandrewmeyer Nov 16, 2023
412b68a
Provide instructions for generating the doc requirements.
tonyandrewmeyer Nov 19, 2023
2ce49c1
Fix spacing around import.
tonyandrewmeyer Nov 19, 2023
063abdc
Ensure that the requirements are kept in sync. Ensure version.py exists.
tonyandrewmeyer Nov 19, 2023
c856db6
Ensure setuptools-svm is available to generate the version.py file.
tonyandrewmeyer Nov 19, 2023
e4d41a7
Add setuptools to the dev dependencies so that the version.py file ca…
tonyandrewmeyer Nov 19, 2023
5a921a6
Extend the information on tooling.
tonyandrewmeyer Nov 19, 2023
3ad12d0
Don't bundle the .gitignore, that's just for when you have a clone.
tonyandrewmeyer Nov 19, 2023
e9f0cb6
Merge branch 'main' into pyproject-build-893
tonyandrewmeyer Nov 19, 2023
2947d92
Remove the setuptools-scm dependency.
tonyandrewmeyer Nov 22, 2023
fc47085
Merge branch 'main' into pyproject-build-893
tonyandrewmeyer Nov 22, 2023
993892d
Tell setuptools where to find the version.
tonyandrewmeyer Nov 22, 2023
3190143
We need the dependencies installed to import ops to get the version.
tonyandrewmeyer Nov 22, 2023
3fe6176
Explicitly list the files to include (no longer using setuptools-scm).
tonyandrewmeyer Nov 22, 2023
20ca414
Pin using Python 3.8.
tonyandrewmeyer Nov 22, 2023
adde28f
Merge branch 'main' into pyproject-build-893
tonyandrewmeyer Dec 12, 2023
2a50930
Merge branch 'main' into pyproject-build-893
tonyandrewmeyer Jan 11, 2024
71cedc2
Update HACKING.md
tonyandrewmeyer Jan 12, 2024
3b7c10a
Update HACKING.md
tonyandrewmeyer Jan 12, 2024
e6a23c3
Bump pyright version to match main's requirements-dev.txt value.
tonyandrewmeyer Jan 11, 2024
7f32a09
Address review comments.
tonyandrewmeyer Jan 12, 2024
9edc6fc
Fix smoke name.
tonyandrewmeyer Jan 12, 2024
339740d
Style fix.
tonyandrewmeyer Jan 12, 2024
556d139
requirements-dev is now not used.
tonyandrewmeyer Jan 12, 2024
b6837c9
Per code review.
tonyandrewmeyer Jan 12, 2024
0d726d9
Bump to next expected version.
tonyandrewmeyer Jan 12, 2024
50b6386
Fix the manifest to pick up the correct files.
tonyandrewmeyer Jan 12, 2024
1560224
Update HACKING.md for latest changes.
tonyandrewmeyer Jan 12, 2024
fadd413
Move to putting the dev dependencies direclty in tox.ini.
tonyandrewmeyer Jan 14, 2024
e1be35e
Minor cleanup after change to deps in tox.ini.
tonyandrewmeyer Jan 14, 2024
06de826
Merge branch 'main' into pyproject-build-893
tonyandrewmeyer Jan 14, 2024
8867b1d
Ensure there is a requirements file for readthedocs to use - and it's…
tonyandrewmeyer Jan 14, 2024
84fcb74
Update pyproject.toml
tonyandrewmeyer Jan 15, 2024
99ae4ee
Update tox.ini
tonyandrewmeyer Jan 15, 2024
18db4e5
Regenerate using Python 3.8
tonyandrewmeyer Jan 15, 2024
c9c325d
Do a local install for rtd.
tonyandrewmeyer Jan 15, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
15 changes: 11 additions & 4 deletions .github/workflows/framework-tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -93,20 +93,27 @@ jobs:
PEBBLE: /tmp/pebble

pip-install:
runs-on: ubuntu-latest
runs-on: ${{ matrix.os }}

strategy:
matrix:
os: [ubuntu-latest, macos-latest]
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]

steps:
- uses: actions/checkout@v3
- name: Set up Python 3
uses: actions/setup-python@v2
with:
python-version: '3.11'
python-version: ${{ matrix.python-version }}

- name: Install build dependencies
run: pip install wheel build

- name: Build
run: python setup.py sdist
run: python -m build

# Test that a pip install of the source dist .tar.gz will work
- name: Test 'pip install'
# Shouldn't happen, but pip install will fail if ls returns multiple lines
run: pip install $(ls dist/ops*.gz)

8 changes: 3 additions & 5 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,10 @@ jobs:
- uses: actions/checkout@v3
- name: Setup Python
uses: actions/setup-python@v1
with:
python-version: "3.10"
- name: Install wheel
run: pip install wheel
- name: Install build dependencies
run: pip install wheel build
- name: Build
run: python setup.py sdist bdist_wheel
run: python -m build
- name: Publish
uses: pypa/gh-action-pypi-publish@release/v1
with:
Expand Down
9 changes: 6 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,10 +1,7 @@
__pycache__
/sandbox
/build
/dist
/ops.egg-info
.idea
/docs/_build
*~
.venv
venv
Expand All @@ -13,6 +10,12 @@ venv
.coverage
/.tox

# Build artifacts
/dist
/build
/docs/_build
ops/version.py

# Smoke test artifacts
*.tar.gz
*.charm
Expand Down
67 changes: 64 additions & 3 deletions HACKING.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,13 +107,74 @@ next to the relevant content (e.g. headings, etc.).

Noteworthy changes should also get a new entry in [CHANGES.md](CHANGES.md).

To update the requirements list for the documentation, ensure that
[pip-tools](https://pypi.org/project/pip-tools/) is installed and run, in the
root project folder:

## Dependencies
```sh
pip-compile --extra=docs -o docs/requirements.txt pyproject.toml
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder -- should/could we make these tox actions, rather than having the developer manually install pip-compile and enter these commands (also below)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's only required when you actually want to bump a dependency, but yes a tool is likely better.

We could have it like this instead, avoiding any lock file at all:

[testenv:docs]
description = Build the Sphinx docs
deps = .[docs]
commands =
    sphinx-build -W --keep-going docs/ docs/_build/html

The biggest issue is that if the dependencies change, pip doesn't pick up on that, and you generally have to blow away the .tox/env folder, which is not much better than running a command.

We could have the dev dependencies in tox.ini, particularly since they're being spilt out, and especially if we don't group many of them (or if there is a way to say "use this same dependency list for multiple envs", other than having them in another file). One advantage of having them in standard places is that tools (like security scanners) easily find them, though, and I'm not sure if they would there.

Or I can just do as you suggest and we keep the generated files (probably not in version control, per a different thread) and I can add a tox -e update-deps (name to be bikesheded later).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just having the .[unit] style deps value in tox.ini is no good - it avoids needing any compilation of what's in pyproject.toml but the failure to pick up on changes is too inconvenient.

In terms of speed (I did these 3 times and picked the median - not super accurate, but should be enough to spot order-of-magnitude differences), for a full tox:

Clean Existing env
current main 1m47s 1m15s
compile & sync 1m38s 1m27s
sync only 1m23s 1m12s
deps-in-tox.ini 1m19s 1m12s

However, most of the time is (thankfully!) in the tests. For a quick env, like lint:

Clean Existing env
current main 13s 3s
compile & sync 11s 8s
sync only 6s 3s
deps-in-tox.ini 5s 3s

I did some more investigation, and dependabot will find dependencies in pyproject.toml without having a lock file committed, but won't find them in tox.ini. Other tools seem mixed, but as far as I know we aren't using any at the moment, so I guess we could address that when/if that changes.

I don't want to be significantly slower than current main. So that eliminates having both the pip-compile and pip-sync in tox.ini.

Having the dependencies in optional-dependencies sections in pyproject.toml is recommended in some places and in others it seems like that's used only for optional features rather than dev dependencies. Being able to do a local install with pip install .[static,unit] is nice, but it seems like the vast majority of the time people will be using tox, and doing pip install ops[unit] doesn't seem like it's of use. Having requirements files generated seems inconvenient if we're not using them for anything else.

So, on balance, I think putting them directly in tox.ini is the cleanest for now, and we can reconsider this if the scanning issue ever comes up.

Except for the docs: readthedocs doesn't have tox, so we do need to have a generated requirements.txt file for there.

```

# Dependencies

The Python dependencies of `ops` are kept as minimal as possible, to avoid
bloat and to minimise conflict with the charm's dependencies. The dependencies
are listed in [requirements.txt](requirements.txt).
are listed in [pyproject.toml](pyproject.toml) in the `[project] dependencies` section.
tonyandrewmeyer marked this conversation as resolved.
Show resolved Hide resolved

If the dependencies change, use [pip-tools](https://pypi.org/project/pip-tools/)
to update the generated dependency lockfiles:

```sh
pip-compile -o requirements.txt pyproject.toml
pip-compile --extra=dev -o requirements-dev.txt pyproject.toml
```

# Dev Tools

## Formatting and Checking

Test environments are managed with [tox](https://tox.wiki/) and executed with
[pytest](https://pytest.org), with coverage measured by
[coverage](https://coverage.readthedocs.io/en/7.3.2/).
Static type checking is done using [pyright](https://github.com/microsoft/pyright),
and extends the Python 3.8 type hinting support through the
[typing_extensions](https://pypi.org/project/typing-extensions/) package.

Formatting uses [isort](https://pypi.org/project/isort/) and
[autopep8](https://pypi.org/project/autopep8/), with linting also using
[flake8](https://github.com/PyCQA/flake8), including the
[docstrings](https://pypi.org/project/flake8-docstrings/),
[builtins](https://pypi.org/project/flake8-builtins/) and
[pep8-naming](https://pypi.org/project/pep8-naming/) extensions.

All tool configuration is kept in [project.toml](pyproject.toml).

## Building

The build backend is [setuptools](https://pypi.org/project/setuptools/), and
the build frontend is [build](https://pypi.org/project/build/).

The version number is automatically generated from Git metadata using the
[setuptools-scm package](https://pypi.org/project/setuptools-scm/). During the
build process, an ``ops/version.py`` is automatically generated so that if you
are running from an installed package (including an editable install), you can
statically access the version:

```python
import ops
print(ops.__version__)
import ops.version
print(ops.version.version_tuple)
```

[This is pretty awful, and it seems like this cannot be the best way to do this]: #

Importing `ops` outside of an install will fail because the generated version
module does not exist. It can be generated using:

```sh
python -c 'import setuptools_scm;conf=setuptools_scm.Configuration.from_file();setuptools_scm.dump_version(conf.root,setuptools_scm.get_version(),conf.version_file,conf.version_file_template)'
```

# Publishing a Release

Expand All @@ -126,7 +187,7 @@ To make a release of the ops library, do the following:
4. Drop notes and a changelog in the description.
5. When you are ready, click "Publish". (If you are not ready, click "Save as Draft".)

This will trigger an automatic build for the Python package and publish it to PyPI (the API token/secret is already set up in the repository settings).
This will trigger an automatic build for the Python package and publish it to PyPI (authorization is handled via a [Trusted Publisher](https://docs.pypi.org/trusted-publishers/) relationship).

See [.github/workflows/publish.yml](.github/workflows/publish.yml) for details. (Note that the versions in publish.yml refer to versions of the GitHub actions, not the versions of the ops library.)

Expand Down
5 changes: 5 additions & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
recursive-exclude .github/
recursive-exclude docs/
exclude .readthedocs.yaml
exclude .gitignore

7 changes: 0 additions & 7 deletions docs/requirements.in

This file was deleted.