Proposal: Poetry setup

[mooster531]

We could use poetry to ship project configuration, which would make it easier to get up and running with a development setup after cloning the repo, as well as managing external dependencies inside the repository.

The following is a very quick crash course to Poetry, for reference.

Poetry Quick Start Guide

A crash course in Poetry for experienced Python developers

Installation

# macOS/Linux
curl -sSL https://install.python-poetry.org | python3 -

# Windows PowerShell
(Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | py -

# Verify installation
poetry --version

Key Concepts

Poetry manages project dependencies, virtual environments, and packaging in a single tool. It uses pyproject.toml for configuration and poetry.lock for dependency locking.

Essential Commands

Project Setup

# Start a new project
poetry new my-project

# Initialize poetry in existing project
poetry init

# Activate virtual environment
poetry shell

Dependency Management

# Add dependencies
poetry add requests           # Add latest version
poetry add requests@^2.25.1   # Add specific version
poetry add --dev pytest      # Add dev dependency

# Remove dependencies
poetry remove requests

# Install from poetry.lock
poetry install

# Update dependencies
poetry update                # Update all
poetry update requests       # Update specific package

Running Scripts

# Run a command in the virtual environment
poetry run python script.py
poetry run pytest

# Execute configured scripts from pyproject.toml
poetry run my-script

pyproject.toml Example

[tool.poetry]
name = "my-project"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]

[tool.poetry.dependencies]
python = "^3.8"
requests = "^2.28.0"

[tool.poetry.dev-dependencies]
pytest = "^7.1.0"

[tool.poetry.scripts]
my-script = "my_project.cli:main"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Advanced Features

Environment Management

# Configure virtualenv location
poetry config virtualenvs.in-project true  # Create venv in project directory

# Show path to virtual environment
poetry env info

# List all virtual environments
poetry env list

# Remove virtual environments
poetry env remove python3.8

Dependency Resolution

# Show dependency tree
poetry show --tree

# Export requirements.txt
poetry export -f requirements.txt --output requirements.txt

# Export with dev dependencies
poetry export -f requirements.txt --with dev --output dev-requirements.txt

Multiple Environment Dependencies

[tool.poetry.dependencies]
python = "^3.8"

[tool.poetry.group.dev.dependencies]
pytest = "^7.1.0"
black = "^22.3.0"

[tool.poetry.group.docs.dependencies]
sphinx = "^4.5.0"

Best Practices

1. Always commit both pyproject.toml and poetry.lock
2. Use poetry add instead of manually editing pyproject.toml
3. Use groups to organize dependencies by environment
4. Specify Python version constraints appropriately
5. Use poetry.lock for reproducible builds

Common Issues & Solutions

• If Poetry is slow: Use poetry config installer.max-workers 10
• Virtual environment conflicts: Delete poetry.lock and .venv, then run poetry install
• Path issues: Ensure Poetry's bin directory is in your PATH
• Dependencies not found: Try poetry cache clear . --all

[bh2smith]

This seems cool and I would be happy to give it a try. However I think there seems to be some stuff that we should update accordingly.

• makefile
• rm -rf requirements/
• Maybe the readme.

[bh2smith]

I think we are missing pyyaml here:

https://github.com/bh2smith/dune-sync/blob/main/requirements/prod.txt

Shouldn't we also delete our requirements directory as well?

[mooster531]

The contents of the requirements directory are still being used by the Dockerfile. I will adapt that to use poetry as well so we can remove it.

Sorry about the missing pyyaml dependency, I added it.

[mooster531]

Done in 817b0a2

[mooster531]

We now clear CI jobs too. Let me know if anything else stands out or needs changing.
Wasn't sure if @fleupold would be okay with his email listed in the pyproject.toml so I masked it. Please let me know if I should put some other (real) email there :)

[bh2smith]

I tried this out and everything looks pretty good here. We can merge this whenever.