Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and

## Unreleased

- [#874](https://github.com/pytask-dev/pytask/pull/874) improves the lockfile
documentation by restructuring related guides around user workflows and introducing
`pytask.lock` in the tutorials.
- [#868](https://github.com/pytask-dev/pytask/pull/868) resets the global marker
configuration during unconfigure so `--strict-markers` no longer leaks into later
marker access in the same process.
Expand Down
5 changes: 5 additions & 0 deletions docs/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@

## General

- The structure of the documentation follows https://diataxis.fr/. When writing or
editing an article, read the relevant guidance from the Diataxis Framework before:
https://diataxis.fr/tutorials, https://diataxis.fr/how-to-guides/,
https://diataxis.fr/explanation/, https://diataxis.fr/reference/.
https://diataxis.fr/compass/ tells you where belongs what and how do they relate.
- Document only public APIs and user-facing behavior - exclude internals, framework
abstractions, and implementation plumbing - Users need actionable documentation on
what they can use, not confusing details about internal mechanics they can't control
Expand Down
1 change: 0 additions & 1 deletion docs/CLAUDE.md

This file was deleted.

4 changes: 2 additions & 2 deletions docs/source/how_to_guides/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ specific tasks with pytask.

- [Migrating From Scripts To Pytask](migrating_from_scripts_to_pytask.md)
- [Interfaces For Dependencies Products](interfaces_for_dependencies_products.md)
- [Portability](portability.md)
- [Update the Lockfile to Match Project State](reconciling_lockfile_state.md)
- [Move a Project to Another Machine](move_project_to_another_machine.md)
- [Update the Lockfile to Match Project State](update_the_lockfile_to_match_project_state.md)
- [Remote Files](remote_files.md)
- [Functional Interface](functional_interface.md)
- [Capture Warnings](capture_warnings.md)
Expand Down
77 changes: 77 additions & 0 deletions docs/source/how_to_guides/move_project_to_another_machine.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Move a Project to Another Machine

This guide teaches you how to move a pytask project to another machine or environment
and reuse existing outputs where possible.

## Update the lockfile on the source machine

Run a normal build with [`pytask build`](../reference_guides/commands.md#pytask-build)
before moving the project with its `pytask.lock` and files and outputs are up-to-date:

```console
$ pytask build
```

## Move the project files and reusable outputs

If you have not done it yet, commit `pytask.lock` to your repository and move it with
the project. In practice, move:

- the project files tracked in version control, including source files, configuration,
data inputs, and `pytask.lock`
- the build artifacts you want to reuse, often in `bld/` if you follow the tutorial
layout
- the `.pytask` folder if you use the data catalog and it manages some of your files

## Keep external files in the same relative layout

If tasks use files outside the project root, keep the same relative layout on the target
machine. The project root is the folder with the `pyproject.toml` file.

For example, if a task reads `../shared/input.csv` from the source machine, the moved
project also needs a readable `../shared/input.csv` next to the project root on the
target machine.

## Run pytask on the target machine

After you moved the project to the target machine, run pytask to build the project:

```console
$ pytask build
```

Assuming that the project was fully built before the move, pytask will not rebuild the
project and skip all tasks.

## Clean stale lockfile entries

If you removed, renamed, or moved tasks before transferring the project, clean up stale
lockfile entries on the source machine before you move the project:

```console
$ pytask build --clean-lockfile
```

This rewrites the lockfile after a successful build with only the currently collected
tasks and their current state values.

## If your project uses custom nodes

Make sure custom node IDs and state values stay stable across machines:

- Use project-relative IDs instead of absolute paths.
- Prefer file content hashes over timestamps.
- Avoid machine-specific paths or timestamps in custom
[`state()`](../api/nodes_and_tasks.md#pytask.PNode.state) implementations.
- Provide a custom hash function for
[`PythonNode`](../api/nodes_and_tasks.md#pytask.PythonNode) values that are not
natively stable.

Most projects that only use built-in nodes do not need extra work here.

!!! seealso

The lockfile format and behavior are documented in the
[reference guide](../reference_guides/lockfile.md). For custom nodes, see
[Writing custom nodes](writing_custom_nodes.md). For hashing guidance, see
[Hashing inputs of tasks](hashing_inputs_of_tasks.md).
92 changes: 0 additions & 92 deletions docs/source/how_to_guides/portability.md

This file was deleted.

6 changes: 6 additions & 0 deletions docs/source/tutorials/defining_dependencies_products.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ my_project
│ ├────task_data_preparation.py
│ └────task_plot_data.py
├───pytask.lock
└───pyproject.toml
```

Expand Down Expand Up @@ -107,6 +109,10 @@ Now, let us execute the two paths.

--8<-- "docs/source/_static/md/defining-dependencies-products.md"

The build updates `pytask.lock` with the state of both tasks. When you run the same
tasks again without changing their dependencies, products, or source files, pytask uses
the lockfile to skip them.

## Relative paths

Dependencies and products do not have to be absolute paths. If paths are relative, they
Expand Down
25 changes: 21 additions & 4 deletions docs/source/tutorials/set_up_a_project.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ The following directory tree gives an overview of the project's different parts.
```text
my_project
├───.pytask
├───.pytask # Generated by pytask.
├───bld
│ └────...
Expand All @@ -30,13 +30,16 @@ my_project
│ ├────config.py
│ └────...
├───pytask.lock # Generated by pytask.
└───pyproject.toml
```

Replicate this directory structure for your project or start from pytask's
Create the project files and folders for your project or start from pytask's
[cookiecutter-pytask-project](https://github.com/pytask-dev/cookiecutter-pytask-project)
template or any other
[linked template or example project](../how_to_guides/bp_templates_and_projects.md).
pytask creates the `.pytask` folder and `pytask.lock` file later when you run tasks.

## The `src` directory

Expand Down Expand Up @@ -129,10 +132,24 @@ The `[tool.pytask.ini_options]` section tells pytask to look for tasks in
`src/my_project`. You will learn more about configuration in the
[configuration tutorial](configuration.md).

## The `pytask.lock` file

The `pytask.lock` file records which tasks and products are up to date. pytask updates
it during builds so later runs can skip unchanged tasks. This file should be kept in
version control.

!!! seealso

You will later learn how to sync the state of the lockfile with the project state with
the [`pytask lock`](../reference_guides/commands.md#pytask-lock) command or how the
lockfile enables you to
[move a project to another machine](../how_to_guides/move_project_to_another_machine.md),
but don't worry about it for now.

## The `.pytask` directory

The `.pytask` directory is where pytask stores its information. You do not need to
interact with it.
The `.pytask` directory is where pytask stores some of its ephemeral information. You do
not need to interact with it, nor do you need to keep it in version control.

## Installation

Expand Down
4 changes: 4 additions & 0 deletions docs/source/tutorials/using_a_data_catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ my_project
│ ├────task_data_preparation.py
│ └────task_plot_data.py
├───pytask.lock
└───pyproject.toml
```

Expand Down Expand Up @@ -148,6 +150,8 @@ my_project
├───pyproject.toml
├───pytask.lock
├───src
│ └───my_project
│ ├────config.py
Expand Down
5 changes: 5 additions & 0 deletions docs/source/tutorials/write_a_task.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ my_project
│ ├────config.py
│ └────task_data_preparation.py
├───pytask.lock
└───pyproject.toml
```

Expand Down Expand Up @@ -78,6 +80,9 @@ Now, execute pytask to collect tasks in the current and subsequent directories.

--8<-- "docs/source/_static/md/write-a-task.md"

After the task succeeds, pytask writes `pytask.lock` next to `pyproject.toml`. Keep this
file under version control so later builds can detect unchanged tasks.

<a id="customize-task-names"></a>

## Customize task names
Expand Down
4 changes: 2 additions & 2 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,8 @@ nav:
- Overview: how_to_guides/index.md
- Migrating From Scripts To pytask: how_to_guides/migrating_from_scripts_to_pytask.md
- Interfaces For Dependencies And Products: how_to_guides/interfaces_for_dependencies_products.md
- Portability: how_to_guides/portability.md
- Reconciling Lockfile State: how_to_guides/reconciling_lockfile_state.md
- Move a Project to Another Machine: how_to_guides/move_project_to_another_machine.md
- Update the Lockfile to Match Project State: how_to_guides/update_the_lockfile_to_match_project_state.md
- Remote Files: how_to_guides/remote_files.md
- Functional Interface: how_to_guides/functional_interface.md
- Capture Warnings: how_to_guides/capture_warnings.md
Expand Down