Example: Forking long training runs

[LeoRoccoBreedt]

Description

Include a summary of the changes and the related issue.

Related to: <ClickUp/JIRA task name>

Any expected test failures?
Torch support for Python 3.13.

---

Add a [X] to relevant checklist items

❔ This change

• adds a new feature
• fixes breaking code
• is cosmetic (refactoring/reformatting)

---

✔️ Pre-merge checklist

• Refactored code (sourcery)
• Tested code locally
• Precommit installed and run before pushing changes
• Added code to GitHub tests (notebooks, scripts)
• Updated GitHub README
• Updated the projects overview page on Notion

---

🧪 Test Configuration

• OS: Windows
• Python version: 3.12
• Neptune version: 0.12
• Affected libraries with version:

Summary by Sourcery

Add a new how-to guide notebook demonstrating how to fork and resume long training runs with Neptune and include it in the CI test-notebooks workflow.

New Features:

• Introduce a tutorial notebook on forking and resuming long training runs with Neptune.

Documentation:

• Add user-facing how-to guide for long-run forking in the documentation.

Tests:

• Update the test-notebooks CI workflow to include the new forking-long-runs notebook.

Summary by Sourcery

Add a tutorial notebook for forking long model training runs and include it in the CI test-notebooks workflow

New Features:

• Introduce a how-to guide notebook demonstrating forking and resuming long training runs with Neptune

Documentation:

• Add user-facing documentation for long-run forking in the how-to-guides

Tests:

• Update the test-notebooks CI workflow to include the new forking-long-runs notebook

[sourcery-ai]

Reviewer's Guide

Introduce a new how-to notebook demonstrating forking, checkpointing, and parallel long-training runs with Neptune, and add it to the CI test-notebooks workflow.

Sequence diagram for forking and resuming a training run

Loading

sequenceDiagram
    actor User
    participant Notebook
    participant NeptuneRun
    participant Checkpoint
    User->>Notebook: Load checkpoint
    Notebook->>Checkpoint: Read model/optimizer state
    User->>Notebook: Create forked Neptune Run
    Notebook->>NeptuneRun: Initialize forked run (fork_run_id, fork_step)
    User->>Notebook: Resume training
    Notebook->>NeptuneRun: Log metrics for forked run

Sequence diagram for launching multiple parallel forks

Loading

sequenceDiagram
    actor User
    participant Notebook
    participant ThreadPoolExecutor
    participant NeptuneRun
    participant Checkpoint
    User->>Notebook: Define trial configs
    User->>Notebook: Launch ThreadPoolExecutor
    Notebook->>ThreadPoolExecutor: Submit forked runs
    ThreadPoolExecutor->>Notebook: For each config
    Notebook->>Checkpoint: Load checkpoint
    Notebook->>NeptuneRun: Initialize forked run
    Notebook->>NeptuneRun: Log metrics for forked run (parallel)
    ThreadPoolExecutor->>User: Return run URLs

Class diagram for SimpleModel and training utilities

Loading

classDiagram
    class SimpleModel {
        +__init__(input_size, hidden_size, output_size, num_layers)
        +forward(x)
        model: nn.Sequential
    }
    class Run {
        +log_metrics(data, step)
        +log_configs(configs)
        +add_tags(tags)
        +close()
        +get_run_url()
        _run_id
    }
    class Checkpoint {
        +save_checkpoint(epoch, global_step, model, optimizer, run)
        +load_checkpoint(model, optimizer, checkpoint_path)
        epoch
        global_step
        model_state_dict
        optimizer_state_dict
        run_id
    }
    class train {
        +train(run, model, params, train_loader, optimizer, epoch_start, step_start, forked)
    }
    SimpleModel <.. train : used by
    Run <.. train : used by
    Checkpoint <.. train : used by
    Checkpoint <.. Run : run_id used for forking
    train <.. ThreadPoolExecutor : used for parallel forks

File-Level Changes

Change	Details	Files
Add tutorial notebook illustrating run forking and checkpoint workflows	Create a new notebook outlining environment setup, model definition, checkpoint save/load, and Neptune initialization Implement training functions that handle initial runs, single forks, and parallel forks with metric logging Provide example code blocks for creating base experiments, loading checkpoints, forking runs, and launching multiple forks Document analysis of fork relationships and Neptune lineage tracking	how-to-guides/forking-long-runs/fork_long_training_runs.ipynb
Register the new notebook in the CI test-notebooks workflow	Insert the new notebook path into the alphabetical list of tested notebooks in test-notebooks.yml	.github/workflows/test-notebooks.yml

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey @LeoRoccoBreedt - I've reviewed your changes and they look great!

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}