Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add checkpoint saving and loading functionality to training loop #123

Merged
merged 74 commits into from
Dec 7, 2020
Merged

Add checkpoint saving and loading functionality to training loop #123

merged 74 commits into from
Dec 7, 2020

Commits on Nov 1, 2020

  1. Add checkpoint saving and loading functionality to training loop

    @lvermue
    lvermue committed Nov 1, 2020
    Configuration menu
    Copy the full SHA
    dc55230 View commit details
    Browse the repository at this point in the history

  2. Flake8 and parameters defaults

    @lvermue
    lvermue committed Nov 1, 2020
    Configuration menu
    Copy the full SHA
    9452a01 View commit details
    Browse the repository at this point in the history

  3. Flake8

    @lvermue
    lvermue committed Nov 1, 2020
    Configuration menu
    Copy the full SHA
    21ab40e View commit details
    Browse the repository at this point in the history

  4. Refactor internal completed epoch tracking

    @lvermue
    lvermue committed Nov 1, 2020
    Configuration menu
    Copy the full SHA
    c05b38b View commit details
    Browse the repository at this point in the history

Commits on Nov 2, 2020

  1. Correct epoch number handling in training loop

    @lvermue
    lvermue committed Nov 2, 2020
    Configuration menu
    Copy the full SHA
    3a0f420 View commit details
    Browse the repository at this point in the history

  2. Add automatic loading of early stoppers from training loop checkpoints

    @lvermue
    lvermue committed Nov 2, 2020
    Configuration menu
    Copy the full SHA
    326cf57 View commit details
    Browse the repository at this point in the history

  3. Add training loop checkpoint support for pipelines

    @lvermue
    lvermue committed Nov 2, 2020
    Configuration menu
    Copy the full SHA
    12f5d9a View commit details
    Browse the repository at this point in the history

  4. Fix indentation

    @lvermue
    lvermue committed Nov 2, 2020
    Configuration menu
    Copy the full SHA
    0af8dbe View commit details
    Browse the repository at this point in the history

  5. Change training loop checksum creation from adler32 to md5

    @lvermue
    lvermue committed Nov 2, 2020
    Configuration menu
    Copy the full SHA
    aaa7eb2 View commit details
    Browse the repository at this point in the history

  6. Fix missing random_seed failure, if no training config file is provided

    @lvermue
    lvermue committed Nov 2, 2020
    Configuration menu
    Copy the full SHA
    7ec0d44 View commit details
    Browse the repository at this point in the history

  7. Flake8, refactor code and add CheckpointMismatchError

    @lvermue
    lvermue committed Nov 2, 2020
    Configuration menu
    Copy the full SHA
    7b84f70 View commit details
    Browse the repository at this point in the history

  8. Fix flake8

    @lvermue
    lvermue committed Nov 2, 2020
    Configuration menu
    Copy the full SHA
    21a982c View commit details
    Browse the repository at this point in the history

  9. Update exception

    @cthoyt
    cthoyt committed Nov 2, 2020
    Configuration menu
    Copy the full SHA
    11e7c4e View commit details
    Browse the repository at this point in the history

  10. Add outline for checkpoint tutorial

    @cthoyt
    cthoyt committed Nov 2, 2020
    Configuration menu
    Copy the full SHA
    5f11c15 View commit details
    Browse the repository at this point in the history

  11. Correct random state recovery

    @lvermue
    lvermue committed Nov 2, 2020
    Configuration menu
    Copy the full SHA
    fc9a5a9 View commit details
    Browse the repository at this point in the history

  12. Merge branch 'allow_training_checkpoints' of https://github.com/pykee… 

    …n/pykeen into allow_training_checkpoints
    @lvermue
    lvermue committed Nov 2, 2020
    Configuration menu
    Copy the full SHA
    2bf5b98 View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2020

  1. Remove pipeline checkpoint helper file help function

    @lvermue
    lvermue committed Nov 22, 2020
    Configuration menu
    Copy the full SHA
    256beaa View commit details
    Browse the repository at this point in the history

  2. Remove torch save helper file

    @lvermue
    lvermue committed Nov 22, 2020
    Configuration menu
    Copy the full SHA
    1e170f3 View commit details
    Browse the repository at this point in the history

  3. Fix function argument for torch save

    @lvermue
    lvermue committed Nov 22, 2020
    Configuration menu
    Copy the full SHA
    3ba5288 View commit details
    Browse the repository at this point in the history

  4. Add units tests for training loop checkpoint

    @lvermue
    lvermue committed Nov 22, 2020
    Configuration menu
    Copy the full SHA
    cfd0598 View commit details
    Browse the repository at this point in the history

Commits on Nov 23, 2020

  1. Merge branch 'master' into allow_training_checkpoints

    @cthoyt
    cthoyt committed Nov 23, 2020
    Configuration menu
    Copy the full SHA
    8d2740f View commit details
    Browse the repository at this point in the history

  2. Add unit tests for checkpoints

    @lvermue
    lvermue committed Nov 23, 2020
    Configuration menu
    Copy the full SHA
    07c3d42 View commit details
    Browse the repository at this point in the history

  3. Fix flake8

    @lvermue
    lvermue committed Nov 23, 2020
    Configuration menu
    Copy the full SHA
    37b777b View commit details
    Browse the repository at this point in the history

  4. Add failure fallback checkpoints to training loop

    @lvermue
    lvermue committed Nov 23, 2020
    Configuration menu
    Copy the full SHA
    8af806f View commit details
    Browse the repository at this point in the history

  5. Correct checkpoint root dir handling

    @lvermue
    lvermue committed Nov 23, 2020
    Configuration menu
    Copy the full SHA
    bc20d35 View commit details
    Browse the repository at this point in the history

Commits on Nov 24, 2020

  1. Add implicit random seed handling from checkpoints in pipeline

    @lvermue
    lvermue committed Nov 24, 2020
    Configuration menu
    Copy the full SHA
    17cdfcc View commit details
    Browse the repository at this point in the history

  2. Code cleanup 

    Store results of get() to make code cleaner
    @cthoyt
    cthoyt committed Nov 24, 2020
    Configuration menu
    Copy the full SHA
    b036b9d View commit details
    Browse the repository at this point in the history

  3. More refactoring

    @cthoyt
    cthoyt committed Nov 24, 2020
    Configuration menu
    Copy the full SHA
    5d2e47b View commit details
    Browse the repository at this point in the history

  4. Add CPU/GPU random state differentiation

    @lvermue
    lvermue committed Nov 24, 2020
    Configuration menu
    Copy the full SHA
    583d494 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'allow_training_checkpoints' of https://github.com/pykee… 

    …n/pykeen into allow_training_checkpoints
    @lvermue
    lvermue committed Nov 24, 2020
    Configuration menu
    Copy the full SHA
    604bc2f View commit details
    Browse the repository at this point in the history

  6. Workaround for CUDA rng state

    @cthoyt
    cthoyt committed Nov 24, 2020
    Configuration menu
    Copy the full SHA
    26e0b39 View commit details
    Browse the repository at this point in the history

  7. Refactor tests

    @cthoyt
    cthoyt committed Nov 24, 2020
    Configuration menu
    Copy the full SHA
    a564fef View commit details
    Browse the repository at this point in the history

  8. Remove unnecessary stuff

    @cthoyt
    cthoyt committed Nov 24, 2020
    Configuration menu
    Copy the full SHA
    67776c8 View commit details
    Browse the repository at this point in the history

  9. Unnest logic

    @cthoyt
    cthoyt committed Nov 24, 2020
    Configuration menu
    Copy the full SHA
    578fd1e View commit details
    Browse the repository at this point in the history

  10. Improve typing and safety 

    Since the type of `stopper` is Optional[Stopper], it doesn't know that this is a function of the EarlyStopper subclass. I added the function to the superclass that does nothing, so it will now make sense for the type checker.

Also, I checked for none in case of misconfiguration
    @cthoyt
    cthoyt committed Nov 24, 2020
    Configuration menu
    Copy the full SHA
    09d03e3 View commit details
    Browse the repository at this point in the history

  11. Fix testing

    @lvermue
    lvermue committed Nov 24, 2020
    Configuration menu
    Copy the full SHA
    9ca40dc View commit details
    Browse the repository at this point in the history

  12. Fix pipeline checkpoint unit tests

    @lvermue
    lvermue committed Nov 24, 2020
    Configuration menu
    Copy the full SHA
    5d64d6c View commit details
    Browse the repository at this point in the history

  13. Merge branch 'master' into allow_training_checkpoints

    @lvermue
    lvermue committed Nov 24, 2020
    Configuration menu
    Copy the full SHA
    bf82e79 View commit details
    Browse the repository at this point in the history

  14. Fix usage of forbidden characters for Windows in filepaths

    @lvermue
    lvermue committed Nov 24, 2020
    Configuration menu
    Copy the full SHA
    661eed3 View commit details
    Browse the repository at this point in the history

Commits on Nov 25, 2020

  1. Refactor loading of states for the training loop and stoppers

    @lvermue
    lvermue committed Nov 25, 2020
    Configuration menu
    Copy the full SHA
    675bd59 View commit details
    Browse the repository at this point in the history

  2. Fix flake8

    @lvermue
    lvermue committed Nov 25, 2020
    Configuration menu
    Copy the full SHA
    f618a90 View commit details
    Browse the repository at this point in the history

  3. Fix handling of stopper state dictionaries

    @lvermue
    lvermue committed Nov 25, 2020
    Configuration menu
    Copy the full SHA
    14e14b7 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into allow_training_checkpoints

    @lvermue
    lvermue committed Nov 25, 2020
    Configuration menu
    Copy the full SHA
    cd64e7e View commit details
    Browse the repository at this point in the history

  5. Fix flake8

    @lvermue
    lvermue committed Nov 25, 2020
    Configuration menu
    Copy the full SHA
    d9abffd View commit details
    Browse the repository at this point in the history

Commits on Nov 29, 2020

  1. Merge branch 'master' into allow_training_checkpoints

    @lvermue
    lvermue committed Nov 29, 2020
    Configuration menu
    Copy the full SHA
    f226b0a View commit details
    Browse the repository at this point in the history

  2. Fix unit tests

    @lvermue
    lvermue committed Nov 29, 2020
    Configuration menu
    Copy the full SHA
    240140b View commit details
    Browse the repository at this point in the history

  3. Refactor pipeline unit tests

    @lvermue
    lvermue committed Nov 29, 2020
    Configuration menu
    Copy the full SHA
    3cf1f69 View commit details
    Browse the repository at this point in the history

  4. Refactor training loop unit tests

    @lvermue
    lvermue committed Nov 29, 2020
    Configuration menu
    Copy the full SHA
    29780d0 View commit details
    Browse the repository at this point in the history

  5. Fix flake8

    @lvermue
    lvermue committed Nov 29, 2020
    Configuration menu
    Copy the full SHA
    bf6190b View commit details
    Browse the repository at this point in the history

Commits on Dec 1, 2020

  1. Merge branch 'master' into allow_training_checkpoints

    @lvermue
    lvermue committed Dec 1, 2020
    Configuration menu
    Copy the full SHA
    04b31c9 View commit details
    Browse the repository at this point in the history

  2. Add saving of checkpoints after successful training

    @lvermue
    lvermue committed Dec 1, 2020
    Configuration menu
    Copy the full SHA
    1cabf5f View commit details
    Browse the repository at this point in the history

  3. Add usage of temporary directories for unit tests

    @lvermue
    lvermue committed Dec 1, 2020
    Configuration menu
    Copy the full SHA
    1fa0f18 View commit details
    Browse the repository at this point in the history

  4. Add checkpoint documentation and correct failure checkpoint handling

    @lvermue
    lvermue committed Dec 1, 2020
    Configuration menu
    Copy the full SHA
    46bb380 View commit details
    Browse the repository at this point in the history

  5. Trigger CI

    @PyKEEN-bot
    PyKEEN-bot committed Dec 1, 2020
    Configuration menu
    Copy the full SHA
    2b32fbb View commit details
    Browse the repository at this point in the history

  6. Add missing variable default value

    @cthoyt
    cthoyt committed Dec 1, 2020
    Configuration menu
    Copy the full SHA
    566e9ab View commit details
    Browse the repository at this point in the history

  7. Get rid of tqdms

    @cthoyt
    cthoyt committed Dec 1, 2020
    Configuration menu
    Copy the full SHA
    778ce48 View commit details
    Browse the repository at this point in the history

  8. Pass flake8

    @cthoyt
    cthoyt committed Dec 1, 2020
    Configuration menu
    Copy the full SHA
    317f639 View commit details
    Browse the repository at this point in the history

  9. Use class teardown for handling temporary directory 

    Trigger CI
    @cthoyt
    cthoyt committed Dec 1, 2020
    Configuration menu
    Copy the full SHA
    563e5a4 View commit details
    Browse the repository at this point in the history

Commits on Dec 2, 2020

  1. Update docs

    @cthoyt
    cthoyt committed Dec 2, 2020
    Configuration menu
    Copy the full SHA
    7fba750 View commit details
    Browse the repository at this point in the history

  2. Update argument names and type hints 

    there are a lot of places where either a string or a pathlib.Path is acceptable
    @cthoyt
    cthoyt committed Dec 2, 2020
    Configuration menu
    Copy the full SHA
    65cefa7 View commit details
    Browse the repository at this point in the history

  3. Trigger CI

    @PyKEEN-bot
    PyKEEN-bot committed Dec 2, 2020
    Configuration menu
    Copy the full SHA
    52656ec View commit details
    Browse the repository at this point in the history

  4. Add datetime formatting

    @lvermue
    lvermue committed Dec 2, 2020
    Configuration menu
    Copy the full SHA
    a745853 View commit details
    Browse the repository at this point in the history

  5. Add docs for checkpoint_on_failure_file_path

    @lvermue
    lvermue committed Dec 2, 2020
    Configuration menu
    Copy the full SHA
    12e3fed View commit details
    Browse the repository at this point in the history

Commits on Dec 4, 2020

  1. Merge branch 'master' into allow_training_checkpoints

    @cthoyt
    cthoyt committed Dec 4, 2020
    Configuration menu
    Copy the full SHA
    86749bf View commit details
    Browse the repository at this point in the history

  2. Update constants

    @cthoyt
    cthoyt committed Dec 4, 2020
    Configuration menu
    Copy the full SHA
    37069be View commit details
    Browse the repository at this point in the history

Commits on Dec 5, 2020

  1. Change temp dir creation and teardown during unit tests

    @lvermue
    lvermue committed Dec 5, 2020
    Configuration menu
    Copy the full SHA
    ffd8e55 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'allow_training_checkpoints' of https://github.com/pykee… 

    …n/pykeen into allow_training_checkpoints
    @lvermue
    lvermue committed Dec 5, 2020
    Configuration menu
    Copy the full SHA
    d285704 View commit details
    Browse the repository at this point in the history

  3. Update the checkpoint tutorial

    @lvermue
    lvermue committed Dec 5, 2020
    Configuration menu
    Copy the full SHA
    b5fb84f View commit details
    Browse the repository at this point in the history

  4. Trigger CI

    @PyKEEN-bot
    PyKEEN-bot committed Dec 5, 2020
    Configuration menu
    Copy the full SHA
    92ed682 View commit details
    Browse the repository at this point in the history

  5. Fix temp dir name handling

    @lvermue
    lvermue committed Dec 5, 2020
    Configuration menu
    Copy the full SHA
    33f0776 View commit details
    Browse the repository at this point in the history

  6. Trigger CI

    @PyKEEN-bot
    PyKEEN-bot committed Dec 5, 2020
    Configuration menu
    Copy the full SHA
    cc36bde View commit details
    Browse the repository at this point in the history

Commits on Dec 7, 2020

  1. Small fixes in docs 

    Trigger CI
    @cthoyt
    cthoyt committed Dec 7, 2020
    Configuration menu
    Copy the full SHA
    ed6b98e View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into allow_training_checkpoints

    @lvermue
    lvermue committed Dec 7, 2020
    Configuration menu
    Copy the full SHA
    6231d23 View commit details
    Browse the repository at this point in the history

  3. Trigger CI

    @PyKEEN-bot
    PyKEEN-bot committed Dec 7, 2020
    Configuration menu
    Copy the full SHA
    605b76e View commit details
    Browse the repository at this point in the history