📉☎️ Validation Loss Training Callback #1169

mberr · 2022-11-19T13:51:29Z

This extracts parts of the training loop related to calculating the epoch loss into a function to re-use it for calculating validation losses.

Example:

from pykeen.datasets import get_dataset
from pykeen.pipeline import pipeline

dataset = get_dataset(dataset="nations")
pipeline(
    dataset=dataset,
    model="mure",
    training_kwargs=dict(
        callbacks="validation-loss",
        callback_kwargs=dict(triples_factory=dataset.validation),
    ),
    result_tracker="console",
)

Also move optimizer & scheduler functionality into a callback

src/pykeen/training/training_loop.py

mberr · 2022-11-20T11:39:15Z

src/pykeen/training/callbacks.py

+        # TODO: where to get these from?
+        label_smoothing = 0.0
+        training_data_loader_kwargs = dict(sampler=None)


these are not attributes of the training loop but only present as variables in the _train method

cf. 814d5c2

mberr · 2022-11-20T11:40:52Z

src/pykeen/training/callbacks.py

+            # TODO: this should be num_instances rather than num_triples; also for cpu, we may want to reduce this
+            batch_size=self.triples_factory.num_triples,


This may cause OOM kills on cpu for large datasets. It would be better to derive something from the training batch size (e.g., 4*batch_size or similar) as upper bound.

Performance-wise, a too large initial value will only effect runtime of the first call, since for later calls the previous values will be re-used.

cf. 7d95654

mberr · 2022-11-20T11:41:18Z

tests/test_training/test_callbacks.py

@@ -38,3 +38,15 @@ def test_batch_size(self):
                ),
            )
            assert {c.kwargs.get("batch_size", None) for c in mock_evaluate.call_args_list} != {None}
+
+
+# TODO: more tests


most callbacks seem to lack tests 😕

src/pykeen/training/callbacks.py

cthoyt · 2023-09-23T14:50:47Z

src/pykeen/training/callbacks.py

+
+
+class ValidationLossTrainingCallback(TrainingCallback):
+    """Calculate loss on a development set."""


can we get an end-to-end example usage in the docstring please

cthoyt · 2023-09-23T15:38:01Z

src/pykeen/training/callbacks.py

+
+class ValidationLossTrainingCallback(TrainingCallback):
+    """
+    Calculate loss on a development set.


validation set?

renamed to "evaluation"; I want to highlight that we do not need to have a single validation set, but could also use multiple callbacks with different evaluation sets (and potentially also differet frequencies, e.g., to have a small validation each step, and a bigger one every n-th step)

cthoyt · 2023-09-23T15:38:16Z

src/pykeen/training/callbacks.py

+                callback_kwargs=dict(triples_factory=dataset.validation),
+            ),
+            result_tracker="console",
+        )


can you please explain how to get the validation losses after the fact as a list

I guess someone might 1 use this in combination with a result tracker or 2 want to make their own charts or something

how about adding that to notebooks? (using the # %% style for better VCS): 49ae07e

This is great, but I think it's ideal to keep the code examples in with the code itself so when people are looking through the docs they see it. It's also okay if we double it

Hm, the notebook is now 45 lines (tbf with formatting), and it is really messy to get the code well-formatted into rst...

The short version without the post-processing / plotting logic is already inside the docstring

cthoyt · 2023-09-23T15:41:40Z

src/pykeen/training/callbacks.py

@@ -490,6 +491,8 @@ class ValidationLossTrainingCallback(TrainingCallback):
    def __init__(
        self,
        triples_factory: CoreTriplesFactory,
+        callbacks: TrainingCallbackHint = None,
+        callback_kwargs: TrainingCallbackKwargsHint = None,


might as well fix the name of this while we're here

or not, if it's gonna be a big diff

mberr and others added 2 commits November 19, 2022 14:50

extract epoch loop

7f795f8

Also move optimizer & scheduler functionality into a callback

Merge branch 'master' into validation-loss

56a0121

cthoyt reviewed Nov 19, 2022

View reviewed changes

src/pykeen/training/training_loop.py Show resolved Hide resolved

cthoyt reviewed Nov 19, 2022

View reviewed changes

src/pykeen/training/training_loop.py Show resolved Hide resolved

mberr and others added 12 commits November 19, 2022 18:05

Merge branch 'master' into validation-loss

325dd9b

Merge branch 'master' into validation-loss

076dd7a

improve type annotation

4c2886f

add first draft of validation loss cb

45d86ea

add AMO

dd3efbd

fix annotation

516c34b

add option to disable backward pass

1a46367

fix validation callback

f49b211

add test for validation callback

54ddfb4

reformat

916ec3c

fix code order to fix resolver

a18e5ab

add todo

6a0e540

mberr commented Nov 20, 2022

View reviewed changes

mberr added 5 commits November 20, 2022 17:47

Merge branch 'master' into validation-loss

e8eb7fe

Merge branch 'master' into validation-loss

c1a475c

Merge branch 'master' into validation-loss

3123fc1

Merge branch 'master' into validation-loss

673fb14

update validation loss callback

0690b7c

mberr changed the title ~~Validation Loss~~ 📉☎️ Validation Loss Training Callback Sep 22, 2023

cthoyt reviewed Sep 22, 2023

View reviewed changes

src/pykeen/training/callbacks.py Outdated Show resolved Hide resolved

cthoyt reviewed Sep 22, 2023

View reviewed changes

src/pykeen/training/callbacks.py Outdated Show resolved Hide resolved

mberr added 3 commits September 22, 2023 20:31

move pre_step callbacks to optimizer

28e9805

Merge branch 'master' into validation-loss

cdfca42

fix import and add comment

18d2885

cthoyt approved these changes Sep 23, 2023

View reviewed changes

cthoyt enabled auto-merge (squash) September 23, 2023 00:19

Update callbacks.py

34054a8

mberr disabled auto-merge September 23, 2023 07:52

Merge branch 'master' into validation-loss

673a8fc

cthoyt reviewed Sep 23, 2023

View reviewed changes

mberr added 2 commits September 23, 2023 17:10

include e2e example

2e35f1e

rename parameter and fix device type check

5317c07

cthoyt reviewed Sep 23, 2023

View reviewed changes

enable callbacks for validation loss callback

aa872c4

cthoyt reviewed Sep 23, 2023

View reviewed changes

mberr added 8 commits September 23, 2023 17:48

directly use only_size_probing

b25e032

rename callback

3376073

update PythonResultTracker to allow multiple calls for the same epoch

2b746ca

add example for using the tracked validation loss

49ae07e

fix import order

96118aa

rename parameter

583cf16

Merge branch 'master' into validation-loss

9a36900

Merge branch 'master' into validation-loss

45aceb4

mberr enabled auto-merge (squash) September 23, 2023 21:02

mberr added 2 commits September 24, 2023 00:28

fix test

7bf71c1

maybe fix type annotation *and* sphinx

d9ea976

cthoyt disabled auto-merge September 24, 2023 07:02

cthoyt enabled auto-merge (squash) September 24, 2023 07:02

Update first_steps.rst

b7ef61e

cthoyt disabled auto-merge September 24, 2023 07:29

cthoyt enabled auto-merge (squash) September 24, 2023 07:29

cthoyt merged commit e184f97 into master Sep 24, 2023
11 checks passed

cthoyt deleted the validation-loss branch September 24, 2023 07:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

📉☎️ Validation Loss Training Callback #1169

📉☎️ Validation Loss Training Callback #1169

mberr commented Nov 19, 2022 •

edited

mberr Nov 20, 2022

mberr Sep 22, 2023

mberr Nov 20, 2022

mberr Sep 22, 2023

mberr Nov 20, 2022

cthoyt Sep 23, 2023

mberr Sep 23, 2023

cthoyt Sep 23, 2023

mberr Sep 23, 2023

cthoyt Sep 23, 2023

cthoyt Sep 23, 2023

mberr Sep 23, 2023

mberr Sep 23, 2023

cthoyt Sep 23, 2023

mberr Sep 23, 2023

mberr Sep 23, 2023

cthoyt Sep 23, 2023

cthoyt Sep 23, 2023

mberr Sep 23, 2023

		# TODO: this should be num_instances rather than num_triples; also for cpu, we may want to reduce this
		batch_size=self.triples_factory.num_triples,



		class ValidationLossTrainingCallback(TrainingCallback):
		"""Calculate loss on a development set."""

📉☎️ Validation Loss Training Callback #1169

📉☎️ Validation Loss Training Callback #1169

Conversation

mberr commented Nov 19, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mberr commented Nov 19, 2022 •

edited