Fix Loader checkpointing bug

[szkarpinski]

Category:

Bug fix (non-breaking change which fixes an issue)

Description:

Context

Loader checkpointing works the following way:

• Every full epoch, a new snapshot of Loader state is taken and kept (as current_snapshot_)
• Every iteration, age counter of the current snapshot is incremented.

When restoring from checkpoint, Loader first restores its state from the snapshot provided, and then fast-forwards age iterations.

Bug & fix

On restore, during fast-forwarding phase, the age counter is increased by ReadOne, resulting in incorrect value of age being kept in current_snapshot_. In this PR I fix the restore function to set age to a correct value.

Impact

For the first epoch after restoring from checkpoint, Loader's current_snapshot_ will have invalid age. As a result, checkpoints of the Loader saved during this first epoch will be incorrect. After the first epoch, age will be reset and the checkpoints will be correct again.

Additional information:

Affected modules and functionalities:

Loader checkpointing.

Key points relevant for the review:

Tests:

• Existing tests apply
• New tests added
  • Python tests
  • GTests
  • Benchmark
  • Other
• N/A

Checklist

Documentation

• Existing documentation applies
• Documentation updated
  • Docstring
  • Doxygen
  • RST
  • Jupyter
  • Other
• N/A

DALI team only

Requirements

• Implements new requirements
• Affects existing requirements
• N/A

REQ IDs: N/A

JIRA TASK: N/A

[szkarpinski]

!build

[dali-automaton]

CI MESSAGE: [13636778]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [13636778]: BUILD PASSED

[szkarpinski]

!build

[dali-automaton]

CI MESSAGE: [13932924]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [13932924]: BUILD FAILED

[dali-automaton]

CI MESSAGE: [13932924]: BUILD PASSED