Move the loader snapshot queue to the reader

[stiepan]

Category:

Refactoring (Redesign of existing code that doesn't affect functionality)

Description:

This PR moves the snapshot queue from the loader to the reader.
Reader already is resposible for prefetching batches, so it makes sense for it to make sure to keep loader's state snapshot after each produced batch. This should simplify loaders code and remove the need for extra mutex to access loaders snapshot queue. The snapshot queue is now advanced alongside the batch prefetch queue, which makes accessing snapshots (reader's by the executor and loader's by the reader) to have no side effects.

The snapshots are put in the queue by the prefetching thread. If the loader cannot be checkpointed at the given moment, the reader simply puts None in the queue and the relevant error is raised on the attempt to access the snapshot. This could be extended to the executor as well (so that it does not have to compute epoch sizes in advance).

In the future, the snapshot queue could be extracted from the reader and provided to it as a dependency similar to loader. This way, we could provide different checkpointing mechanisms to different readers and loaders.

Additional information:

Affected modules and functionalities:

Key points relevant for the review:

Tests:

• Existing tests apply
• New tests added
  • Python tests
  • GTests
  • Benchmark
  • Other
• N/A

Checklist

Documentation

• Existing documentation applies
• Documentation updated
  • Docstring
  • Doxygen
  • RST
  • Jupyter
  • Other
• N/A

DALI team only

Requirements

• Implements new requirements
• Affects existing requirements
• N/A

REQ IDs: N/A

JIRA TASK: DALI-3653

[stiepan]

!build

[dali-automaton]

CI MESSAGE: [10161684]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [10161684]: BUILD PASSED