Support checkpointing in Caffe reader

[szkarpinski]

Category:

New feature (non-breaking change which adds functionality)

Description:

This PR adds checkpointing support to fn.readers.caffe and fn.readers.caffe2.

Additional information:

Adding checkpointing support is a simple task, similar for every loader and reader.

The following changes were required to enable checkpointing in LMDB loader:

• Make it inherit from Loader<..., supports_checkpointing=true>
• Implement Skip() method. Skip should behave like ReadSample in terms of side effects, but should skip a sample instead of reading it. This method is used to implement fast-forwarding in Loader baseclass.
• Change Reset to use virtual_shard_id_ (which is the shard currently processed)
  instead of shard_id_ (which is the initial shard requested by the user). See [2] for more details.

The following changes were required to enable checkpointing in caffe and caffe2 readers:

• Make it inherit from DataReader<..., supports_checkpointing=true>. See [1] for more details.
• Store the very first snapshot in the constructor with this->SetInitialSnapshot().
  Subsequent snapshots are saved by DataReader baseclass.

[1] Changing DataReader template parameters

Inheriting from DataReader<..., supports_checkpointing=true> might look strange in the diff, because the DataReader is defined as:

template <typename Backend, typename LoadTarget,
          typename ParseTarget = LoadTarget, bool supports_checkpointing = false>

and is used mostly as:

DataReader<Backend, Target>

so to enable checkpointing one needs to add two parameters:

DataReader<Backend, Target, Target, true>

[2] virtual_shard_id_

virtual_shard_id_ and shard_id_ might differ when stick_to_shard=False.

This change doesn't impact existing code, because Reset is normally called after each full pass over the data, so then those two are equal. It might happen that a checkpoint is saved when the reader was is processing a different shard that shard_id_, so to restore from such checkpoint we need to be able to reset to the current shard (virtual_shard_id_).

The same change was made in FileReader in #4954.

Affected modules and functionalities:

• Caffe and caffe2 readers
• LMDB loader

Key points relevant for the review:

Tests:

• Existing tests apply
• New tests added
  • Python tests
  • GTests
  • Benchmark
  • Other
• N/A

Checklist

Documentation

• Existing documentation applies
• Documentation updated
  • Docstring
  • Doxygen
  • RST
  • Jupyter
  • Other
• N/A

DALI team only

Requirements

• Implements new requirements
• Affects existing requirements
• N/A

REQ IDs: N/A

JIRA TASK: DALI-3693

[szkarpinski]

!build

[dali-automaton]

CI MESSAGE: [10995316]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [10995316]: BUILD PASSED