Misleading warning when exiting iteration while loop

[kilianbreathnach]

I get the following warning in each epoch

Data iterator can not provide data anymore but required total number of iterations to run is not reached.

Looking into it, I see that the exception handler on #L416 for StopIteration, upon reaching the end of the data loader, sets should_exit = True. Then, on the next attempt to iterate, it goes through the except block again and it checks if the entire run is done by running self._is_done. This function checks if all iterations have been completed, not per epoch but for all epochs. Therefore, unless we are in the final epoch, this warning is shown for each epoch.

Is this the intended behaviour or have I done something wrong? I am seeing this behaviour while using an iterator that must be manually reset to the first element. I do this reset in a handler that runs on EPOCH_COMPLETED so after this loop has been exited, making the warning seemingly unavoidable through the API.

[vfdev-5]

@kilianbreathnach could you please share some details on your data iterator and how are you using it for the training ?

Assumptions made in Engine when working with iterators is that they provide data until StopIteration and this should ("normally"?) correspond epoch_length * max_epochs iterations. Once iterator stops and raises StopIteration as we can not restart it, the training should be ended.

How do you use your iterator by a single epoch and restart it for the next epoch ?

[kilianbreathnach]

I am using the DALIGenericIterator from the NVIDIA/DALI package.

Shouldn't StopIteration occur at the end of each epoch? It happens inside the _run_once_on_dataset method, which is called inside a loop over epochs in the _internal_run method:

while self.state.epoch < self.state.max_epochs and not self.should_terminate:
    self.state.epoch += 1
    self._fire_event(Events.EPOCH_STARTED)

    if self._dataloader_iter is None:
        self._setup_engine()

    hours, mins, secs = self._run_once_on_dataset()

    ...

Checking for epoch_length * max_epochs means it doesn't expect the iterator to finish until the end of the entire experiment, like not until say 5000 epochs of X iterations, for X = (dataset size) / (batch size).

[vfdev-5]

Thanks for the details. Do you use auto_reset=True or call data_iterator.reset() ?

I think in your case we need to specify auto_reset=True or call by your self with (need to check this)

@trainer.on(Events.EPOCH_COMPLETED)
def reset_iterator(_):
    data_iterator.reset()

I assume that in this case, the iterator wont stop providing data after a single epoch and can continue the number of epochs.

PS: We can later provide an example of using Nvidia/DALI.

[kilianbreathnach]

Yes, I ,manually call .reset() using a function defined to run on EPOCH_COMPLETED, as I mentioned in the original issue. I see now that if I were to use the auto_reset option, then the iterator would reset on #L436 and the loop would continue. However, the manual reset will not happen until the _run_once_on_dataset method has completed and the EPOCH_COMPLETED event is fired in the _internal_run method.

It turns out that my issue is that the iter_counter condition on #L457 is not being met. In the DaliGenericIterator, there are some options for how to fill or not fill the last batch, outlined here. The default option will lead to batches being processed at a rate that is actually quicker than the epoch_length. It seems the best option is probably to set the iterator with

fill_last_batch = False, last_batch_padded = True

but maybe you will figure out better options when you make a DALI example.

[vfdev-5]

Ok I see the problem. Thanks for the feedback!