MnistDatasetFetcher shuffles on reset causing unexpected behavior

[GuutBoy]

If the shuffle parameter is set on the MnistDatasetFetcher it will shuffle the dataset on each call to reset. Below is the relevant code from MnistDatasetFetcher:

@Override
public void reset() {
  cursor = 0;
  curr = null;
  if (shuffle)
    MathUtils.shuffleArray(order, rng);
}

This appears to be a bug. In particular, this seem to cause an MnistDatasetIterator constructed with numExamples < 60000 to iterate over a new dataset on each call to reset(). The documentation is not super explicit but this does not seem to be the intended behavior.

An example of this is demonstrated by the code below:

import java.io.IOException;
import org.deeplearning4j.datasets.iterator.impl.MnistDataSetIterator;
import org.nd4j.linalg.dataset.DataSet;

public class MnistDatasetExperiments {

  public static void main(String[] args) throws IOException {
    MnistDataSetIterator it = new MnistDataSetIterator(1, 1, false, false, true, 0);
    DataSet data1 = it.next();
    it.reset();
    DataSet data2 = it.next();
    if (data1.get(0).getFeatures().equals(data2.get(0).getFeatures())) {
      System.out.println("Success 😃");
    } else {
      System.out.println("Failure 😭");
    }
  }
}

Here we construct an iterator over a single example with a batch size of 1. I would expect this iterator to return the same dataset on the first and second call to next (after the call to reset).

[Charele]

Why must we get the same dataset?
I think it isn't necessary.

[AlexDBlack]

Thanks for reporting.
Fixed here: #6302

[GuutBoy]

@Charele It is a design choice and of course you could go with the current behavior. Either way, the actual behavior should probably be clearly documented to avoid confusion.

In my case I wanted to train a classifier over a number of epochs on a limited set of examples. To do this I used a MnistDatasetIterator with numExamples set to, say 500, and call fit on a MultiLayerNetwork with the iterator as the argument. Now I would expect this to just train the classifier on a data set of size 500 but, because of this issue, the classifier is trained on a new random subset for each epoch. This was very confusing to me because I would end up with a very good classifier even when training on a very small data set.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.