Why need the data processing step?

[JonnyKong]

Hi Matt,

Thanks for the elegant code in Tensorflow. But why is the data processing step necessary?

It seems to me that it's possible to load the dataset into memory before training (at least for the PacMan dataset), and then randomly select 32*32 patches at runtime. Will that make I/O faster?

Thanks in advance

[serkansulun]

Here is my hypothesis:
Once the dataset is expressive enough for the problem, it is more efficient to train the network over that dataset multiple times (i.e. epochs). In this case, preprocessing the data multiple times is unnecessary.

[dyelax]

Sorry for the delayed response! The initial reason I added a separate pre-processing pipeline was because I was being bottlenecked by i/o and it is much more efficient to read in 32x32 image patches than the full-sized images. I don't remember what the memory specs of my machine were, but I think I was running into issues loading the whole dataset into memory. If you are able to do that, it might make more sense to create the patches at runtime.

edit
One problem with creating the patches at runtime on the Ms. Pac-Man data is that most patches will have no movement in them. To solve this, I randomly sample patches until it finds one with movement. This means that it will take longer to generate some patches than others. It's probably more efficient to get all of this generation cost out of the way once during pre-processing than having to deal with it every epoch during training.

[JonnyKong]

Thanks for your reply? It's very helpful.