[WIP] Indexed recordIO support in InputSplit

[ptrendx]

Not complete yet, don't merge

[piiswrong]

@tqchen

[piiswrong]

not sure defaulting seed to 0 is a good idea.

[ptrendx]

It is added to magic number inside indexedrecordioinputsplit. It is actually the second as in image iterator in MxNet (seed defaults to 0 + magic).

[piiswrong]

batch_size shouldn't have a default value

[ptrendx]

This is only a hint (like the HintChunkSize) and only IndexedRecordioInputSplit actually uses that. Readers can give back any number of samples anyway so consumers of the read data need to be prepared to handle that.

[piiswrong]

?

[ptrendx]

Even though recordio is aligned to 4 bytes, reader was reading to chunk's data vector which was size_t (so 8 bytes aligned). This is problematic, because there is no guarantee that recordio entry is aligned to 8 bytes. That's why I changed chunk's internal data vector to uint32_t

[tqchen]

can you comment a bit about index file format and mechanism, e.g. does every shard need to load the entire index file, etc,

[ptrendx]

It is very simple file format (line per entry with 2 integers - entry id and offset) introduced in apache/mxnet#2887 a year ago, but not actually used anywhere (although im2rec.py script actually generates the index file alongside recordIO file).
Each worker needs the whole index file, then each worker is responsible for only its own portion of the indexes.
Shuffle introduces global shuffling, but only inside a worker part (so that there is no need to share seeds between workers etc.). I thought about making it so that groups of few entries are shuffled in case the performance is bad, but at least on systems I tested it on it was fine as-is. This should really improve the shuffle option in MXNet recordio iterator, because right now it shuffles only inside a single chunk, which often is less than or comparable to single batch (so there is effectively no shuffling).
Motivation for this PR is twofold:

• I introduced new much faster IO pipeline in MXNet ([WIP] New faster version of the RecordIO iterator apache/mxnet#7152) which unfortunately makes it hard to implement shuffling inside iterator, so shuffling needs to be done on lower level.
• having InputSplit return precisely the batch size of images makes the IO pipeline even faster, because the overflow storage is not used, so there are no unnecessary copies being made, and also the cost of OpenMP synchronization between threads is paid only once per batch (In my tests on DGX-1 I went from 6.2k imgs/s to 7k imgs/s when testing just IO on RN50 pipeline).

[piiswrong]

may need to add this to amalgamation