Add minor performance improvements to resnet input pipeline #4247
Conversation
| dataset = dataset.shuffle(buffer_size=_NUM_TRAIN_FILES) | ||
|
|
||
| # Convert to individual records | ||
| # TODO(guptapriya): Should we make this cycle_length a flag similar to |
There was a problem hiding this comment.
Did you get a chance to check how sensitive performance is to this on both something big (DGX, V100 GCE) and something small (1x K80/P100)? I prefer not to have a performance flag unless it makes a big difference. And if it is a constant it would be nice to have a brief comment so it isn't just a magic number.
There was a problem hiding this comment.
No I haven't had the chance to run on things other than DGX-1V. I don't think the performance difference will show up on K80s because the input pipeline will not be the bottleneck. But I haven't tested it. I am talking to the input team to figure out if a constant here makes sense, or should this be tuned (in which case we may need to just remove it)
| batch_size=batch_size, | ||
| num_parallel_batches=1)) | ||
| num_parallel_batches=1, | ||
| drop_remainder=True)) |
There was a problem hiding this comment.
This could matter for cifar with only 60k images.
There was a problem hiding this comment.
It should not matter much as long as the batch size is reasonable, since this only drops the part of the dataset that doesn't fit into a full batch.
There was a problem hiding this comment.
Right, it only drops the last partial batch. so for 60k images for a batch size of 2048 also it should drop only 608 images or so.
Add @lamberta to CODEOWNERS/samples
Add lamberta to CODEOWNERS/samples
Fixed Bug regarding tfrecord shuffling in object_detection
…rleve in imagenet dataset.
… resnet-minor-perf
|
So there's good news and bad news. 👍 The good news is that everyone that needs to sign a CLA (the pull request submitter and all commit authors) have done so. Everything is all good there. 😕 The bad news is that it appears that one or more commits were authored or co-authored by someone other than the pull request submitter. We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that here in the pull request. Note to project maintainer: This is a terminal state, meaning the |
|
The object detection change in this PR has already been approved and merged in other PR so object detection reviews are not necessary. I'm removing derekjchow, jch1 and me from reviewers. |
|
@pkulzc - yes! I have messed up this PR while doing rebase etc.. Hence it pulled in some already merged changes :( Sorry about the spam. I will fix it, and yes, no action is needed from you guys! |
7 participants
Added the following changes to improve input pipeline:
The improvements were observed when testing with 8 V100 GPUs on DGX-1 using mirrored strategy (fp16, resnet v2). The differences may not be noticeable in other cases where input pipeline is not the bottleneck.