Related to #551, #763 ?

I was able to get 68x68 images reading/decoding from TF-examples fast enough to saturate my K40 with input pipeline using 6 threads/1 CPU. These tutorials have not been tuned to work efficiently on GPU, so there could be some small ops that are placed on GPU suboptimally and are causing unnecessary data transfers -- try pinning your input pipeline to CPU manually. See #838 for an example of suboptimal placement making GPU version run 100x slower

Re: "Pinning the pipeline to the CPU":

here's how I would optimize it -- pin everything to CPU (export
TF_MIN_GPU_MULTIPROCESSOR_COUNT=800), and remove all the non-reading ops.
Tweak your pipeline design/number of threads until you get maximum
throughput. Then re-enable GPU, and use manual pinning to make sure that
your input pipeline throughput is unchanged. Then attach your processing
ops (on GPU)

On Fri, Jan 22, 2016 at 2:59 PM, nryant notifications@github.com wrote:

Pinning the pipeline to the CPU helps somewhat, but still is worse than I
would expect. For num_threads=2 the time reduces to 12.6 seconds with 13%
GPU utilization and for num_threads=16 to 9.4 seconds with 18% utilization

—
Reply to this email directly or view it on GitHub
#837 (comment)
.

This actually fixes #838. Pinning the pipeline to CPU for
fully_connected_reader.py helps somewhat, but performance still lags
fully_connected_feed.py. I did some benchmarking this afternoon before
leaving the office and most of the remaining peformance gap seems to be
from the fact that reading an epoch's worth of images takes 20-25x as long
using a reader (.2 seconds vs about 5 seconds; sorry, out of office and
don't have the precise timings with me). For a more realistically sized
network, this additional overhead wouldn't be such an issue, so this
probably should be closed after fully_connected_reader.py has been
modified.

On Friday, January 22, 2016, Vijay Vasudevan notifications@github.com
wrote:

Closed #837 #837 via
ebe109b
ebe109b
.

—
Reply to this email directly or view it on GitHub
#837 (comment).

Neville Ryant
Linguistic Data Consortium
University of Pennsylvania

oops, off-by-1 error on my part.
When you are using a reader, there's more work done at the beginning because of prefetching, could it be that the extra time is due to it filling up a queue of examples?

@ebrevdo: Could you take a look since it's queue related?

@josh11b could this be due to lack of caching in the readers? not sure if this bug is still relevant given the changes that have been pushed between when this bug report and now.

I believe there is now the ability to read batches from a reader that can reduce overhead, assuming there is no problem with the examples having different dimensions. Also I recall someone is working on ParseExample performance improvements?

Yes; the ParseExample work will hopefully get checked in w/in a week or two.

On Wed, Aug 10, 2016 at 10:31 AM, josh11b notifications@github.com wrote:

I believe there is now the ability to read batches from a reader that can
reduce overhead, assuming there is no problem with the examples having
different dimensions. Also I recall someone is working on ParseExample
performance improvements?

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
#837 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABtim6qIclGyr6l59aA8O2h8x9g4HCFOks5qegrVgaJpZM4HKB-s
.

I'm assuming that this is checked in. Feel free to open a new github issue if the problem still persists in recent versions.