Rework FW Plugins to prefetch only as many batches as needed

[Kh4L]

Signed-off-by: Serge Panev spanev@nvidia.com
Co-author @klecki

[JanuszL]

I don't know if moving this here gives any value as we are calling self.next() anyway few lanes bellow.

[Kh4L]

next is now doing:

• ShareOutputs
• copy outputs to FW
• ReleaseOutputs
• Run

The very first next will be missing a Run to get the outputs from. It has to be done once in the ctor.

[JanuszL]

I was asking if moving prefetching from next to ctor improves anything. I was not asking about getting rid of it.

[klecki]

We certainly needed to remove call to _prefetch as it was scheduling queue_size runs of the pipeline for every next, and we consumed only one.

In next we consume the output, so we need to schedule a new _run after that. We also need to start full run for the first iteration (with prefetching that will fill the queues). As we need to call this first run that causes the prefetching once, we do it in constructor.

[jantonguirao]

I'd like to see new L0 python tests:

1. covering all the iterators going around some decent amount of data
2. training one epoch on a small dataset for every framework
   At least (1) to make sure that this fix works

[Kh4L]

I'd like to see new L0 python tests:

1. covering all the iterators going around some decent amount of data

2. training one epoch on a small dataset for every framework
   At least (1) to make sure that this fix works

I totally agree here
I am planning to add them in a next PR, but running and training on 3 epochs on a small dataset, to see how the iterator handles epoch incrementing

[Kh4L]

Build 687809

[jantonguirao]

I'll submit the iterator tests in a separate PR