DALI Pipeline inputs as named arguments to Pipeline.run()

[szalpal]

Category:

New feature (non-breaking change which adds functionality)

Description:

Currently, providing inputs to DALI Pipeline is highly cumbersome. We have a couple of ways:

1. Using Readers,
2. Using external_source operator and its argument source,
3. Using an input operator (e.g. external_source) and feed_input function.

All of these are quite problematic. This PR attempts to tackle this with allowing yet another way of providing inputs.

My proposal is to augment pipe.run() function with **kwargs in Python. Every such argument, provided that it has a corresponding input operator in DALI pipeline, can accept any data assigned to the name of such input operator. For example:

@pipeline_def
def pipe():
    return fn.external_source(name="my_inp")

p = pipe(...)
p.build()
p.run(my_inp=np.random(...))

The change is added run .run() only, since I figured it's always possible to add it to the schedule_run too, but removing it will be problematic due to backwards compatibility issues.

Additional information:

Affected modules and functionalities:

Key points relevant for the review:

Tests:

• Existing tests apply
• New tests added
  • Python tests
  • GTests
  • Benchmark
  • Other
• N/A

Checklist

Documentation

• Existing documentation applies
• Documentation updated
  • Docstring
  • Doxygen
  • RST
  • Jupyter
  • Other
• N/A

DALI team only

Requirements

• Implements new requirements
• Affects existing requirements
• N/A

REQ IDs: N/A

JIRA TASK: N/A

[szalpal]

!build

[dali-automaton]

CI MESSAGE: [7581426]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [7581426]: BUILD FAILED

[szalpal]

!build

[dali-automaton]

CI MESSAGE: [7583127]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [7583127]: BUILD FAILED

[szalpal]

!build

[dali-automaton]

CI MESSAGE: [7593105]: BUILD STARTED

[mzient]

That's bad.... It would most likely cause very weird behavior of the pipeline and explaining it to the user would be extremely hard. Instead, rather enforce that queue depth is 1 when using pipeline inputs.

[szalpal]

Unfortunately, with how currently the running logic is organised, it has to be there. if self._first_iter and self._exec_pipelined: means, that when we legitimately set exec_pipelined=True and prefetch_queue_depth=1, the prefetch function will be executed. And we want to have pipeline_inputs in this scenario.

I've added the enforce and extensive explanation in the run function. Hopefully this will solve the issue.

[mzient]

(to put our call in writing):
We don't have to propagate it down to _prefetch and _run_once - we could just call feed_input in schedule_run or even restrict the new API to run, so we avoid yet another way to feed the (prefetching) pipeline.

[mzient]

Note: I wonder what's the value of prefetch_queue_depth in case of separated queues - and, specifically, what prefetch_queue_depth > 1 would evaluate to. At least I'd try to specify separated queues and see the error below pops up.

[dali-automaton]

CI MESSAGE: [7593105]: BUILD FAILED

[szalpal]

!build

[dali-automaton]

CI MESSAGE: [7604393]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [7604393]: BUILD FAILED

[szalpal]

!build

[dali-automaton]

CI MESSAGE: [7606050]: BUILD STARTED

[szalpal]

!build

[dali-automaton]

CI MESSAGE: [7606101]: BUILD STARTED

[szalpal]

!build

[dali-automaton]

CI MESSAGE: [7606282]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [7606282]: BUILD FAILED

[szalpal]

!build

[dali-automaton]

CI MESSAGE: [7607431]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [7607431]: BUILD FAILED

[dali-automaton]

CI MESSAGE: [7606050]: BUILD FAILED

[dali-automaton]

CI MESSAGE: [7606101]: BUILD FAILED

[dali-automaton]

CI MESSAGE: [7606101]: BUILD PASSED