Improve error reporting on partially unused output of external source

[stiepan]

Improve error reporting on partially unused output of external source

Signed-off-by: Kamil Tokarski ktokarski@nvidia.com

Category:

Description:

If one creates external source (and provides the source to the operator) with multiple outputs, but does not use some of the outputs, it results in vague error on feed_input that tries to feed input to non-existing operator. Catch that early (in pipeline.build()) and provide clearer error.

Find instances of partially unused external source groups during the Python pipeline build and explicitly disable the unused instances in the relevant ExternalSourceGroup, so that the group skips feeding the pruned external source nodes during the run. Emit relevant warning.

Additional information:

Affected modules and functionalities:

Python pipeline wrapper.

Nose utility module: added assert_warns mirroring asser_raises, extracted common code.

Key points relevant for the review:

Given the number of ways to define functioning pipeline, please think if the check does not interrupt some valid use case of external source that would work otherwise (I don't think it can, but it would be good to double check it with someone else).

Tests:

• Existing tests apply
• New tests added
  • Python tests
  • GTests
  • Benchmark
  • Other
• N/A

Checklist

Documentation

• Existing documentation applies
• Documentation updated
  • Docstring
  • Doxygen
  • RST
  • Jupyter
  • Other
• N/A

DALI team only

Requirements

• Implements new requirements
• Affects existing requirements
• N/A

REQ IDs: N/A

JIRA TASK: DALI-2802

[JanuszL]

When there is more than one ES it will be hard to guess which one causes this but I guess we cannot distinguish ES instances anyhow.

[stiepan]

I hope for group.source_desc.source to give some clue (depending how top-level object it is it may be of some value).

[klecki]

Just for the sake of documenting the discussion, the alternative to erroring out on this would be to automatically mark all the outputs of the externals source as sinks. Or at least those that are not returned, potentially issuing a warning?

Edit:
As discussed, we can extract the graph backtracking, run it for outputs and _sinks, report the warnings for the missing external sources as we do the error now, and resume the backtracking for unused external sources.

Any use that would previously error out in this case would be also missing the node in C++ backend, so I don't think there would be any problematic use cases.

[jantonguirao]

Suggested change

	f"The external source '{source_str}' produces {num_outputs} outputs, but "
	f"The external source node '{source_str}' produces {num_outputs} outputs, but "

[stiepan]

done

[stiepan]

Okay, so to sum up the discussion: we do not want to prevent the pipeline from removing unused external sources from graph, but rather want to prevent python code from feeding them,

[stiepan]

!build

[dali-automaton]

CI MESSAGE: [5130776]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [5130776]: BUILD FAILED

[stiepan]

!build

[dali-automaton]

CI MESSAGE: [5131731]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [5131731]: BUILD PASSED