Is your feature request related to a problem? Please describe.
When using a set of extractors, if the DAG module being built does not reference the extractor targets, an exception is thrown during building: ValueError: Could not find node with name: input3 in function graph. Available nodes: ['input1', 'input2', Ellipsis].
Describe the solution you'd like
When using from_ to create extractors, support an additional flag which allows builds to skip an extractor if the target node does not exist.
Describe alternatives you've considered
- Additional introspection on the dag to check what targets are used and explicitly avoid registering those unused extractors. This adds additional inspection steps outside of the DAG building paths, and does not work if the target feeds a decorator.
Additional context
Originally discussed in Hamilton dev slack here.
Is your feature request related to a problem? Please describe.
When using a set of extractors, if the DAG module being built does not reference the extractor targets, an exception is thrown during building:
ValueError: Could not find node with name: input3 in function graph. Available nodes: ['input1', 'input2', Ellipsis].Describe the solution you'd like
When using
from_to create extractors, support an additional flag which allows builds to skip an extractor if the target node does not exist.Describe alternatives you've considered
Additional context
Originally discussed in Hamilton dev slack here.