Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dm-31054 Fix issue running Quantum when inputs are missing #131

Merged
merged 1 commit into from Jul 21, 2021

Conversation

natelust
Copy link
Contributor

The recent NoWorkFound changes were intended to let a Quantum
continue to run if some of its inputs were missing as long as
enough inputs were present. However, there was a bug related
to wrong dataset types and use of defaultdict that caused the
system to always think NO inputs were present if any were
missing. This fixes that.

For those interested, adjustQuantum on a connections class indexes
things by dataset type name, a string. Most other paces use
DatasetType object in mappings. This is not a problem with
NameDict containers because they know how to translate. However in
this case a defaultdict was used to construct the mapping, and the
keys were all DatasetTypes.

Then this was passed through to update the Quantum and adjust it.
Because the container type was still defaultdict, when the updater
indexed with dataset type names (str) it created a new entry in the
defaultdict that was empty, instead of finding the entry associated
with the DatasetType that had the corresponding name.

This only triggered if there was at least one missing input, so in
normal operations everything worked as expected. A subsequent
container was then passed into adjustQuantum that had no datasetRefs
(because the defaultdict created a new empty entry) and the method
rightly said 0 is less than 1 required ref and raised a NoWorkFound.

Solution was to change to the proper NamedDict object before calling
into the update code in pipe_base.

Checklist

  • ran Jenkins
  • added a release note for user-visible changes to doc/changes

The recent NoWorkFound changes were intended to let a Quantum
continue to run if some of its inputs were missing as long as
enough inputs were present. However, there was a bug related
to wrong dataset types and use of defaultdict that caused the
system to always think NO inputs were present if any were
missing. This fixes that.

For those interested, adjustQuantum on a connections class indexes
things by dataset type name, a string. Most other paces use
DatasetType object in mappings. This is not a problem with
NameDict containers because they know how to translate. However in
this case a defaultdict was used to construct the mapping, and the
keys were all DatasetTypes.

Then this was passed through to update the Quantum and adjust it.
Because the container type was still defaultdict, when the updater
indexed with dataset type names (str) it created a new entry in the
defaultdict that was empty, instead of finding the entry associated
with the DatasetType that had the corresponding name.

This only triggered if there was at least one missing input, so in
normal operations everything worked as expected. A subsequent
container was then passed into adjustQuantum that had no datasetRefs
(because the defaultdict created a new empty entry) and the method
rightly said 0 is less than 1 required ref and raised a NoWorkFound.

Solution was to change to the proper NamedDict object before calling
into the update code in pipe_base.
Copy link

@yalsayyad yalsayyad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think I would have figured this out. Thanks.

@natelust natelust merged commit 33f136f into master Jul 21, 2021
@natelust natelust deleted the tickets/DM-31054 branch July 21, 2021 18:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants