Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-28208: Add python access to readComponent() and associated Storable tests. #556

Merged
merged 2 commits into from Jan 14, 2021

Conversation

erykoff
Copy link
Contributor

@erykoff erykoff commented Jan 5, 2021

No description provided.

@erykoff erykoff changed the title Add test for persisting exposure storable blob. DM-28208: Add test for persisting exposure storable blob. Jan 5, 2021
@erykoff erykoff changed the title DM-28208: Add test for persisting exposure storable blob. DM-28208: Add python access to readComponent() and associated Storable tests. Jan 12, 2021
Copy link
Contributor

@jmeyers314 jmeyers314 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

First time I've seen a dataclass in the wild, so that was fun :)

One question though: how would one get a Blob from a butler? I suppose it's too much to ask that blob = butler.get("BLOB", dataId=dataId) would work?

@erykoff
Copy link
Contributor Author

erykoff commented Jan 12, 2021

For gen2, it's just not possible, and won't be. You have to do the read-an-exposure-with-a-single-pixel-bbox, and get it out of the getInfo(). For gen3, I'm working on it, and the readComponent() here should make that possible. I hope.

Copy link
Member

@kfindeisen kfindeisen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking care of this. Just a few questions.

Comment on lines +150 to +151
/// Read an arbitrary non-standard component by name.
std::shared_ptr<typehandling::Storable> readComponent(std::string const &componentName);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be non-standard, or any component? Consider providing full API docs with e.g. parameter descriptions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, this is by string name, and the standard ones are by enumerated component, so I think this has to be non-standard. I certainly couldn't figure out the appropriate names to input to get the "standard" ones, which is why I put that comment.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a shame because if readComponent could read any component it would simplify the ExposureFormatter code a lot.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems like a lot more work here, because of passing enumerations up the chain, I think. But the readComponent(component) could also be exposed in python, I suppose.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ExposureInfo exposes keys for looking up standard components, though because of my short-sightedness (i.e., assuming it was for internal use only) ArchiveReader::getComponent takes a string instead of a full Key. Assuming you don't change that, you'd need to call something like reader.readComponent(ExposureInfo::KEY_WCS.getId()) in C++ or reader.readComponent(ExposureInfo.KEY_WCS) in Python, but it certainly can be done.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not that it matters for your case, but having ExposureFitsReader::readComponent take a Key would make it more useful in C++; I'm happy to take that on as a separate ticket.

Copy link
Member

@TallJimbo TallJimbo Jan 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 for making this take enums and handle standard components in another ticket. IIRC, at least some of the standard components are also unusual components here, at least in terms of how the affect they metadata reading and hence interact with each other, and really the only safe way to handle them here would be to intercept them and delegate to the special method for that component. And "intercept all standard components" feels like a case where you really want to be using enums somehow to make sure the check is exhaustive (and remains exhaustive even if new standard components are added in other parts of the codebase).

Edit: I think my memory of how those enums worked and where they should be used is just faulty. Please ignore the above; I don't think it actually made sense, and I'm happy that the rest of you have it under control.

@@ -148,6 +153,69 @@ def testNewValue(self):
cppLib.assertCppValue(self.testbed, newstr)


@dataclass
class Blob(Storable):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this class add that DemoStorable, above, does not?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That isn't persistable, afaict.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could it be made persistable? Or does Python-side support for Persistable require something (e.g., @dataclass) that's incompatible with the existing class?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose it can, but I also thought it was useful to have an example here that was a straight dataclass with yaml persistence...it's a bit redundant, is that a problem?

Comment on lines +150 to +151
/// Read an arbitrary non-standard component by name.
std::shared_ptr<typehandling::Storable> readComponent(std::string const &componentName);
Copy link
Member

@TallJimbo TallJimbo Jan 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 for making this take enums and handle standard components in another ticket. IIRC, at least some of the standard components are also unusual components here, at least in terms of how the affect they metadata reading and hence interact with each other, and really the only safe way to handle them here would be to intercept them and delegate to the special method for that component. And "intercept all standard components" feels like a case where you really want to be using enums somehow to make sure the check is exhaustive (and remains exhaustive even if new standard components are added in other parts of the codebase).

Edit: I think my memory of how those enums worked and where they should be used is just faulty. Please ignore the above; I don't think it actually made sense, and I'm happy that the rest of you have it under control.

@erykoff erykoff merged commit d9c33f4 into master Jan 14, 2021
@erykoff erykoff deleted the tickets/DM-28208 branch January 14, 2021 05:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants