Skip to content

Conversation

kfindeisen
Copy link
Member

@kfindeisen kfindeisen commented Sep 6, 2022

This PR adds explicit instrument and skymap dimensions to all Butler queries, and removes code that the use of default dimensions had made necessary. The result is much more likely to behave consistently in different conditions, increasing the value and relevance of the existing unit tests. The first few commits fix some smaller bugs discovered while examining the code or testing the explicit dimensions.

This PR depends on, and cannot be merged before, #24.

@kfindeisen kfindeisen changed the base branch from main to tickets/DM-35053 September 7, 2022 22:05
@kfindeisen kfindeisen marked this pull request as ready for review September 7, 2022 22:15
@kfindeisen kfindeisen requested a review from parejkoj September 7, 2022 22:16
Copy link
Contributor

@parejkoj parejkoj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still slightly grumpy that we can't use constant things like instrument and skymap as butler defaults, but I think I understand why you changed it to this. My grumpiness shouldn't prevent merging; this looks fine otherwise (note a few comment-related comments).

Should we just be storing instrument.getName(), or do we need other properties of the Instrument object?

collections=self.instrument.makeUnboundedCalibrationRunName()
)
self.skymap = self.central_butler.get("skyMap")
# TODO: is central_butler guaranteed to have only one skymap dimension?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the assert below, I think you don't need this TODO. We're defining the central repo as having exactly one skymap as part of this, and I think that's something that we as the AP group can ensure.

Copy link
Member Author

@kfindeisen kfindeisen Sep 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see why the assert makes the TODO irrelevant. The assert is computer-readable but easy for a human to overlook.

I'm not so confident that we will only have one skymap -- I can imagine a lot of experimentation over which skymap gives e.g. the best performance, especially during commissioning, and there might even be a policy some day that different datasets or data sources are based on different skymaps. Ultimately, it's not our call, but K-T's and the commissioning team's.

@kfindeisen
Copy link
Member Author

I'm still slightly grumpy that we can't use constant things like instrument and skymap as butler defaults, but I think I understand why you changed it to this.

But the problem was that these things aren't constant, especially when the local repo is first being set up -- the code made assumptions that weren't actually true IRL.

Should we just be storing instrument.getName(), or do we need other properties of the Instrument object?

Well, we use a lot of Instrument's collection-management methods. I'd keep it around just for flexibility, because it's easier to go from an Instrument to a name than from a name to an Instrument.

@timj
Copy link
Member

timj commented Sep 20, 2022

Name to instrument is standard API. We use it all the time for the command-line tooling: Instrument.fromName().

@kfindeisen kfindeisen force-pushed the tickets/DM-35941 branch 2 times, most recently from 63b2cae to ca496da Compare September 20, 2022 20:45
@kfindeisen
Copy link
Member Author

@parejkoj, I'm going to try to rearrange the branches so that this one can be merged before tickets/DM-35053. Since 9a19f13 and ca496da only make sense after both sets of changes, I'll have to cherry-pick them (including the explanatory comment you requested) to #24 and remove them here.

The new code avoids keeping around a tuple when we only want one component.
The meaning assumed by upload.py and activator.py is taken as truth; tests have
been updated to match.
Coadded templates are band-specific; the previous query pulled templates for
all bands even though only one was needed.
All queries already used full dimensions, so this is just a matter of removing
a few comments.
This avoids bugs related to the inconsistent and idiosyncratic handling of
governor dimensions inside Middleware.
This avoids bugs related to the inconsistent and idiosyncratic handling of
governor dimensions inside Middleware.
This change protects the code from depending on defaults in the future.
The function requires that all queries be expressed in a way that is valid (but
potentially returns nothing) for both central and local repositories.
@kfindeisen kfindeisen changed the base branch from tickets/DM-35053 to main September 21, 2022 20:07
@kfindeisen kfindeisen merged commit e158d96 into main Sep 21, 2022
@kfindeisen kfindeisen deleted the tickets/DM-35941 branch September 21, 2022 21:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants