DM-28389: Create a gen3 task to collate tract-level parquet tables for QA analyses

[laurenam]

No description provided.

[timj]

It seems wrong to me that we have to duplicate the mapping from band to physical filter here (and it doesn't have the second physical filters) since this information is already defined explicitly in obs_subaru in hscFilters.py and also directly discoverable from the butler itself.

[laurenam]

Oh, that’s great! I would be very happy to be getting this from the butler and get rid of this file altogether 😄

[timj]

You can use butler.registry.queryDimensionRecords. From the command line you get something like:

$ butler query-dimension-records tmp physical_filter
instrument              name                          band            
---------- ----------------------------- -----------------------------
       HSC                        ENG-R1                            r1
       HSC                         HSC-G                             g
       HSC                         HSC-I                             i
       HSC                        HSC-I2                             i
       HSC                         HSC-R                             r
       HSC                        HSC-R2                             r
       HSC                         HSC-Y                             y
       HSC                         HSC-Z                             z
       HSC                        IB0945                          I945
       HSC                        NB0387                          N387
       HSC                        NB0400                          N400
       HSC                        NB0468                          N468
       HSC                        NB0515                          N515
       HSC                        NB0527                          N527
       HSC                        NB0656                          N656
       HSC                        NB0718                          N718
       HSC                        NB0816                          N816
       HSC                        NB0921                          N921
       HSC                        NB0926                          N926
       HSC                        NB0973                          N973
       HSC                        NB1010                         N1010
       HSC                          NONE                  UNRECOGNISED
       HSC                            PH                            PH
       HSC                            SH                            SH

[laurenam]

Perfect...thanks!

[timj]

You can restrict the query to just the filter you are interested in for the instrument you use.

[laurenam]

Exactly... I think the confusion was the thought that I was trying to switch to physical rather than sticking with band. This is not the case at all (I’m totally on board with just band for any and all things coadd!). It’s just that I currently need physical to get at the data in the _ojb tables.

I have also been speculating that this’ll just naturally get fixed along with the Gen3-ification of the code making the _obj tables, hence me very much referring to this as a temporary fix. I’m perfectly happy to hold off merging this until the fix is in so that this never shows up as part of the history on master (it shouldn’t be too painful for me to just run off the two branches manually for the intervening RC2 runs).

[laurenam]

Thanks, Nate. Yeah, I fully landed on NOT using ButlerQC.regitry, but it did seem worthwhile pointing out that it could be done (and hence a trap others could fall into).

Ah, yes, I knew there had to be a way to reuse that config...thanks for the tip!

[timj]

It still seems fundamentally wrong to me that a pipeline file should define a filter-to-band mapping when we already have that mapping defined in two other places (such as in obs_subaru itself in the filter definitions).

[natelust]

It depends on what it is. I see nothing wrong with having a config file that can be imported and used anywhere you want "here is what is defined in obs_subaru" But also these are configurable fields, so if someone wants to configure them in a non-standard way for whatever their reason (emulate a wider band from a different camera etc) then they should be able to specify the config parameter to their liking.

[TallJimbo]

It still seems fundamentally wrong to me that a pipeline file should define a filter-to-band mapping when we already have that mapping defined in two other places (such as in obs_subaru itself in the filter definitions).

It's not a physical_filter-to-band mapping; it's a band-to-pseudo/historical-physical_filter mapping, and we don't have one of those in obs_subaru because it's an ill-defined thing. Unfortunately it's exactly the ill-defined thing that's still in use in Gen2.