DM-25470: Define better Gen2 path template for raw images in obs_lsst

[rgruendl]

Update to configuration to better distribute files within the raw repos.
Only alteration is in policy/lsstCamMapper.yaml

[ktlim]

This looks fine. dayObs should be known (provided or looked up from expId).

[laurenam]

I couldn't help but notice you are having a "shooting in the dark" struggle on this one. I haven't followed the ticket closely, but I ran into a similar issue on DM-24585, so wanted to let you know how I got to the bottom of things in case it's of any help to you here (see comments on PR: #217).

I just set up your branch and then ran the following:

In [1]: import lsst.daf.persistence as dafPersist                                                             
In [2]: butlerDir = "/home/lauren/LSST/obs_lsst/data/input/comCam"                                            
In [3]: butler = dafPersist.Butler(butlerDir)                                                                 
LsstCamMapper WARN: Unable to find valid calib root directory
CameraMapper INFO: Loading exposure registry from /home/lauren/LSST/obs_lsst/data/input/comCam/registry.sqlite3

In [4]: levels = ['sensor', 'skyTile', 'filter', 'expId']                                                     
In [5]: for level in levels:  
   ...:     subsets = butler.subset("raw", level=level) 
   ...:     for subset in subsets: 
   ...:         print("Subset dataIds for level {}".format(level), subset.dataId) 
   ...:                                                                                                       
Subset dataIds for level sensor {'dayObs': '2019-05-30', 'expId': 3019053000001, 'raftName': 'R22', 'detectorName': 'S00', 'detector': 0}
Subset dataIds for level skyTile {'dayObs': '2019-05-30', 'expId': 3019053000001}
Subset dataIds for level filter {'expId': 3019053000001}
Subset dataIds for level expId {'dayObs': '2019-05-30', 'expId': 3019053000001}

which shows you what entries are actually expected (e.g. expId should include dayObs in addition to expId, which I don't think yours currently does). Hope this helps at least a bit...and feel free to ping me if you think I could be of further (any!) help.

[laurenam]

I think you also need:

diff --git a/policy/lsstCamMapper.yaml b/policy/lsstCamMapper.yaml
index 2e582c2..9675927 100644
--- a/policy/lsstCamMapper.yaml
+++ b/policy/lsstCamMapper.yaml
@@ -40,7 +40,7 @@ exposures:
     python: lsst.daf.base.PropertyList
     storage: FitsStorage
     tables: raw
-    template: raw/%(dayObs)s/%(expId)d/%(expId)013d-%(raftName)s-%(detectorName)s-det%(detector)03d.fits[0]
+    template: raw/%(dayObs)s/%(expId)013d/%(expId)013d-%(raftName)s-%(detectorName)s-det%(detector)03d.fits[0]
   raw:
     assembler: lsst.obs.lsst.lsstCamMapper.assemble_raw
     python: lsst.afw.image.DecoratedImageF
@@ -58,7 +58,7 @@ exposures:
     python: lsst.afw.image.DecoratedImageF
     storage: FitsStorage
     tables: raw
-    template: raw/%(dayObs)s/%(expId)d/%(expId)013d-%(raftName)s-%(detectorName)s-det%(detector)03d.fits[%(ch
+    template: raw/%(dayObs)s/%(expId)013d/%(expId)013d-%(raftName)s-%(detectorName)s-det%(detector)03d.fits[%
   postISRCCD:
     template: postISRCCD/%(expId)013d-%(filter)s/%(raftName)s/postISRCCD_%(expId)013d-%(filter)s-%(raftName)s
   postISRCCD_uninterpolated:
@@ -448,7 +448,7 @@ datasets:
     python: lsst.daf.base.PropertyList
     storage: FitsStorage
     tables: raw
-    template: raw/%(run)s/%(raftName)s/%(expId)013d-%(raftName)s-%(detectorName)s-det%(detector)03d.fits[0]
+    template: raw/%(dayObs)s/%(expId)013d/%(expId)013d-%(raftName)s-%(detectorName)s-det%(detector)03d.fits[0
   refcat:
     persistable: SimpleCatalog

(I'm procrastinating doing my taxes...)

[ktlim]

Looks great.