What stack version to read NERSC Gen3 skymap?

[wmwv]

Description
A clear and concise description of what the issue is.

Can't read skyMap for Gen 3 Run2.2i, Run3.1i with recent desc-stack-weekly.

@jchiang87 @heather999 What version of the stack was used to produce the skyMap for the NERSC Gen3 Run2.2i and Run3.1i datasets? I think it's incompatible with recent stack versions. Specifically I'm using the Jupyter kernel desc-stack-weekly as of today (2012-04-12).

/global/cfs/cdirs/lsst/production/gen3/DC2/Run2.2i/repo/skymaps/skyMap/

Choose all applicable topics by placing an 'X' between the [ ]:

• jupyter
• jupyter terminal
• Cori interactive command line
• Batch jobs
• python
• CSCRATCH
• Community File System
• HPSS (tape)
• Data transfer and Globus
• New User Account or account access problems

To Reproduce
Steps to reproduce the behavior:

1. Go to Jupyter
2. Click on desc-stack-weekly for kernel
3. Run

from lsst.daf.butler import Butler
butler = Butler("/global/cfs/cdirs/lsst/production/gen3/DC2/Run2.2i/repo")
skymap = butler.get("skyMap", skymap="DC2", collections=collection)

5. See error

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.5.0/Linux64/daf_butler/21.0.0-84-g88f84237+36a54c4eaa/python/lsst/daf/butler/datastores/fileDatastore.py in _read_artifact_into_memory(self, getInfo, ref, isComponent)
   1080             try:
-> 1081                 result = formatter.fromBytes(serializedDataset,
   1082                                              component=getInfo.component if isComponent else None)

/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.5.0/Linux64/daf_butler/21.0.0-84-g88f84237+36a54c4eaa/python/lsst/daf/butler/formatters/file.py in fromBytes(self, serializedDataset, component)
    226         # mypy can not understand that the previous line protects this call
--> 227         data = self._fromBytes(serializedDataset,  # type: ignore
    228                                self.fileDescriptor.storageClass.pytype)

/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.5.0/Linux64/daf_butler/21.0.0-84-g88f84237+36a54c4eaa/python/lsst/daf/butler/formatters/pickle.py in _fromBytes(self, serializedDataset, pytype)
    104         try:
--> 105             data = pickle.loads(serializedDataset)
    106         except pickle.PicklingError:

/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.5.0/Linux64/pex_config/21.0.0-1-ga51b5d4+2b9330c81e/python/lsst/pex/config/config.py in unreduceConfig(cls, stream)
   1540     config = cls()
-> 1541     config.loadFromStream(stream)
   1542     return config

/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.5.0/Linux64/pex_config/21.0.0-1-ga51b5d4+2b9330c81e/python/lsst/pex/config/config.py in loadFromStream(self, stream, root, filename)
   1074                 local = {root: self}
-> 1075                 exec(stream, globals, local)
   1076             except NameError as e:

<string> in <module>

ModuleNotFoundError: No module named 'lsst.skymap.tractBuilder'

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
<ipython-input-124-8b34c6477891> in <module>
----> 1 skymap = butler.get("skyMap", skymap="DC2", collections=collection)  #u/descdm/sfp_ddf_visits_part_01")

/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.5.0/Linux64/daf_butler/21.0.0-84-g88f84237+36a54c4eaa/python/lsst/daf/butler/_butler.py in get(self, datasetRefOrType, dataId, parameters, collections, **kwds)
   1013         log.debug("Butler get: %s, dataId=%s, parameters=%s", datasetRefOrType, dataId, parameters)
   1014         ref = self._findDatasetRef(datasetRefOrType, dataId, collections=collections, **kwds)
-> 1015         return self.getDirect(ref, parameters=parameters)
   1016 
   1017     def getURIs(self, datasetRefOrType: Union[DatasetRef, DatasetType, str],

/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.5.0/Linux64/daf_butler/21.0.0-84-g88f84237+36a54c4eaa/python/lsst/daf/butler/_butler.py in getDirect(self, ref, parameters)
    880             The dataset.
    881         """
--> 882         return self.datastore.get(ref, parameters=parameters)
    883 
    884     def getDirectDeferred(self, ref: DatasetRef, *,

/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.5.0/Linux64/daf_butler/21.0.0-84-g88f84237+36a54c4eaa/python/lsst/daf/butler/datastores/fileDatastore.py in get(self, ref, parameters)
   1533                 getInfo.formatter.fileDescriptor.storageClass.validateParameters(parameters)
   1534 
-> 1535             return self._read_artifact_into_memory(getInfo, ref, isComponent=isComponent)
   1536 
   1537     @transactional

/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.5.0/Linux64/daf_butler/21.0.0-84-g88f84237+36a54c4eaa/python/lsst/daf/butler/datastores/fileDatastore.py in _read_artifact_into_memory(self, getInfo, ref, isComponent)
   1082                                              component=getInfo.component if isComponent else None)
   1083             except Exception as e:
-> 1084                 raise ValueError(f"Failure from formatter '{formatter.name()}' for dataset {ref.id}"
   1085                                  f" ({ref.datasetType.name} from {uri}): {e}") from e
   1086         else:

ValueError: Failure from formatter 'lsst.daf.butler.formatters.pickle.PickleFormatter' for dataset 8c7ea7cf-6bcb-4cd4-aed1-855c02afc2d8 (skyMap from file:///global/cfs/cdirs/lsst/production/gen3/DC2/Run2.2i/repo/skymaps/skyMap/skyMap_DC2_skymaps.pickle): No module named 'lsst.skymap.tractBuilder'

Screenshots
If applicable, add screenshots to help explain your problem.

[wmwv]

Even simpler reproduce script is:

import pickle
filename = "/global/cfs/cdirs/lsst/production/gen3/DC2/Run2.2i/repo/skymaps/skyMap/skyMap_DC2_skymaps.pickle"
with open(filename, "rb") as file:
    obj = pickle.load(file)

[arunkannawadi]

I don't know much about the NERSC setup, but the skyMap itself is intended to be backwards compatible. I wonder if the skymap package has not been updated recently. The traceback points to changes made in November 2021 here.

[jchiang87]

It appears I used w_2021_48 to set up the Run2.2i repo in /global/cfs/cdirs/lsst/production/gen3/DC2/Run2.2i/repo.

[heather999]

If it's helpful @wmwv I can make w_2021_48 available as a jupyter kernel - basically importing that specific docker image into shifter and pass you the kernel.json file you'd need to set it up.

[wmwv]

That'd be great, thanks, @heather999 .

[jchiang87]

I'd recommend using v23.0.1 instead of a weekly. I think that's originally based off of w_2021_50, which I've been using for the single frame processing.

[wmwv]

Oh, I see. I was also confused because desc-stack-weekly-latest is recent (w_2022_10). But desc-stack-weekly is not recent.

[jchiang87]

Actually, a quick sanity check has the same issue with that version:

[cori04] source /cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/v23.0.1/loadLSST-ext.bash 
(lsst-scipipe-0.8.0-ext) [cori04] setup lsst_distrib
(lsst-scipipe-0.8.0-ext) [cori04] ipython
Python 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.0.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pickle
   ...: filename = "/global/cfs/cdirs/lsst/production/gen3/DC2/Run2.2i/repo/skymaps/skyMap/skyMap_DC2_sk
   ...: ymaps.pickle"
   ...: with open(filename, "rb") as file:
   ...:     obj = pickle.load(file)
   ...: 
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Input In [1], in <module>
      2 filename = "/global/cfs/cdirs/lsst/production/gen3/DC2/Run2.2i/repo/skymaps/skyMap/skyMap_DC2_skymaps.pickle"
      3 with open(filename, "rb") as file:
----> 4     obj = pickle.load(file)

File /cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/v23.0.1/conda/envs/lsst-scipipe-0.8.0-ext/share/eups/Linux64/pex_config/g87000a63fb+536b1ee016/python/lsst/pex/config/config.py:1541, in unreduceConfig(cls, stream)
   1521 """Create a `~lsst.pex.config.Config` from a stream.
   1522 
   1523 Parameters
   (...)
   1538 lsst.pex.config.Config.loadFromStream
   1539 """
   1540 config = cls()
-> 1541 config.loadFromStream(stream)
   1542 return config

File /cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/v23.0.1/conda/envs/lsst-scipipe-0.8.0-ext/share/eups/Linux64/pex_config/g87000a63fb+536b1ee016/python/lsst/pex/config/config.py:1075, in Config.loadFromStream(self, stream, root, filename)
   1073 try:
   1074     local = {root: self}
-> 1075     exec(stream, globals, local)
   1076 except NameError as e:
   1077     if root == "config" and "root" in e.args[0]:

File <string>:3, in <module>

ModuleNotFoundError: No module named 'lsst.skymap.tractBuilder'

Let me poke around. I'm surprised this didn't work.

[wmwv]

w_2022_10 is working for me.

[heather999]

Right desc-stack-weekly is not recent, desc-stack-weekly-latest hasn't moved past w_2022_10 - on the list to fix this week :)