DM-39434: Avoid defaultdict with lambda in pickled dataclasses

[andy-slac]

Pickle does not like lambdas or local/nested methods, using a dataclass.field
with defaultdict and default_factory as a lambda breaks pickle. Instead of defining
a module-level factory method I decided to use a plain dict and its setdefault method.
Also had to add explicit pickle method to StoredFileInfo, being a frozen
dataclass it does not work by default with pickle.

Checklist

• ran Jenkins
• added a release note for user-visible changes to doc/changes

[codecov]

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.02 🎉

Comparison is base (ec51e16) 87.91% compared to head (c3c26c9) 87.93%.

❗ Current head c3c26c9 differs from pull request most recent head 719fcf9. Consider uploading reports for the commit 719fcf9 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #844      +/-   ##
==========================================
+ Coverage   87.91%   87.93%   +0.02%     
==========================================
  Files         268      268              
  Lines       35240    35256      +16     
  Branches     7396     7392       -4     
==========================================
+ Hits        30980    31004      +24     
+ Misses       3119     3112       -7     
+ Partials     1141     1140       -1     

Impacted Files	Coverage Δ
python/lsst/daf/butler/core/datastoreRecordData.py	96.00% <100.00%> (+11.06%)	⬆️
python/lsst/daf/butler/core/storedFileInfo.py	100.00% <100.00%> (ø)
python/lsst/daf/butler/datastores/fileDatastore.py	82.33% <100.00%> (+0.01%)	⬆️
tests/test_datastore.py	99.54% <100.00%> (+<0.01%)	⬆️

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[timj]

I forget why we can't use a single pydantic model for this and instead have to have a pydantic model and a dataclass. Is it for performance reasons? Presumably the pydantic model would pickle just fine.

Can you add a bit more detail about having to drop defaultdict in the commit message?

[andy-slac]

I forget why we can't use a single pydantic model for this and instead have to have a pydantic model and a dataclass. Is it for performance reasons? Presumably the pydantic model would pickle just fine.

I'm not sure what the exact reason was, but I see that pydantic and dataclass have different structure, maybe pydantic is more YAML/JSON-friedly than the in-memory dataclass?

Can you add a bit more detail about having to drop defaultdict in the commit message?

I updated the commit message.