Save cache metadata in JSON format #1353
Conversation
| try: | ||
| with open(fn, "r") as f: | ||
| return json.load(f) | ||
| except Exception: |
There was a problem hiding this comment.
Exception should be way more specific.
There was a problem hiding this comment.
IOError or JSON decode error. The latter will depend on the JSON library used, I don't think they nicely call it the same thing - but probably both subclass ValueError.
There was a problem hiding this comment.
The Exception was purposely generic. If anything fails with the JSON load it tried the pickle load. A corner case failure in the first will be repeated in the second.
It is not as simple as IOError or JSON decode errror. It is easy to obtain a UnicodeDecodeError for example, and I expect there are others that I haven't explicitly tested for.
But I don't have a strong opinion either way. I have changed it to a ValueError as suggested.
This changes the file format used for storing cached metadata from pickle to JSON. It eliminates the problem that pickled files written using one version of Python might not be readable using a different version of python, and an extra benefit is that such files are now human readable so easier to understand/debug.
All loading and saving of cache metadata is now performed in specific low-level functions
CacheMetadata._loadandCacheMetadata._save.We still support the reading of cached metadata from pickled files so that pre-existing caches should still be usable. Such files will be saved as JSON the next time they are saved.
The implementation uses
ujsonif it is installed, falling back tojson. This is the same approach used inReferenceFileSystem.The only slight complexity here is that we have to test both JSON and pickle formats in CI, so I have added a private boolean attribute
CacheMetadata._force_save_pickle. This should only be used in tests, not by end users. If set toTruesubsequent saves are in pickle format so we can simulate old pickled metadata to confirm it can still be read and is converted to JSON on next save.Fixes #825.