DM-26662: ap_verify import slow in Gen 3 #107
Conversation
There was a problem hiding this comment.
Looks fine. Regarding config validation, Config is just a clever dict. It knows nothing about the content. Later on this will be part of a DatastoreConfig that can do validation but will also ignore things it doesn't care about. These are not pex_config. That can be good and it can be bad. In this case it's good because if you end up with an in-memory datastore nothing cares about the weird extra.
python/lsst/ap/verify/dataset.py
Outdated
| repoConfig = dafButler.Butler.makeRepo(repoDir) | ||
| initConfig = dafButler.Config() | ||
| # Checksums greatly slow importing of large repositories | ||
| initConfig[".datastore.checksum"] = False |
There was a problem hiding this comment.
Inside butler we prefer to write this as initConfig["datastore", "checksum"] rather than relying on the delimiter at all. You can of course keep it with the leading delimiter.
There was a problem hiding this comment.
If you prefer that format, consider emphasizing it in the class docs. Right now they focus almost entirely on delimiters, and all the alternatives are presented as if they were equal.
Checksums greatly slow down repository copying, particularly for large repositories.
kfindeisen
force-pushed
the
tickets/DM-26662
branch
from
49500b1
to
619f760
Compare
This PR creates the
ap_verifyrepository without checksums, to ensure that copying the repository is a fast operation (minutes vs. hours, for large repositories). Testing suggests thatdaf.butler.Configsilently ignores nonexistent keys, so the new code will work for both file-based and non-file datastores. However, this behavior doesn't seem to be documented anywhere.