-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DM-14191: Merge butler configuration files with defaults #39
Merged
Merged
Changes from 24 commits
Commits
Show all changes
26 commits
Select commit
Hold shift + click to select a range
e929ce8
Butler config can now merge with defaults
timj 611cf94
Fix typo in docstring
timj f365908
Provide config defaults
timj 120a0c9
Rewrite doImport to use importlib rather than __import__
timj fee316f
Move ButlerConfig to its own file
timj 39dc7ef
Registry configuration needs schema and registry config entries
timj 5b170a0
Remove StorageClassFactory.fromConfig
timj d9b94b4
Add config subclass that can extract a sub component
timj c88286f
Use ConfigSubset
timj 48ac511
Simplify registry constructor
timj aa1472a
Rename registry defaults file
timj 21dad01
Use self-describing config classes
timj 2612595
Update tests to use ButlerConfig and butler config directory
timj 2fc0e3d
Prefer to throw ImportError rather than AttributeError
timj 3bcf273
Rewrite config merging to use subset configs
timj fdc02e9
Use defaulting to read schema
timj 76b6020
Move some test configuration information to defaults
timj 86d9613
Allow default butler config
timj 76e603e
Add parens around pdb example
timj 6f65bbc
Add lots of tests for configuration merging
timj f022d5b
Move merging discussion to Config class
timj 4e4620d
Disable C++ check
timj e0fc9cf
If YAML file is empty create empty dict
timj 0507933
Overhaul merging code to pass tests
timj 29cb101
Rename registry_schema to schema config file
timj bd82bd6
Fix dangling sentence
timj File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| # -*- python -*- | ||
| from lsst.sconsUtils import scripts | ||
|
|
||
| scripts.BasicSConstruct("daf_butler") | ||
| scripts.BasicSConstruct("daf_butler", disableCc=True) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| datastore: | ||
| # Use the PosixDatastore as a default for a default butler | ||
| cls: lsst.daf.butler.datastores.posixDatastore.PosixDatastore |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| datastore: | ||
| cls: lsst.daf.butler.datastores.posixDatastore.PosixDatastore | ||
| root: default_butler_repository | ||
| records: | ||
| table: PosixDatastoreRecords | ||
| create: true | ||
| templates: | ||
| default: "{datasetType}/{tract:?}/{patch:?}/{filter:?}/{camera:?}_{visit:?}" | ||
| formatters: | ||
| TablePersistable: lsst.daf.butler.formatters.fitsCatalogFormatter.FitsCatalogFormatter | ||
| TablePersistableWcs: lsst.daf.butler.formatters.fitsCatalogFormatter.FitsCatalogFormatter | ||
| TablePersistablePsf: lsst.daf.butler.formatters.fitsCatalogFormatter.FitsCatalogFormatter | ||
| TablePersistableCoaddInputs: lsst.daf.butler.formatters.fitsCatalogFormatter.FitsCatalogFormatter | ||
| TablePersistableVisitInfo: lsst.daf.butler.formatters.fitsCatalogFormatter.FitsCatalogFormatter | ||
| TablePersistableApCorr: lsst.daf.butler.formatters.fitsCatalogFormatter.FitsCatalogFormatter | ||
| TablePersistableCalib: lsst.daf.butler.formatters.fitsCatalogFormatter.FitsCatalogFormatter | ||
| ImageF: lsst.daf.butler.formatters.fitsCatalogFormatter.FitsCatalogFormatter | ||
| MaskX: lsst.daf.butler.formatters.fitsCatalogFormatter.FitsCatalogFormatter | ||
| SourceCatalog: lsst.daf.butler.formatters.fitsCatalogFormatter.FitsCatalogFormatter | ||
| Exposure: lsst.daf.butler.formatters.fitsExposureFormatter.FitsExposureFormatter | ||
| ExposureF: lsst.daf.butler.formatters.fitsExposureFormatter.FitsExposureFormatter | ||
| ExposureI: lsst.daf.butler.formatters.fitsExposureFormatter.FitsExposureFormatter | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| # Default with a sqlLite registry | ||
| registry: | ||
| cls: lsst.daf.butler.registries.sqliteRegistry.SqliteRegistry | ||
| db: 'sqlite:///:memory:' |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm a bit surprised to not see the default mapping from StorageClass to Formatter here. Those are specific for PosixDatastore, right?
I don't actually care if they go here or elsewhere, but we need an easy way to construct a Butler that knows about those defaults.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wasn't entirely sure what defaults would mean since the formatters really do lock in the file format, but you are right that FITS is where we are so our default for "how to write an Exposure" should be in this file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A lot of those defaults currently live in the
gen3module ofci_hsc(on thegen3-middlewarebranch). I'd love to get those moved out of Python code into YAML. I'd also be happy to take care of making a ci_hsc branch compatible with this ticket once you're done with the daf_butler changes.