DM-13496: Prototype Butler Registry schema definition in YAML #13
Conversation
pschella
force-pushed
the
tickets/DM-13496
branch
from
1a96b2a
to
9109263
Compare
config/registry/default_schema.yaml
Outdated
| name: Visit | ||
| dependencies: | ||
| - Camera | ||
| - PhysicalFilter |
There was a problem hiding this comment.
I think we need some way of distinguishing between these relationships in this structure:
camerais part ofVisit's primary key, and needs to be provided along with thevisitnumber field to uniquely identify aVisit.physical_filteris not a part ofVisit's primary key, and is not necessary to uniquely identify aVisit.
config/registry/default_schema.yaml
Outdated
| doc: > | ||
| An inclusive range of Exposures that may be open in either direction. There | ||
| is no SQL representation; its fields (duplicated below) are all part of its | ||
| compound primary key and are hence just present in Dataset. |
There was a problem hiding this comment.
We should have a machine-readable way to represent the fact that the columns here are just fields that must be present Dataset, not something that requires an new Table to be created.
There was a problem hiding this comment.
My thought on that was that if all columns are values then no table is needed and all that exist are those columns in the Datasets table (e.g. the respective DataUnit table is skipped). Otherwise a separate table is created for this DataUnit.
There was a problem hiding this comment.
By "values", you mean "appears in the Dataset table"? I don't see those indicated some other way (though I think the condition you want might actually be "if all columns are primary keys").
In any case, even that condition isn't enough, because it can't distinguish between "have a definition-only table" and "no table".
config/registry/default_schema.yaml
Outdated
| Link to the Quantum table (only present for SuperTask-generated Datasets). | ||
| # DataUnit values | ||
| - !Column | ||
| <<: *camera |
There was a problem hiding this comment.
Are these going to inherit the primary_key attributes these column definitions have in the DataUnits section?
There was a problem hiding this comment.
Yes, unless overridden in the same way as nullable is.
config/registry/default_schema.yaml
Outdated
| name: parent_dataset_id | ||
| type: int | ||
| primary_key: true | ||
| nullable: false |
There was a problem hiding this comment.
In SQL, primary_key=True always implies nullable=False. Should we take advantage of that?
There was a problem hiding this comment.
Probably, although I'd want to check the standard on that to be sure (e.g. the actual primary key in sqllite is some auto-generated thing that you don't see so perhaps it does allow it to be nullable? Not sure.
pschella
force-pushed
the
tickets/DM-13496
branch
3 times, most recently
from
3562207
to
97e7517
Compare
| - | ||
| name: physical_filter | ||
| type: string | ||
| foreign_key: PhysicalFilter.physical_filter |
There was a problem hiding this comment.
Should this have one link (physical_filter) or two (physical_filter and camera)? I think you have all of the necessary information here, but just to confirm it's structured:
- No two
DataUnitsmay have the samelink. - A single
DataUnitsmay (rarely) have more than onelink(e.g.ExposureRange). - A
DataUnitis fully specified by itslinkand thelinks of allrequireddependencies. If theDataUnithas a table, "fully specified" means those columns correspond to its (compound) primary key.
There was a problem hiding this comment.
Yes, that is correct. I am assuming that when a link is inserted the dependencies are first expanded and also included. Thus dupplicate links are not allowed.
| Exposure: | ||
| dependencies: | ||
| required: | ||
| - Camera |
There was a problem hiding this comment.
Also:
optional:
- Visit
right?
(Just making sure this an accidental omission, not something about the structure I don't understand.)
There was a problem hiding this comment.
Yes, this was an accidental omission (there may be more).
pschella
force-pushed
the
tickets/DM-13496
branch
from
97e7517
to
a479a0b
Compare
No description provided.