Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-14428: Spatial joins tables and views #55

Merged
merged 18 commits into from
Jun 28, 2018
Merged

DM-14428: Spatial joins tables and views #55

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
1216dab
Separate region part of DataUnit insertion.
TallJimbo Jun 2, 2018
3f0ff2c
Add support for multi-DataUnit regions.
TallJimbo Jun 2, 2018
f44d069
Include component Datasets in Gen2->Gen3 conversion.
TallJimbo Jun 2, 2018
39fe0fe
Slicing and more efficient component access for Exposure.
TallJimbo Jun 2, 2018
5f94268
Add metadata to Exposure StorageClass.
TallJimbo Jun 4, 2018
25b82ba
Add observation regions to Gen3 conversion.
TallJimbo Jun 4, 2018
911dcfb
Include SkyPix definition in Registry configuration.
TallJimbo Jun 4, 2018
4fecf0f
Use lists consistently in lhs, rhs for DataUnit joins.
TallJimbo Jun 19, 2018
b645292
Improvements to DataUnitJoin support.
TallJimbo Jun 19, 2018
21ed73d
Populate SkyPix join tables when setting DataUnit regions.
TallJimbo Jun 19, 2018
158255e
Add PATH to EUPS environment variables.
TallJimbo Jun 19, 2018
5406ba4
Add view for AbstractFilter.
TallJimbo Jun 19, 2018
d2af903
Improve docs for region columns in YAML.
TallJimbo Jun 26, 2018
0e704bf
Use more concise YAML list from for DataUnitJoins.
TallJimbo Jun 26, 2018
dd4d9fc
Rename comp argument to component.
TallJimbo Jun 26, 2018
0646e10
Remove unnecessary table file envvars.
TallJimbo Jun 26, 2018
44c303f
Use new-style super().
TallJimbo Jun 26, 2018
342b357
Add custom column type for regions.
TallJimbo Jun 26, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
10 changes: 10 additions & 0 deletions config/gen2convert.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,16 @@ mappers:
# as the Gen3 value.
uniqueKeys:
- visit
regions:
-
# The Gen3 DatasetType (Exposure StorageClass) to use when creating
# Visit-Sensor regions.
DatasetType: calexp
# Padding (in pixels) to add to the edges of each Sensor region
# to account for WCS inaccuracies.
padding: 5.0
# Collection from which WCSs should be read.
collection: null
collections:
substitutions:
# Regular expression patterns and replacement strings (passed directly
Expand Down
3 changes: 3 additions & 0 deletions config/registry.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,6 @@
registry:
cls: lsst.daf.butler.registries.sqliteRegistry.SqliteRegistry
db: 'sqlite:///:memory:'
skypix:
cls: lsst.sphgeom.HtmPixelization
level: 7
222 changes: 115 additions & 107 deletions config/schema.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,21 @@ schema:
type: string
doc: >
String name for the abstract filter, frequently a single character.
tables:
AbstractFilter:
doc: >
A view aggregating all distinct AbstractFilters references by any
PhysicalFilter.
columns:
-
name: abstract_filter
type: string
primary_key: true
doc: >
Name of the filter.
sql: >
SELECT DISTINCT abstract_filter
FROM PhysicalFilter;

PhysicalFilter:
dependencies:
Expand Down Expand Up @@ -276,14 +291,17 @@ schema:
effective area (arcsec).
-
name: region
type: blob
type: region
doc: >
A spatial region on the sky that bounds the area covered by the
Visit. This is expected to be more precise than the region covered
by the SkyPixels associated with the Visit, but may still be larger
than the Visit as long as it fully covers it. Must also fully
cover all regions in the VisitSensorRegion entries associated with
this Visit.
Regions are lsst.sphgeom.ConvexPolygon objects persisted as
portable (but not human-readable) bytestrings using the encode
and decode methods.
foreignKeys:
-
src: camera
Expand All @@ -295,59 +313,6 @@ schema:
tgt:
- PhysicalFilter.camera
- PhysicalFilter.physical_filter
VisitSensorRegion:
doc: >
A one-to-many table join table that stores the spatial region
associated with a particular Visit-Sensor combination. Entries are
expected to be calculated outside the database and added/updated
whenever the Visit's region is. This table contains exactly one
entry for each Visit+Sensor combination.
columns:
-
name: camera
type: string
primary_key: true
nullable: false
doc: Name of the Camera associated with the Visit and Sensor.
-
name: visit
type: int
primary_key: true
nullable: false
doc: Visit ID
-
name: sensor
type: int
primary_key: true
nullable: false
doc: visit
-
name: region
type: string
doc: >
A spatial region on the sky that bounds the area associated with
this Visit+Sensor combination. This is expected to be more
precise than the SkyPixels associated with the Visit+Sensor (see
VisitSensorSkyPixJoin), but may still be larger than the true
region as long as it fully covers it.
foreignKeys:
-
src: camera
tgt: Camera.camera
-
src:
- camera
- visit
tgt:
- Visit.camera
- Visit.visit
-
src:
- camera
- sensor
tgt:
- Sensor.camera
- Sensor.sensor
Exposure:
dependencies:
required:
Expand Down Expand Up @@ -592,12 +557,15 @@ schema:
Unique (with SkyMap) integer identifier for the Tract.
-
name: region
type: blob
type: region
doc: >
A spatial region on the sky that bounds the area associated
with the Tract. This is expected to be more precise than the
SkyPixels associated with the Visit (see TractSkyPixJoin), but
may still be larger than the Tract as long as it fully covers it.
Regions are lsst.sphgeom.ConvexPolygon objects persisted as
portable (but not human-readable) bytestrings using the encode
and decode methods.
foreignKeys:
-
src: skymap
Expand Down Expand Up @@ -664,13 +632,15 @@ schema:
Which row this Patch occupies in the Tract's grid of Patches.
-
name: region
type: blob
nullable: false
type: region
doc: >
A spatial region on the sky that bounds the area associated
with the Patch. This is expected to be more precise than the
SkyPixels associated with the Visit (see PatchSkyPixJoin), but
may still be larger than the Patch as long as it fully covers it.
Regions are lsst.sphgeom.ConvexPolygon objects persisted as
portable (but not human-readable) bytestrings using the encode
and decode methods.
foreignKeys:
-
src: skymap
Expand All @@ -694,16 +664,74 @@ schema:
type: string
doc: A string value composed only of letters, numbers, and underscores.

dataUnitRegions:
-
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason why this is a list rather than associative array like dataUnits and dataUnitJoins?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I felt like the conceptual key for one of these items is the "relates" entry, which is itself a list and hence can't be used as a key directly. So a list seemed like the next-best thing.

relates:
- Visit
- Sensor
tables:
VisitSensorRegion:
doc: >
A one-to-many table join table that stores the spatial region
associated with a particular Visit-Sensor combination. Entries are
expected to be calculated outside the database and added/updated
whenever the Visit's region is. This table contains exactly one
entry for each Visit+Sensor combination.
columns:
-
name: camera
type: string
primary_key: true
nullable: false
doc: Name of the Camera associated with the Visit and Sensor.
-
name: visit
type: int
primary_key: true
nullable: false
doc: Visit ID
-
name: sensor
type: int
primary_key: true
nullable: false
doc: Sensor ID
-
name: region
type: region
doc: >
A spatial region on the sky that bounds the area associated with
this Visit+Sensor combination. This is expected to be more
precise than the SkyPixels associated with the Visit+Sensor (see
VisitSensorSkyPixJoin), but may still be larger than the true
region as long as it fully covers it.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we describe string representation for regions anywhere or do we depend on sphgeom for it? If latter them maybe add couple of words here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do rely on sphgeom for this. I'll add some docs.

Regions are lsst.sphgeom.ConvexPolygon objects persisted as
portable (but not human-readable) bytestrings using the encode
and decode methods.
foreignKeys:
-
src:
- camera
- visit
tgt:
- Visit.camera
- Visit.visit
-
src:
- camera
- sensor
tgt:
- Sensor.camera
- Sensor.sensor

dataUnitJoins:
ExposureRangeJoin:
doc: >
A calculated join between Datasets identified with an Exposure
(typically raw science frames) and Datasets identified with ExposureRange
(typically master calibrations).
lhs:
- Exposure
rhs:
- ExposureRange
lhs: [Exposure]
rhs: [ExposureRange]
sql:
(lhs.camera = rhs.camera) AND
(lhs.exposure BETWEEN rhs.valid_first AND rhs.valid_last)
Expand All @@ -715,10 +743,8 @@ schema:
observations with the main camera observations they support. It could
also be used to relate CBP state (represented as CBP "Exposures") with
actual main camera observations of the CBP.
lhs:
- Exposure
rhs:
- Exposure
lhs: [Exposure]
rhs: [Exposure]
tables:
MultiCameraExposureJoin:
columns:
Expand Down Expand Up @@ -766,10 +792,8 @@ schema:
Entries are expected to be calculated outside the database and
added/updated whenever VisitSensorRegion is.

lhs:
- Visit
- Sensor
rhs: SkyPix
lhs: [Visit, Sensor]
rhs: [SkyPix]
tables:
VisitSensorSkyPixJoin:
columns:
Expand Down Expand Up @@ -812,12 +836,9 @@ schema:
This can and probably should be implemented as a trivial view on
VisitSensorSkyPixJoin.

lhs:
- Visit
rhs:
- SkyPix
summarizes:
- VisitSensorSkyPixJoin
lhs: [Visit]
rhs: [SkyPix]
summarizes: [VisitSensorSkyPixJoin]
tables:
VisitSkyPixJoin:
columns:
Expand Down Expand Up @@ -856,8 +877,8 @@ schema:
Entries are expected to be calculated outside the database and
added along with the Patch itself.

lhs: Patch
rhs: SkyPix
lhs: [Patch]
rhs: [SkyPix]
tables:
PatchSkyPixJoin:
columns:
Expand Down Expand Up @@ -900,10 +921,9 @@ schema:
This can and probably should be implemented as a trivial view on
PatchSkyPixJoin.

lhs: Tract
rhs: SkyPix
summarizes:
- PatchSkyPixJoin
lhs: [Tract]
rhs: [SkyPix]
summarizes: [PatchSkyPixJoin]
tables:
TractSkyPixJoin:
columns:
Expand Down Expand Up @@ -941,15 +961,12 @@ schema:
Should be implemented as a view; it may be materialized as long as it
can be kept up to date when new Visits or SkyMaps are added.

If a database UDF is available to determine whether two region blobs
If a database UDF is available to determine whether two regions
overlap, we could include that in this view to refine the results.
For now, we will assume that such a UDF is not available.

lhs:
- Visit
- Sensor
rhs:
- Patch
lhs: [Visit, Sensor]
rhs: [Patch]
tables:
VisitSensorPatchJoin:
columns:
Expand Down Expand Up @@ -1022,16 +1039,13 @@ schema:
Should be implemented as a view; it may be materialized as long as it
can be kept up to date when new Visits or SkyMaps are added.

If a database UDF is available to determine whether two region blobs
If a database UDF is available to determine whether two regions
overlap, we could include that in this view to refine the results.
For now, we will assume that such a UDF is not available.

lhs:
- Visit
rhs:
- Patch
summarizes:
- VisitSensorPatchJoin
lhs: [Visit]
rhs: [Patch]
summarizes: [VisitSensorPatchJoin]
tables:
VisitPatchJoin:
columns:
Expand Down Expand Up @@ -1096,17 +1110,13 @@ schema:
Should be implemented as a view; it may be materialized as long as it
can be kept up to date when new Visits or SkyMaps are added.

If a database UDF is available to determine whether two region blobs
If a database UDF is available to determine whether two regions
overlap, we could include that in this view to refine the results.
For now, we will assume that such a UDF is not available.

lhs:
- Visit
- Sensor
rhs:
- Tract
summarizes:
- VisitSensorPatchJoin
lhs: [Visit, Sensor]
rhs: [Tract]
summarizes: [VisitSensorPatchJoin]
tables:
VisitSensorTractJoin:
columns:
Expand Down Expand Up @@ -1171,14 +1181,12 @@ schema:
Should be implemented as a view; it may be materialized as long as it
can be kept up to date when new Visits or SkyMaps are added.

If a database UDF is available to determine whether two region blobs
If a database UDF is available to determine whether two regions
overlap, we could include that in this view to refine the results.
For now, we will assume that such a UDF is not available.

lhs:
- Visit
rhs:
- Tract
lhs: [Visit]
rhs: [Tract]
summarizes:
- VisitSensorPatchJoin
- VisitSensorTractJoin
Expand Down