Skip to content

Commit

Permalink
Prototype partial(?) replacements relation-tree objects.
Browse files Browse the repository at this point in the history
The current daf_relation tree in our Query objects is too contaminated
with implementation details to serve us well in the RemoteButler
client, and we could only partially mitigate that by defining a new
daf_relation Engine.  The fundamental issue is that daf_relation
objects expect to know exactly which columns they have at any given
time, and that's at odds with user expectations that columns
"magically" appear whenever they're requested (e.g. referenced by a
`where` string), and that this joins in the tables that provide them
as-needed.

The two new files here *heavily* duplicate stuff in daf_relation, and
in addition to being more vague about what the columns are, they're
simpler for two key reasons:

- They're just data, with no logic for maintaining invariants,
  contructing trees, serialization, or anything else.  This will have
  to change as we actually start to use them.

- They fully enumerate the kinds of expressions and operations we care
  about in the butler query system rather than trying to define
  abstract versions of those upstream of daf_butler that could be
  specialized in daf_butler.  I had not appreciated how much of a
  simplification this could be when writing daf_relation as a separate
  package, and if it holds up it may suggest that the right way to
  resolve the duplication is to rip a lot of stuff out of daf_relation.
  serialization.
  • Loading branch information
TallJimbo committed Nov 30, 2023
1 parent e2e7c40 commit c53102e
Show file tree
Hide file tree
Showing 3 changed files with 519 additions and 0 deletions.
26 changes: 26 additions & 0 deletions python/lsst/daf/butler/queries/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# This file is part of daf_butler.
#
# Developed for the LSST Data Management System.
# This product includes software developed by the LSST Project
# (http://www.lsst.org).
# See the COPYRIGHT file at the top-level directory of this distribution
# for details of code ownership.
#
# This software is dual licensed under the GNU General Public License and also
# under a 3-clause BSD license. Recipients may choose which of these licenses
# to use; please see the files gpl-3.0.txt and/or bsd_license.txt,
# respectively. If you choose the GPL option then the following text applies
# (but note that there is still no warranty even if you opt for BSD instead):
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
211 changes: 211 additions & 0 deletions python/lsst/daf/butler/queries/abstract_expressions.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,211 @@
# This file is part of daf_butler.
#
# Developed for the LSST Data Management System.
# This product includes software developed by the LSST Project
# (http://www.lsst.org).
# See the COPYRIGHT file at the top-level directory of this distribution
# for details of code ownership.
#
# This software is dual licensed under the GNU General Public License and also
# under a 3-clause BSD license. Recipients may choose which of these licenses
# to use; please see the files gpl-3.0.txt and/or bsd_license.txt,
# respectively. If you choose the GPL option then the following text applies
# (but note that there is still no warranty even if you opt for BSD instead):
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.

from __future__ import annotations

Check warning on line 28 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L28

Added line #L28 was not covered by tests

__all__ = (

Check warning on line 30 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L30

Added line #L30 was not covered by tests
"AbstractExpression",
"AbstractOrderExpression",
"AbstractPredicate",
)


import dataclasses
from typing import TYPE_CHECKING, Literal, TypeAlias, Union

Check warning on line 38 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L37-L38

Added lines #L37 - L38 were not covered by tests

import astropy.time
from lsst.sphgeom import Region

Check warning on line 41 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L40-L41

Added lines #L40 - L41 were not covered by tests

from .._column_tags import DatasetColumnTag, DimensionKeyColumnTag, DimensionRecordColumnTag
from .._timespan import Timespan
from ..dimensions import DataCoordinate

Check warning on line 45 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L43-L45

Added lines #L43 - L45 were not covered by tests

if TYPE_CHECKING:
from .abstract_relations import AbstractRelation


LiteralValue: TypeAlias = Union[int, bytes, str, float, astropy.time.Time, Timespan, Region]

Check warning on line 51 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L51

Added line #L51 was not covered by tests


@dataclasses.dataclass(frozen=True)
class ColumnLiteral:

Check warning on line 55 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L55

Added line #L55 was not covered by tests
"""A column expression that is a literal Python value."""

value: LiteralValue

Check warning on line 58 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L58

Added line #L58 was not covered by tests


@dataclasses.dataclass(frozen=True)
class ColumnReference:

Check warning on line 62 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L62

Added line #L62 was not covered by tests
"""A column expression that refers to a column obtainable from an abstract
relation.
"""

column: DimensionKeyColumnTag | DimensionRecordColumnTag | DatasetColumnTag

Check warning on line 67 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L67

Added line #L67 was not covered by tests


@dataclasses.dataclass(frozen=True)
class UnaryExpression:

Check warning on line 71 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L71

Added line #L71 was not covered by tests
"""A unary operation on a column expression that returns a non-bool."""

operand: AbstractExpression
operator: Literal["-", "begin_of", "end_of"]

Check warning on line 75 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L74-L75

Added lines #L74 - L75 were not covered by tests


@dataclasses.dataclass(frozen=True)
class BinaryExpression:

Check warning on line 79 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L79

Added line #L79 was not covered by tests
"""A binary operation on column expressions that returns a non-bool."""

a: AbstractExpression
b: AbstractExpression
operator: Literal["+", "-", "*", "/", "%"]

Check warning on line 84 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L82-L84

Added lines #L82 - L84 were not covered by tests


AbstractExpression: TypeAlias = Union[ColumnLiteral, ColumnReference, UnaryExpression, BinaryExpression]

Check warning on line 87 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L87

Added line #L87 was not covered by tests


@dataclasses.dataclass(frozen=True)
class Reversed:

Check warning on line 91 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L91

Added line #L91 was not covered by tests
"""A tag wrapper for `AbstractExpression` that indicate sorting in
reverse order.
"""

operand: AbstractExpression

Check warning on line 96 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L96

Added line #L96 was not covered by tests


AbstractOrderExpression: TypeAlias = Union[AbstractExpression, Reversed]

Check warning on line 99 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L99

Added line #L99 was not covered by tests


@dataclasses.dataclass(frozen=True)
class LogicalAnd:

Check warning on line 103 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L103

Added line #L103 was not covered by tests
"""A boolean column expression that is `True` only if all of its operands
are `True`.
"""

operands: tuple[AbstractPredicate]

Check warning on line 108 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L108

Added line #L108 was not covered by tests


@dataclasses.dataclass(frozen=True)
class LogicalOr:

Check warning on line 112 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L112

Added line #L112 was not covered by tests
"""A boolean column expression that is `True` if any of its operands are
`True`.
"""

operands: tuple[AbstractPredicate]

Check warning on line 117 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L117

Added line #L117 was not covered by tests


@dataclasses.dataclass(frozen=True)
class LogicalNot:

Check warning on line 121 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L121

Added line #L121 was not covered by tests
"""A boolean column expression that inverts its operand."""

operand: AbstractPredicate

Check warning on line 124 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L124

Added line #L124 was not covered by tests


@dataclasses.dataclass(frozen=True)
class IsNull:

Check warning on line 128 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L128

Added line #L128 was not covered by tests
"""A boolean column expression that tests whether its operand is NULL."""

operand: AbstractExpression

Check warning on line 131 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L131

Added line #L131 was not covered by tests


@dataclasses.dataclass(frozen=True)
class Comparison:

Check warning on line 135 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L135

Added line #L135 was not covered by tests
"""A boolean columns expression formed by comparing two non-boolean
expressions.
"""

a: AbstractExpression
b: AbstractExpression
operator: Literal["=", "!=", "<", ">", ">=", "<=", "overlaps"]

Check warning on line 142 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L140-L142

Added lines #L140 - L142 were not covered by tests


@dataclasses.dataclass(frozen=True)
class InContainer:

Check warning on line 146 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L146

Added line #L146 was not covered by tests
"""A boolean column expression that tests whether one expression is a
member of an explicit sequence of other expressions.
"""

member: AbstractExpression
container: tuple[AbstractExpression, ...]

Check warning on line 152 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L151-L152

Added lines #L151 - L152 were not covered by tests


@dataclasses.dataclass(frozen=True)
class InRange:

Check warning on line 156 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L156

Added line #L156 was not covered by tests
"""A boolean column expression that tests whether its expression is
included in an integer range.
"""

member: AbstractExpression
range: range

Check warning on line 162 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L161-L162

Added lines #L161 - L162 were not covered by tests


@dataclasses.dataclass(frozen=True)
class InRelation:

Check warning on line 166 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L166

Added line #L166 was not covered by tests
"""A boolean column expression that tests whether its expression is
included single-column projection of a relation.
This is primarily intended to be used on dataset ID columns, but it may
be useful for other columns as well.
"""

member: AbstractExpression
column: DimensionKeyColumnTag | DimensionRecordColumnTag | DatasetColumnTag
relation: AbstractRelation

Check warning on line 176 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L174-L176

Added lines #L174 - L176 were not covered by tests


@dataclasses.dataclass(frozen=True)
class StringPredicate:

Check warning on line 180 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L180

Added line #L180 was not covered by tests
"""A tag wrapper for boolean column expressions created by parsing a string
expression.
Remembering the original string is useful for error reporting.
"""

where: str
tree: AbstractPredicate

Check warning on line 188 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L187-L188

Added lines #L187 - L188 were not covered by tests


@dataclasses.dataclass(frozen=True)
class DataCoordinateConstraint:

Check warning on line 192 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L192

Added line #L192 was not covered by tests
"""A boolean column expression defined by interpreting data ID's key-value
pairs as a logical AND of equality constraints.
"""

data_coordinate: DataCoordinate

Check warning on line 197 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L197

Added line #L197 was not covered by tests


AbstractPredicate: TypeAlias = Union[

Check warning on line 200 in python/lsst/daf/butler/queries/abstract_expressions.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/queries/abstract_expressions.py#L200

Added line #L200 was not covered by tests
LogicalAnd,
LogicalOr,
LogicalNot,
IsNull,
Comparison,
InContainer,
InRange,
InRelation,
StringPredicate,
DataCoordinateConstraint,
]

0 comments on commit c53102e

Please sign in to comment.