LIKE, IN, BETWEEN operators -- filter extension

[jamesfisher-gis]

Related Issue(s):

• Flexible full-text search #169

Description:
Adds the CQL2 LIKE search operator utilizing the Elasticsearch wildcard query. The code supports CQL wildcard characters ('_' and '%'), Elasticsearch wildcard characters ('?' and '*'), and escape characters in queries.

PR Checklist:

• [x ] Code is formatted and linted (run pre-commit run --all-files)
• [x ] Tests pass (run make test)
• [x ] Documentation has been updated to reflect changes, if applicable
• Changes are added to the changelog

[gadomski]

Seems fine to me, but I'd like to get @philvarner or @jonhealy1 's eyes on it as well as I don't have great context on ES.

[philvarner]

this shouldn't be advertised unless IN and BETWEEN are also implemented.

[jamesfisher-gis]

Okay, I have implemented IN and BETWEEN in the latest commit.

The conformance class endpoint above aligns with what STAC Browser expects. The Filter extension README has the conformance class endpoint as http://www.opengis.net/spec/cql2/1.0/conf/advanced-comparison-operators. Which should be used? I get a 404 response from both...

I followed this example which has the BETWEEN bounds being passed as an array.

{
  "filter-lang": "cql2-json",
  "filter": {
    "op": "between",
    "args": [
      { "property": "eo:cloud_cover" },
      [ 0, 50 ]
    ]
  }
}

This required adjustments to the Clause filter class to allow lists and some typing checks in IN and BETWEEN

class Clause(BaseModel):
    """Filter extension clause."""

    op: Union[LogicalOp, ComparisonOp, AdvancedComparisonOp, SpatialIntersectsOp]
    args: List[Union[Arg, List[Arg]]]

elif self.op == AdvancedComparisonOp.between:
            if not isinstance(self.args[1], List):
                raise RuntimeError(f"Arg {self.args[1]} is not a list")
            return {
                "range": {
                    to_es(self.args[0]): {
                        "gte": to_es(self.args[1][0]),
                        "lte": to_es(self.args[1][1]),
                    }
                }
            }
elif self.op == AdvancedComparisonOp._in:
    if not isinstance(self.args[1], List):
        raise RuntimeError(f"Arg {self.args[1]} is not a list")
    return {
        "terms": {to_es(self.args[0]): [to_es(arg) for arg in self.args[1]]}
    }

[philvarner]

Okay, I have implemented IN and BETWEEN in the latest commit.

nice, will review.

The conformance class endpoint above aligns with what STAC Browser expects. The Filter extension README has the conformance class endpoint as http://www.opengis.net/spec/cql2/1.0/conf/advanced-comparison-operators. Which should be used? I get a 404 response from both...

Ah, that seems to be a bug in STAC Browser - they should be the urls with conf in them, not req. They aren't resolvable as a URL -- many of the OGC URL-like conformance classes aren't.

I followed this example which has the BETWEEN bounds being passed as an array.

This has been changed in newer versions of CQL2 (or v1.0.0-beta.5 had bad examples?). The current version of Filter Extension is here, specifically BETWEEN: https://github.com/stac-api-extensions/filter/blob/main/README.md#example-11-using-between
I think you'll probably need to make some changes to work with this instead of the beta.5 version.

[jamesfisher-gis]

Okay. Thank you, Phil.

Updated BETWEEN syntax and the conformance class endpoint in e7330dc

[philvarner]

I think this should be case sensitive

[jamesfisher-gis]

Do you mean that case_insensitive should be set to false?

[philvarner]

yes, since AFAIK LIKE is a case sensitive operation (e.g., b% matches ba but not Ba

[jamesfisher-gis]

Got it. Updated in e7330dc

[jonhealy1]

Looks great @jamesfisher-gis