REST: get things of type

[maggu2810]

There is currently no method to get all things of a special type only (or I didn't find them).
We are using setups that "get all things" call results into a > 2 MiB response.
Sometimes we need to know only things of a given type first.
I can create that interface in our custom product or you could extend the thing resource to support this request officially.
WDYT?

[sjsf]

I don't see a reason not to put it here. "go" from my side.

[kaikreuzer]

We didn't go into detailed filtering mechanisms on any rest resource from the start, but the it was clear that this would be required sooner or later.

When adding those now for certain cases, we should make sure to find a good general scheme that we can apply to all different kinds of resources. Ideally, we should have flexible query parameters, maybe also support substring matching and somehow cleverly feed that into a stream-based get method on the registries.

[maggu2810]

The GET method does not support data in the content, so we could use the URL for the filter information. Correct?

[htreu]

Correct. We could provide something like /rest/things?thing-type=hue:lamp which will make thing-type a pre defined query parameter for the things resource. The alternative is to encode the whole filter into a single parameter like /rest/things?filter=thing-type%3Dhue%3Alamp%3Blabel%3DMy%20Hue%20Lamp which would be more flexible in a way that we create filters dynamically from the filter string. This way nearly every attribute or property of a resource may be used as a filter.

[maggu2810]

If we use a single parameter filter string we can do a lot of stuff.
Do we want to consider an expression filter like the LDAP expression, so allow "&" (and) and "|" (or) and "!" (not) and interpret the values as regular expression?

So, e.g.

(&(thing-type=hue:*),(!(label=test))

It would allow doing a lot of this, but perhaps this would be to complex.

[maggu2810]

We would need a implementation that could be reused e.g.

boolean testLdapExpression(final String expression, final Map<String, String> map)

The map is filled with key value pairs and is under control of the caller.
The thing resource could handle it this way (pseudo code):

for (final ThingDTO thing : getAllThings()) {
    final Map<String,String> props = new HashMap<>();
    props.put("thing-type", thing.thingType);
    props.put("label", thing.label);
    if (testLdapExpression(filter, props) {
        result.add(thing);
    }
}

another implementation will put other properties...

[kaikreuzer]

I am personally not a huge fan of LDAP filters and imho this is also rather unusual in the JS/REST world - but maybe we should ask some web experts on that :-)

[htreu]

What I like better on the explicit query parameter name (i.e. /rest/things?thing-type=hue:lamp) is the fact that the swagger documentation will give you direct access to all available parameters. In addition the URI is more human readable and I suspect we will not have that many query params.

[maggu2810]

I "like" the query parameter, too as it is more readable.

we should make sure to find a good general scheme that we can apply to all different kinds of resources. Ideally, we should have flexible query parameters, maybe also support substring matching

The query parameter seems to be not very flexible. At least it is not clear to me how to differ between a "logical and" and a "logical or". Perhaps we can agree that the parameters are always read as logical and / or, or it could be at least a global combination defined.

[sjsf]

In that respect the query parameters look like a dead-end. I mean, the don't need to be extremely powerful, but having both "and" and "or" sounds feasible.

I personally also don't really LDAP expressions, as they are hard to read & write for humans, despite the fact that they are incredibly simple to parse for computers which would be a big plus on the positive side.

However, it also somehow feels wrong to invent our own. If it's not gonna be LDAP, how about using some other popular one, e.g. the filter language from OData? It's basically what anybody would write in code in almost every language, except that the special characters are replaced by abbreviations which are already familiar from e.g. HTML.

[htreu]

I agree on the fact that specific query parameters may become too basic very quick. Going with the OData filter language looks promissing. Although we have to separate the OData protocol with lots of other resource defining specs from just the query/filter part.
Apache Olingo does provide an implementation but from the first look it seems to be bloated when we just want the filter stuff to work. And libs are not broken into specific aspects of OData but client/server only...
Nevertheless its a good proposal and we should further investigate implementation possibilities.
Maybe this is a "help wanted" topic? wdyt?

[maggu2810]

If it's not gonna be LDAP, how about using some other popular one

Why not using the RFC 1960-based filter that is used for ServiceReference matches already by the OSGi framework. I assume that filters are (at least for OSGi bundle developers) popular (but hey isn't that LDAP 😉).

• BundleContext#createFilter
• Filter

On the OSGi container side it would come without any overhead (I assume that nearly all products are using components).
And "too complicated" shouldn't be a problem. The UI don't need to support the building of complicated filters.

final Filter filter = bundleContext.createFilter(userFilter);
final Stream<EnrichedThingDTO> thingStream = thingRegistry.stream().filter(thing -> {
    final Dictionary<String, Object> dict = new Hashtable<>();
    dict.put("thingTypeUID", thing.getThingTypeUID().getAsString());
    dict.put("label", thing.getLabel());
    dict.put("location", thing.getLocation());
    return filter.match(dict);
}).map(t -> convertToEnrichedThingDTO(t, locale)).distinct();

[maggu2810]

I have added a simple demonstration for my suggestion, see diff

You can use e.g.

• http://127.0.0.1:8080/rest/things/filtered/(thingTypeUID=magic:*) to get all things using a thing type UID of the magic bundle
• http://127.0.0.1:8080/rest/things/filtered/(!(thingTypeUID=magic:*)) to get all things using a thing type UID not from the magic bundle

I used that exmaples to demonstrate that wildcards could be used, too.

IMHO which complex filters (AND, OR, NOT, wildcards, ...) are offered by the UI should be UI specific. At least the REST API itself, that could be used by "whatever" should be generic and this is a very simple solution as in an OSGi runtime all that magic filter handling is already present.

[RobWin]

Hello,

please keep in mind that the LDAP-filter notation contains URL-unsafe characters which must be escaped by clients.
There is another interesting notation called RSQL (Rest Query Language) which is based on FIQL (Feed Item Query Language) – an URI-friendly syntax for expressing filters across the entries in an Atom Feed. RSQL provides a friendlier syntax for logical operators and some of the comparison operators.

Apache CXF provides a search functionality based on FIQL. See http://cxf.apache.org/docs/jax-rs-search.html#JAX-RSSearch-FIQL

Article: https://jaxenter.com/tutorial-smarter-search-with-fiql-and-apache-cxf-106000.html