[sensorthings] Handle array results from Observations #57583
Conversation
|
@nyalldawson and what about the 'json' case as a result? Not sure from the code if this is handled too. (could be a last resort to just create a string from it?) |
|
It's messy -- we can only have "json" fields which contain map or list values, not single objects. So we'd have to use an array type for this field always. Ultimately it's a limitation in qgis vector data provider API -- we'd ideally have a "any" field type which means the field can contain data of any type. |
|
@nyalldawson Thanks! I build this PR and did some test. Below some observations (sorry for the length/messiness):
Some issues/ideas around use when I tried to retrieve FeaturesOfInterest from the service (main goal: retrieve points with values of a given type of measurement, to be able to use the Temporal Filter to 'see' the values on the map in time change):
Would become:
The 'limit' is set to 100 in the Expansions dialog, while (I think) for Datastreams and Observer Properties, a better default value would be lower? (not sure if this would speed up retrieval though):
I'm trying to 'create' a filter for my main goal: retrieving (latest or a recent set) of measurements on my map which I can control with the Temporal Controller OR view using Plotly... Ui thingie: I'm not able to see the (long, because of expanding) Field (names) easy even when I make the dialog huge:
Issue? I see the 'Expansions' in the main dialog is not cleaned when changing the (main) Entity type, while I think that should be done(?), as the Expansion is never valid anymore if you change Entity type? Trying to fetch only Observations of NO2: but I end up with a query part in the url's with By crafting a good url by hand, I still get a timeout/500 though: https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1/FeaturesOfInterest?$top=200&$count=false&$filter=(feature/type eq 'Point' or feature/geometry/type eq 'Point') and (geo.intersects(feature, geography'Polygon ((4.96998038694149358 51.98031302864895054, 5.32698782307856611 51.98031302864895054, 5.32698782307856611 52.18820000000000192, 4.96998038694149358 52.18820000000000192, 4.96998038694149358 51.98031302864895054))')) and (Observations/Datastream/ObservedProperty/name eq 'NO2')&$expand=Observations($orderby=phenomenonTime desc;$top=100;$expand=Datastream($top=2;$expand=ObservedProperty($top=2))) @nyalldawson I thought to Retrieve Things on their Location, but... I cannot 'expand' the Locations of a Thing?? Should it not be possible to Expand Locations? I'm really eager to know if my 'goal' is doable @hylkevds do you have an idea? Of is the STA model really about retrieving data step by step (so first Things/Location and then one by one get data from a Thing/Datastream)? I found an earlier question of me FraunhoferIOSB/FROST-Server#1107 in which I retrieve geojson (working, but slow) : https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1/Datastreams?$count=true&$filter=ObservedProperty/name eq 'SO2'&$expand=ObservedProperty($select=name,description,@iot.id),Observations($select=result,phenomenonTime,@iot.id;$orderby=phenomenonTime%20desc;$top=1),Thing/Locations&$resultFormat=GeoJSON&$top=2000 |
|
Other things I see: in the attribute table (of a STA layer) the tooltips all show 'Name \nNULL'
And another observation: If you fetch Locations, with an expanded Thing/Datastream/Observation, instead of the 'phenemenonTime' there are properties: phenemenonTimeStart and phenemenonTimeEnd:
|
|
@rduivenvoorde
I'll put these into #57574
This is already addressed in #57574
I'll fix in #57574
Already addressed in #57574
This one should be possible after #57574, where per-entity filters are introduced
Same here. I suspect it's one of those rough edges in the sensor things specifications/server implementations, where we can easily craft a request which results in a non-indexed, very expensive database lookup on the server. It's something we should handle by filing bugs with the server, and see if future versions of the server software can add indexes/optimisations to cover these use cases.
That's because we currently have a constraint in place that the entity with geometry must be the base level entity, not an expansion.
That's not specific to SensorThings, but it's improved in #57621 . It's supposed to show the field type + the default value, so the "NULL" here is the default field value. (You'll see the same with any vector layer.)
This is because QGIS doesn't have a field type for "date time range", so we have to split the ranges out to two separate fields |
|
The air quality service is currently optimised for querying starting at the Thing/Location side, not at the Feature side. The reason for that is that the Datastreams are attached to Things, and not to Features. When starting at the Features side, each feature has one big bag of observations, not sorted into separate time-series per property. The expensive part in that query: is filtering the Features based on the ObservedProperties of the Observations. That is very inefficient, because in this service, each Feature has very many Observations. So in this case it's better to start from the Thing/Location side. We're changing this model in Version 2, so that a Feature can be linked to Datastreams too, and you'd only get the Observations of a Feature for those Features that are Samples, and that thus don't have that many Observations. Also note, that although the above query filters on only those Features that have NO2 Observations, in the expand, all Observations would be returned, not just those for NO2. As for the multiple Locations of a Thing: Yes, a Thing can have multiple Locations, as long as those are different representations of the same real-world location. We also have a demo service that has all the European NUTS regions in 5 different resolutions, so each Thing has 5 Locations: |
|
Thanks for the new PR will test it later
But an Observation does not have a date time range, but only a phenemenonTime I thought?
Argh.. now I see: https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1/Observations(522318087) phenemenonTime has "2017-12-31T23:00:00Z/2018-01-01T23:00:00Z" So using such data with the Temporal Controller would mean:
|
|
Exactly, a TM_Object can be either a TM_Instant or a TM_Period. |
@hylkevds thanks for the explanation! It's just that I'm trying to fit STA on the way QGIS is able to handle 'temporal features' using the Temporal Controller: (and this is also my use-case: show (on a map, the changes of) the hourly measurements of a certain quantity/ObservedProperty of static sensors, but ALSO of sensors attached to driving vehicles) Actually it means that you'd want a feature build from one observation PLUS (at least) the Thing AND it's location:A
Or of the driving sensor:
|
Fixes #57577