Feat/sensor data source filtering#2083
Feat/sensor data source filtering#2083BelhsanHmida wants to merge 19 commits intocopilot/add-account-id-to-data-sourcefrom
Conversation
Context: - source filtering needs to support data sources linked to accounts Change: - add a reusable query criterion for filtering by source account ids
Context: - account-based source filtering needs to reach belief queries from sensor search helpers Change: - add source_account_ids to the time-series search interfaces and pass it into source criteria generation
Context: - the GET sensor data schema needs to expose account-linked source filtering Change: - add source_account_id to the sensor data schema and pass it into belief searches - define a query schema for documenting the actual GET parameters
Context: - the GET sensor data route was pointing Swagger at the wrong query schema Change: - switch the documented query schema to the GET-specific one - clarify source and source_account_id in the endpoint docs
Context: - the sensor data endpoint now accepts source_account_id for source filtering Change: - add a GET sensor data test that verifies account-linked sources are filtered correctly
Context: - the notation docs should match the actual source filtering supported by the endpoint Change: - document source and source_account_id for GET sensor data - clarify that source type filtering is not supported there
Context: - the endpoint behavior and Swagger exposure changed in a user-visible way Change: - add a changelog line for source account filtering and Swagger query parameter exposure
Context: - the sensor data schema and route docs changed the exposed GET query parameters Change: - regenerate the checked-in OpenAPI spec to match the current API surface
Documentation build overview
Show files changed (9 files in total): 📝 9 modified | ➕ 0 added | ➖ 0 deleted
|
…-data-source-filtering Signed-off-by: Mohamed Belhsan Hmida <149331360+BelhsanHmida@users.noreply.github.com>
There was a problem hiding this comment.
Preliminary review. Direction looks good.
The failing test in the pipeline is doing its job well, informing us of an oversight (that almost always trips us up when we extend schemas). In the long run, the UI schemas should be automatically derived from the API schemas. We should probably open an issue for that.
| class GetSensorDataQuerySchema(SensorDataTimingDescriptionSchema): | ||
| """Document the actual query parameters for GET /sensors/<id>/data.""" | ||
|
|
||
| resolution = DurationField(required=False) | ||
| source = SourceIdField(required=False) | ||
| source_account = AccountIdField(required=False, data_key="source_account_id") |
There was a problem hiding this comment.
At first glance it seems redundant to define this again, given GetSensorDataSchema already exists. Maybe this new class should become a mixin for GetSensorDataSchema.
There was a problem hiding this comment.
Good point. I refactored the shared GET sensor-data filter fields into a mixin, so GetSensorDataSchema and GetSensorDataQuerySchema now reuse the same resolution / source / account definitions instead of duplicating them.
| schema: SensorDataTimingDescriptionSchema | ||
| schema: GetSensorDataQuerySchema |
There was a problem hiding this comment.
Originally, this switch was only there to make OpenAPI expose the actual GET query parameters (resolution, source, and now account), because SensorDataTimingDescriptionSchema did not include them. At that point the downside was that GetSensorDataQuerySchema duplicated fields already defined elsewhere, which indeed made it look redundant.
With the latest refactor, the shared GET filter fields now live in a mixin that both GetSensorDataSchema and GetSensorDataQuerySchema inherit from. So the reason for keeping GetSensorDataQuerySchema is now just that the route needs a query-only schema for OpenAPI, while the actual field definitions stay shared and in sync.
| values = response.json["values"] | ||
| # The fixture also stores data from an accountless "Other source". | ||
| # Filtering by the user account should exclude those points. | ||
| assert all(a == b for a, b in zip(values, [91.5, 92.1, None, None])) |
There was a problem hiding this comment.
Note to self: check what the test is really doing here. Cross-reference where in the fixtures these values are defined.
…-data-source-filtering
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
2 participants
Description
source_account_idfiltering toGET /api/v3_0/sensors/<id>/dataso sensor data can be filtered by the account linked to a data source.sourceas the existing data-source ID filter for backward compatibility.documentation/changelog.rstLook & Feel
GET /api/v3_0/sensors/<id>/datanow showsresolution,source, andsource_account_idas query parameters.Example request:
How to test
Run:
pytest flexmeasures/api/v3_0/tests/test_sensor_data.pyOptionally inspect Swagger/OpenAPI and verify GET /api/v3_0/sensors//data shows:
Related Items
Sign-off