feat: support source filters in sensor references for flex-model and flex-context#2209
Conversation
…ext (issue #1807) Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
…rence
Add 8 tests covering:
- backward compatibility: plain {sensor: id} still yields a Sensor
- source-types deserialization → SensorReference with source_types set
- exclude-source-types deserialization → SensorReference with exclude_source_types set
- sources list deserialization → SensorReference with DataSource objects resolved
- 4 parametrized invalid-payload cases (ValidationError expected)
Fix: test_sensor_reference_with_sources adds db fixture and calls
db.session.flush() to ensure DataSources created by setup_sources
(which never commits) get DB-assigned primary keys before the test
accesses seita_source.id.
Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
- Add `source_account: list[Account] | None` field to the SensorReference dataclass - Import Account from flexmeasures.data.models.user and AccountIdField from flexmeasures.data.schemas.account in sensors.py - Add `source_account` field (data_key="source-account") to SensorReferenceSchema - Extend `_SOURCE_FILTER_KEYS` with "source-account" so the backward-compat plain-Sensor path is preserved when no filter keys are present - Extract source-filter deserialization into `_deserialize_source_filters` helper to keep `_deserialize_dict` within the C901 complexity limit (≤13) Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
…to TimedBelief.search Pass `source_account_ids` derived from `variable_quantity.source_account` (a list of Account objects) when querying beliefs for a SensorReference in `get_series_from_quantity_or_sensor`. Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
…at_start_from_sensor - Widen the `state_of_charge_sensor` parameter type to accept Sensor | SensorReference - Unpack SensorReference at the top of the method: extract the underlying Sensor and all four source filter attributes (source_types, exclude_source_types, sources, source_account_ids) - Forward these filters to `soc_sensor.search_beliefs(...)` - Update the SensorReference caller in `_resolve_soc_at_start_from_state_of_charge` to pass the SensorReference directly instead of `state_of_charge.sensor`, so source filters are no longer silently discarded Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
…source_filters Use `list[DataSource] | None` and `list[Account] | None` instead of bare `list | None` in the return type tuple, making the contract self-documenting. Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
- Add schema deserialization test for source-account in test_sensor.py - Add invalid-input test case for source-account (non-list) in test_sensor.py - Add integration test for source-account filtering in test_utils.py - Fix circular import: move SensorReference import in validation_utils.py inside the function body to break the cycle introduced by sensors.py importing AccountIdField from account.py Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
…s-and-update-docs # Conflicts: # flexmeasures/data/schemas/scheduling/storage.py # flexmeasures/data/schemas/tests/test_sensor.py
Signed-off-by: F.N. Claessen <claessen@seita.nl>
BelhsanHmida
left a comment
There was a problem hiding this comment.
@copilot please take another pass on this PR. The core source-filter lookup path is useful, but a few PR-introduced behaviors still need to be handled:
-
Fix
state-of-chargefiltering whensoc-at-startis inferred.
ensure_soc_at_start()runs beforeStorageFlexModelSchematurns raw JSON intoSensorReference, so_resolve_soc_at_start_from_state_of_charge()currently drops the new filter fields in the raw dict branch. Please preservesource-types,exclude-source-types,sources, andsource-accountthere. -
Preserve filters when serializing
SensorReference.
VariableQuantityField._serialize()currently dumps a filtered reference as only{"sensor": id}. Please include the filter fields so source-filtered references can round-trip. -
Handle filtered
storage-efficiency.
With this PR, filteredstorage-efficiencydeserializes toSensorReference, but storage runtime still only treats plainSensoras sensor-backed efficiency. Please make this path work and keep the resolution conversion correct. -
Clarify/fix
aggregate-power.
aggregate-poweris an output target, so source filters probably should not be accepted there. If the intended behavior is to allow it, please unwrap to the underlying sensor before using it as an output key. Otherwise, reject filtered references for this field.
Please add focused regression tests for these paths, especially the soc-at-start inference case with two beliefs on the same SoC sensor from different sources.
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
…ld._serialize Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
Flix6x
left a comment
There was a problem hiding this comment.
@BelhsanHmida thanks again for the review and revisions. I code reviewed everything once more, except for the tests. Maybe you can take a final look at those, to make sure they have proper coverage (I sometimes still spot copilot creating tests that cover trivial cases only).
Looks good to me now. I re-checked the test coverage too, and the important filter paths are covered with non-trivial cases. |
Sensor references in flex-model and flex-context fields can now carry source filters (
source-types,exclude-source-types,sources,source-account) to narrow which beliefs are used when querying a sensor. This enables use cases like rescheduling a PV asset without separate sensors for forecasts vs. schedules — source filtering masks the unwanted beliefs on the same sensor.Source filter support
source_accountfield toSensorReference; all four filter fields are now forwarded toTimedBelief.searchinget_series_from_quantity_or_sensorVariableQuantityField._serializenow round-trips source filter fields (previously dropped them)_resolve_soc_at_start_from_sensorpropagates source filters when inferring SoC from a filtered sensor referencestorage-efficiencyin the storage scheduler now acceptsSensorReference(filtered sensor) in addition to plainSensorValidation (source filters rejected where meaningless)
aggregate-powerinFlexContextSchema: rejectsSensorReference— this field identifies a meter, not a belief queryconsumptionandproductioninStorageFlexModelSchema/DBStorageFlexModelSchema: reject source filters — these fields designate output sensors to write the schedule to, not belief queriesA module-level constant
_SENSOR_REFERENCE_SOURCE_FILTER_KEYSis shared across all four source-filter validators to keep the key list in one place.