Skip to content

feat: support source filters in sensor references for flex-model and flex-context#2209

Merged
Flix6x merged 21 commits into
mainfrom
copilot/revise-schemas-and-update-docs
Jun 2, 2026
Merged

feat: support source filters in sensor references for flex-model and flex-context#2209
Flix6x merged 21 commits into
mainfrom
copilot/revise-schemas-and-update-docs

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 29, 2026

Sensor references in flex-model and flex-context fields can now carry source filters (source-types, exclude-source-types, sources, source-account) to narrow which beliefs are used when querying a sensor. This enables use cases like rescheduling a PV asset without separate sensors for forecasts vs. schedules — source filtering masks the unwanted beliefs on the same sensor.

Source filter support

  • Added source_account field to SensorReference; all four filter fields are now forwarded to TimedBelief.search in get_series_from_quantity_or_sensor
  • VariableQuantityField._serialize now round-trips source filter fields (previously dropped them)
  • _resolve_soc_at_start_from_sensor propagates source filters when inferring SoC from a filtered sensor reference
  • storage-efficiency in the storage scheduler now accepts SensorReference (filtered sensor) in addition to plain Sensor

Validation (source filters rejected where meaningless)

  • aggregate-power in FlexContextSchema: rejects SensorReference — this field identifies a meter, not a belief query
  • consumption and production in StorageFlexModelSchema / DBStorageFlexModelSchema: reject source filters — these fields designate output sensors to write the schedule to, not belief queries
// ✅ Valid: filter SoC beliefs by source account
{
  "state-of-charge": {
    "sensor": 42,
    "source-account": [7]
  }
}

// ❌ Invalid: consumption/production are write targets, not belief queries
{
  "consumption": {
    "sensor": 42,
    "source-types": ["scheduler"]
  }
}

A module-level constant _SENSOR_REFERENCE_SOURCE_FILTER_KEYS is shared across all four source-filter validators to keep the key list in one place.

Copilot AI and others added 3 commits May 29, 2026 11:39
…ext (issue #1807)

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
…rence

Add 8 tests covering:
- backward compatibility: plain {sensor: id} still yields a Sensor
- source-types deserialization → SensorReference with source_types set
- exclude-source-types deserialization → SensorReference with exclude_source_types set
- sources list deserialization → SensorReference with DataSource objects resolved
- 4 parametrized invalid-payload cases (ValidationError expected)

Fix: test_sensor_reference_with_sources adds db fixture and calls
db.session.flush() to ensure DataSources created by setup_sources
(which never commits) get DB-assigned primary keys before the test
accesses seita_source.id.

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
Copilot AI and others added 5 commits May 29, 2026 13:21
- Add `source_account: list[Account] | None` field to the SensorReference dataclass
- Import Account from flexmeasures.data.models.user and AccountIdField from
  flexmeasures.data.schemas.account in sensors.py
- Add `source_account` field (data_key="source-account") to SensorReferenceSchema
- Extend `_SOURCE_FILTER_KEYS` with "source-account" so the backward-compat
  plain-Sensor path is preserved when no filter keys are present
- Extract source-filter deserialization into `_deserialize_source_filters` helper
  to keep `_deserialize_dict` within the C901 complexity limit (≤13)

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
…to TimedBelief.search

Pass `source_account_ids` derived from `variable_quantity.source_account` (a list of
Account objects) when querying beliefs for a SensorReference in
`get_series_from_quantity_or_sensor`.

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
…at_start_from_sensor

- Widen the `state_of_charge_sensor` parameter type to accept Sensor | SensorReference
- Unpack SensorReference at the top of the method: extract the underlying Sensor
  and all four source filter attributes (source_types, exclude_source_types, sources,
  source_account_ids)
- Forward these filters to `soc_sensor.search_beliefs(...)`
- Update the SensorReference caller in `_resolve_soc_at_start_from_state_of_charge`
  to pass the SensorReference directly instead of `state_of_charge.sensor`, so source
  filters are no longer silently discarded

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
…source_filters

Use `list[DataSource] | None` and `list[Account] | None` instead of bare `list | None`
in the return type tuple, making the contract self-documenting.

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
- Add schema deserialization test for source-account in test_sensor.py
- Add invalid-input test case for source-account (non-list) in test_sensor.py
- Add integration test for source-account filtering in test_utils.py
- Fix circular import: move SensorReference import in validation_utils.py
  inside the function body to break the cycle introduced by sensors.py
  importing AccountIdField from account.py

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
Copilot finished work on behalf of Flix6x May 29, 2026 14:05
Copilot AI requested a review from Flix6x May 29, 2026 14:05
Flix6x added 3 commits May 29, 2026 17:35
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
@Flix6x Flix6x added this to the 1.0.0 milestone May 29, 2026
@Flix6x Flix6x marked this pull request as ready for review May 29, 2026 15:45
Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
Copilot AI changed the title feat: support source filters in sensor references for flex-model and flex-context fix: move fresh_db tests to separate module to prevent CI hang May 29, 2026
Copilot finished work on behalf of Flix6x May 29, 2026 17:51
Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
Copilot AI changed the title fix: move fresh_db tests to separate module to prevent CI hang fix: prevent CI hang from mixed fresh_db/db fixtures in same test module May 29, 2026
Copilot finished work on behalf of Flix6x May 29, 2026 21:59
@Flix6x Flix6x changed the title fix: prevent CI hang from mixed fresh_db/db fixtures in same test module feat: support source filters in sensor references for flex-model and flex-context May 29, 2026
@Flix6x Flix6x requested a review from BelhsanHmida May 30, 2026 11:34
Flix6x added 2 commits June 2, 2026 15:14
…s-and-update-docs

# Conflicts:
#	flexmeasures/data/schemas/scheduling/storage.py
#	flexmeasures/data/schemas/tests/test_sensor.py
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Copy link
Copy Markdown
Contributor

@BelhsanHmida BelhsanHmida left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot please take another pass on this PR. The core source-filter lookup path is useful, but a few PR-introduced behaviors still need to be handled:

  1. Fix state-of-charge filtering when soc-at-start is inferred.
    ensure_soc_at_start() runs before StorageFlexModelSchema turns raw JSON into SensorReference, so _resolve_soc_at_start_from_state_of_charge() currently drops the new filter fields in the raw dict branch. Please preserve source-types, exclude-source-types, sources, and source-account there.

  2. Preserve filters when serializing SensorReference.
    VariableQuantityField._serialize() currently dumps a filtered reference as only {"sensor": id}. Please include the filter fields so source-filtered references can round-trip.

  3. Handle filtered storage-efficiency.
    With this PR, filtered storage-efficiency deserializes to SensorReference, but storage runtime still only treats plain Sensor as sensor-backed efficiency. Please make this path work and keep the resolution conversion correct.

  4. Clarify/fix aggregate-power.
    aggregate-power is an output target, so source filters probably should not be accepted there. If the intended behavior is to allow it, please unwrap to the underlying sensor before using it as an output key. Otherwise, reject filtered references for this field.

Please add focused regression tests for these paths, especially the soc-at-start inference case with two beliefs on the same SoC sensor from different sources.

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Copilot AI and others added 2 commits June 2, 2026 17:02
…ld._serialize

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
Copilot finished work on behalf of Flix6x June 2, 2026 17:04
Copy link
Copy Markdown
Member

@Flix6x Flix6x left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@BelhsanHmida thanks again for the review and revisions. I code reviewed everything once more, except for the tests. Maybe you can take a final look at those, to make sure they have proper coverage (I sometimes still spot copilot creating tests that cover trivial cases only).

Comment thread flexmeasures/data/schemas/sensors.py
@BelhsanHmida
Copy link
Copy Markdown
Contributor

@BelhsanHmida thanks again for the review and revisions. I code reviewed everything once more, except for the tests. Maybe you can take a final look at those, to make sure they have proper coverage (I sometimes still spot copilot creating tests that cover trivial cases only).

Looks good to me now. I re-checked the test coverage too, and the important filter paths are covered with non-trivial cases.

@Flix6x Flix6x merged commit 56cfa67 into main Jun 2, 2026
12 checks passed
@Flix6x Flix6x deleted the copilot/revise-schemas-and-update-docs branch June 2, 2026 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support source filter in sensor references in flex-model and flex-context

3 participants