Making reporters/data generators easier to configure, rerun, and schedule from FlexMeasures

[BelhsanHmida]

As a starting point for discussion, I would like to explore how reporters and other data generators could become easier to use operationally, especially without having to maintain YAML files, shell scripts, and external cron entries for recurring reports.

Context

This came up while working on a building-consumption reporter. The actual business logic is simple:

building_consumption =
  total_house_power
  + pv_power
  - ev_charger_1_power
  - ev_charger_2_power

But running it in practice currently means managing several pieces outside the UI:

• reporter config YAML
• reporter parameter YAML
• input/output sensor IDs
• source filters
• shell scripts
• cron entries
• operational logs

That works, but it feels like the kind of workflow FlexMeasures could eventually make more user-friendly.

Related context:

• #829, about the PandasReporter / TimelyBeliefsReporter direction
• #1319, about adding more reporter examples
• Discussion #662, especially the ideas around reporter config, data sources, and when a new DataSource should be created

After discussion, I see three related problems.

1. Data Generator Templates

Similar to how we have structured ways of defining assets, it could be useful to have templates or recipes for data generators.

For reporters, this could mean named templates such as:

• daily energy total
• ratio / efficiency between two sensors
• clipped values within a valid range
• building consumption from several power sensors
• profit/loss report

These templates could hide most of the raw PandasReporter machinery from normal users, while still keeping the generic reporter infrastructure available for advanced use cases.

2. Creating and Rerunning Data Generators

Reporter config is already saved on DataSource attributes when using --save-config, so we already have part of the mechanism needed to rerun a reporter from a stored source.

The open question is whether DataSource should remain mainly a provenance object, or whether we need a separate user-facing concept such as a ReportDefinition / DataGeneratorDefinition.

A possible model could be:

• ReportDefinition: editable recipe/template chosen by the user
• ReportRun: one execution with resolved start/end/resolution and job status
• ReportSchedule: optional recurrence rule
• DataSource: immutable-ish provenance for the data that was produced

This would avoid mutating historical provenance when a user edits a report definition.

3. Scheduling from FlexMeasures Instead of External Cron

Once a user can create a data generator definition, the next step is scheduling it.

Today, one analogy is:

flexmeasures add report ...

plus an external cron line.

A more integrated version could be a CLI command that is run once, for example conceptually:

flexmeasures add report-schedule ...

Or a UI flow from the sensor or asset page:

• choose/create report
• configure input sensors and output sensor
• define reporting window, e.g. "previous local day"
• define recurrence, e.g. "every morning at 06:00"
• enable/disable schedule
• view last run, next run, failures, and linked jobs

The sensor page already has a useful precedent with the "Trigger forecast" UI: configure a small set of fields, queue a job, poll status, and show feedback. Reports could reuse that interaction pattern, although some reporters are multi-input or asset-level, so a dedicated Reports page may also be needed.

Design Questions

• Should reports be attached primarily to output sensors, input sensors, assets, accounts, or a dedicated Reports page?
• Should the first UI support arbitrary PandasReporter configs, or only curated templates?
• Which fields should define a new DataSource: reporter class, config, input sensors, input sources, output sensors, resolution?
• Can a report be rerun from a DataSource alone, or do we need a separate editable definition?
• Should recurring reports overwrite existing beliefs, append new beliefs, or skip windows that already have output?
• How should missed runs after downtime be handled: skip, backfill, or run latest only?
• What timezone owns the schedule: output sensor, account, user, or explicit schedule timezone?
• How should DST be handled for "daily" reports?
• Who is allowed to create report definitions and schedules?
• Should scheduling use RQ's current scheduling/cron support, rq-scheduler, or a DB-backed recurrence table that enqueues RQ jobs?