481 extend documentation and toy tutorial with scheduling within the context of a building with PV

[Flix6x]

Most of this PR ended up being about making it technically feasible to create the envisioned chart design in #481, and also making more sensible use of the color encoding and data source attribute. Here's roughly what I did:

Expand tutorial:

• Expand toy tutorial with creation of solar sensor and solar forecasts from CSV
• Expand toy tutorial with new schedule that takes into account solar
• Check whether scheduler now takes into account the limited headroom (yay, the new scheduling feature worked without having to touch the scheduler logic)
• Add a chart showing the new schedule [Moved to Expand toy tutorial with a sensor that reports available headroom #584]

Try to create the envisioned chart design:

• Allow nesting sensors in sensors_to_show, to denote that multiple sensors should be shown together, and deal with all of its implications (axis labels, use of colors, etc.)
• Visually distinguish forecasts, schedules and measurements by source type rather than by belief horizon (incl. a simple db migration)
• Add a CLI command to create a new DataSource (so that forecasts are visually presented as originating from a forecaster)
• Add a 'capacity' or 'headroom' sensor whose data is derived from an Asset attribute (capacity) in combination with sensor data (in this case that would only be the solar sensor). I feel that showing the headroom (like in the envisioned chart design) is what makes the example click. I don't have this (yet), but I feel we should first have a work session on derived sensor data (and/or KPIs) before I continue. [Moved to Expand toy tutorial with a sensor that reports available headroom #584]

Current status of the chart:

• For the toy tutorial:

• For V2G use case:

I also encountered a number of fixes that we should definitely cherry-pick for the 0.12 release (see #545). Specifically:

• cf16ecf [already merged]
• 7dcb001 [already merged]
• 141a794 [already merged]

Needs SeitaBV/timely-beliefs#122.
Closes #481.

[coveralls]

Pull Request Test Coverage Report for Build 4677953700

• 59 of 164 (35.98%) changed or added relevant lines in 12 files are covered.
• 6 unchanged lines in 3 files lost coverage.
• Overall coverage decreased (-0.3%) to 64.5%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
flexmeasures/api/v3_0/assets.py	1	2	50.0%
flexmeasures/data/schemas/generic_assets.py	18	19	94.74%
flexmeasures/utils/flexmeasures_inflection.py	1	2	50.0%
flexmeasures/data/services/data_sources.py	2	4	50.0%
flexmeasures/data/models/data_sources.py	0	3	0.0%
flexmeasures/cli/data_show.py	0	4	0.0%
flexmeasures/data/models/generic_assets.py	3	12	25.0%
flexmeasures/cli/data_add.py	9	24	37.5%
flexmeasures/data/services/utils.py	8	30	26.67%
flexmeasures/data/models/charts/belief_charts.py	8	55	14.55%

Files with Coverage Reduction	New Missed Lines	%
flexmeasures/cli/data_add.py	1	33.27%
flexmeasures/data/models/generic_assets.py	1	45.52%
flexmeasures/data/models/charts/belief_charts.py	4	11.21%

Totals
Change from base Build 4677372721:	-0.3%
Covered Lines:	6940
Relevant Lines:	10012

---

💛 - Coveralls

[victorgarcia98]

Running the tutorial with a fresh DB creates different assets to those shown in the tutorial.

Creating Toy Account

Input:

flexmeasures add toy-account --kind battery

Output:

  Created <GenericAsset None: 'toy-battery' (battery)>
  Created discharging
  Adding transmission zone type ...
  Adding NL transmission zone ...
  Created day-ahead prices
  Created <GenericAsset None: 'toy-solar' (solar)>
  Created production
  Toy account Toy Account with user toy-user@flexmeasures.io created successfully. You might want to run `flexmeasures show account --id 1`
  The sensor recording battery discharging is discharging (ID: 2).
  The sensor recording day-ahead prices is day-ahead prices (ID: 3).
  The sensor recording solar forecasts is production (ID: 4).

Showing account

Input:

flexmeasures show account --id 1

Output

===========================
Account Toy Account (ID: 1)
===========================

Account has no roles.

All users:
 
  ID  Name      Email                     Last Login    Roles
----  --------  ------------------------  ------------  -------------
   1  toy-user  toy-user@flexmeasures.io  None          account-admin

All assets:
 
  ID  Name         Type     Location
----  -----------  -------  -----------------
   1  toy-battery  battery  (52.374, 4.88969)
   3  toy-solar    solar    (52.374, 4.88969)

In the tutorial, here appears toy-building, as well.

[nhoening]

I am still getting the following error when I add a schedule (on different laptop, using a fresh database):

sqlalchemy.exc.InvalidRequestError: Can't attach instance <Sensor at 0x7f6257982550>; another instance with key (<class 'flexmeasures.data.models.time_series.Sensor'>, (1,), None) is already present in this session.

I wonder if you guys have also tried to compute both types of schedules in one run through the tutorial? With --inflexible-device-sensor and without? Maybe you're only testing the former option right now?

It could also be that this behavior is not caused by this PR (and we can merge it), but by some other artifact we need to address some day in another PR. But this is a good opportunity to describe it well.

I can now explain a bit more what causes this for me:

• If I do either way first, the other way will fail, like described above.
• If I re-do the same way, it does not fail.
• If I delete the computed belief data (delete from timed_belief where sensor_id = 1;), then computing the schedule in whichever of the two ways will work again. So it has to do with existing data in the database.
• The complaint is always about sensor 1, in both cases.

[nhoening]

As my previous research showed, an error occurs if there is previous data. It does not seem to matter whether the extra sensor is used in computation or not. What might be the culprit here, is that the pre-existing data is not the same as the new data that would overwrite it (I'm computing a schedule for a time interval which already has a schedule).

So I propose to open a new issue to investigate this. Overwriting of pre-existing data might be broken.

[victorgarcia98]

My requests were addressed.

Thanks for updating the tutorial @Flix6x :D

[nhoening]

Good work!