Refactor scheduler interface - API and inner logic

[nhoening]

• Add flex_model & flex_context in API endpoint
• refactor design for Scheduler implementations (moving some endpoint logic here, scheduler is responsible for deciding which flex_model is to be used / validated)

TODO:

• Error handling of validation errors within endpoint (->422)
• Docs pages which explain flex_model(s) and flex_context. Also link to there from endpoint docs and plugin developer docs
• Has soc-sensor-id ever been used at all?
• Adapt CLI data_addand tutorial to use make_schedule and create_scheduling_job in the new way.
• Also add deprecation logic for flex_context params
• Changelog entry
• Try to not use underscore notation in API parameters, as "-" is more user-friendly to read.

[coveralls]

Pull Request Test Coverage Report for Build 3799875511

• 280 of 325 (86.15%) changed or added relevant lines in 13 files are covered.
• 3 unchanged lines in 2 files lost coverage.
• Overall coverage increased (+0.5%) to 65.59%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
flexmeasures/data/services/scheduling.py	24	25	96.0%
flexmeasures/api/common/utils/deprecation_utils.py	19	22	86.36%
flexmeasures/api/v1_3/implementations.py	13	17	76.47%
flexmeasures/cli/utils.py	7	12	58.33%
flexmeasures/data/models/planning/init.py	44	49	89.8%
flexmeasures/api/v3_0/sensors.py	41	48	85.42%
flexmeasures/data/models/planning/storage.py	83	92	90.22%
flexmeasures/cli/data_add.py	7	18	38.89%

Files with Coverage Reduction	New Missed Lines	%
flexmeasures/data/models/planning/init.py	1	86.15%
flexmeasures/cli/data_add.py	2	32.26%

Totals
Change from base Build 3743385027:	0.5%
Covered Lines:	6841
Relevant Lines:	9802

---

💛 - Coveralls

[Flix6x]

Impressive work so far. Here are some documentation suggestions already.

[nhoening]

For me, only one item remains in this PR: I chose for underscores in the flex model and flex context parameter names to the API, but I now believe I could give users the "-" notation which we have used before as well.

[Flix6x]

I noticed some unaddressed review comments, some of which may be worthwhile to discuss, like #537 (comment).

[nhoening]

Oh yes, that one did escape, thanks for reminding.

[Flix6x]

Well done! I only added some comments for a potential function rename and opening up a new issue to advance our use of Marshmallow.

[Flix6x]

Maybe just a note for later: we use a Marshmallow schema in this function, which by now does not only do validation, but also deserialization (which is what changes the flex config fields with dash notation to the flex config variables with underscore notation). This is already disclosed in the docstring, but I believe the function name could bring that to light better, because it's more than an inspection at this point. Maybe deserialize_flex_config. The function also fills in defaults (if needed), but this can be done in the schema, too, using load_default.

[Flix6x]

FYI: I found out how to access the sensor object (deserialized from location="path") in the schema used for deserializing fields from location="json" (using https://webargs.readthedocs.io/en/latest/advanced.html#meta-locations). Knowing this, filling in missing values can also happen using Marshmallow, e.g. using @validates(field_name) or @validates_schema. So all three of these ideas can happen in Marshmallow, which may be worth a note and/or a follow-up ticket.

Btw, I started using these meta-locations in #563, so I know how to make that work now.