Define the pipeline & decide ecosystem

[clizbe]

Description

What do we need from Toolbox/ESDL and what's our pipeline going to be?

• What are all the pieces of the pipeline?
• How much of the pipeline can SpineToolbox cover?
• How much of the pipeline can ESDL/EDR cover?
• What is left uncovered?
• Do we need to change the input of the model to "match" SpineToolbox?

Related Issues

Blocking #94, #106, #105, #118, #88, #89, #115, #36

[clizbe]

(draft - feel free to edit)

Pipeline	SpineToolbox	ESDL	Tulipa	Other
1. Reference/Store Raw Data Sources	Plug-in	YES	-	-
- MIDDEN (EDR or Energy.nl)	Reference	Reference	-	-
- Profiles (CoreRES for instance)	Reference	Reference	-	-
- Networks (ENTSO-E, TenneT)	Reference	Reference	-	-
- Others?	"	Reference	-	SQL
2. Preprocessing Steps	Plug-in	-	Some	-
3. Specifying scenarios	YES	Partly	Possibly	Other file
4. More pre-processing according to scenarios?	Plug-in	-	Can	-
5. Conversion into model in Julia	Plug-in?	-	Possibly	-
6. Running model	-	-	YES	-
7. Output results to somewhere	New SQL db	Tulipa DB	-	-
8. UI for graphs and analyses	Simple	MapEditor?	-	ESM UI, others
9. Share results with other models	-	YES	-	-

[clizbe]

The more I look at this, the more I think we should use SpineToolbox to integrate everything and the EDR as just one of several data sources. And maybe also be able to put results back into ESDL to share results with other models and use the MapEditor etc for analysis (depending on squeeze-to-juice ratio).

Mostly when looking at the SpineToolbox column - it really highlights that SpineToolbox is an integrator. Really, all of these pieces are different tools that have just a bit of overlap. We should decide where the interfaces are.

What I'm a bit concerned about is the whole system still working in 5 years. Seems a bit complex.

[clizbe]

Assigned myself although this is a group effort.

[clizbe]

Discussion:
Suvayu recommends not using SpineToolbox because it has its own limitations. It can be added as a bonus later.
ESDL is usable - we just need to figure out the other parts of the process and how we want it to flow.

[clizbe]

WHAT WE WANT

• Build the network once (in a while)
  • Use draft networks to build new networks
• Sufficient flexibility for ad-hoc code for experimentation
• Definition of temporal stuff
• Definition of solver specifications
• Be able to mix data sources (ESDL + ENTSO-E for example)