issues exporter

[nesnoj]

Thanks for the exporter, this makes life a lot easier.

While messing around with the export, I stumbled across 2 things:
(I used a network obj with several MVGDs)

1. Only 1 HV/MV station is exported. While you use indices for other components, you hard-coded this in stations:
   

   ding0/ding0/io/export.py

   Lines 176 to 185 in c04e8eb

   	# MV station
   	mv_station = mv_district.mv_grid._station
   	mv_station_name = '_'.join(
   	['MVStationDing0', 'MV', str(mv_station.id_db),
   	str(mv_station.id_db)])
   	mv_stations_dict[0] = {
   	'id': mv_district.mv_grid.id_db,
   	'name': mv_station_name,
   	'geom': wkt_dumps(mv_station.geo_data),
   	'run_id': run_id}

2. The schema in db_export is hard coded:

   ding0/ding0/io/db_export.py

   Line 41 in c04e8eb

   SCHEMA = "model_draft"

Sorry for not creating a PR due to lack of time for testing..

[nesnoj]

One more thing: In MVGD (DP v0.4.5, run 20180823154014) I get the following error:

sqlalchemy.exc.DataError: (psycopg2.DataError) value too long for type character varying(22)
 [SQL: 'INSERT INTO dingo.ego_ding0_mv_generator (subtype, type, is_aggregated, nominal_capacity, geom, name, weather_cell_id, v_level, run_id, id_db) VALUES (%(subtype)s, %(type)s, %(is_aggregated)s, %(nominal_capacity)s, ST_GeomFromEWKT(%(geom)s), %(name)s, %(weather_cell_id)s, %(v_level)s, %(run_id)s, %(id_db)s)'] [parameters: ({'subtype': 'biogas', 'type': 'biomass', 'is_aggregated': False, 'nominal_capacity': 366.0, 'geom': 'SRID=4326;POINT (13.6704972419341004 53.1837755031507982)', 'name': 'GeneratorDing0_MV_739_816809', 'v_level': 5, 'run_id': '20181017174124', 'weather_cell_id': None, 'id_db': 816809}, {'subtype': 'biogas', 'type': 'biomass', 'is_aggregated': False, 'nominal_capacity': 400.0, 'geom': 'SRID=4326;POINT (13.8148046652506995 53.2035544635309989)', 'name': 'GeneratorDing0_MV_739_816850', 'v_level': 5, 'run_id': '20181017174124', 'weather_cell_id': None, 'id_db': 816850}, {'subtype': 'biogas', 'type': 'biomass', 'is_aggregated': False, 'nominal_capacity': 526.0, 'geom': 'SRID=4326;POINT (13.6603850343025996 53.1536577562063002)', 'name': 'GeneratorDing0_MV_739_817052', 'v_level': 5, 'run_id': '20181017174124', 'weather_cell_id': None, 'id_db': 817052}, {'subtype': 'biogas', 'type': 'biomass', 'is_aggregated': False, 'nominal_capacity': 526.0, 'geom': 'SRID=4326;POINT (13.7916936875298006 53.1319633285071973)', 'name': 'GeneratorDing0_MV_739_817053', 'v_level': 5, 'run_id': '20181017174124', 'weather_cell_id': None, 'id_db': 817053}, {'subtype': 'biogas', 'type': 'biomass', 'is_aggregated': False, 'nominal_capacity': 625.0, 'geom': 'SRID=4326;POINT (13.6390457516435006 53.2374090519392027)', 'name': 'GeneratorDing0_MV_739_817445', 'v_level': 5, 'run_id': '20181017174124', 'weather_cell_id': None, 'id_db': 817445}, {'subtype': 'biogas', 'type': 'biomass', 'is_aggregated': False, 'nominal_capacity': 537.0, 'geom': 'SRID=4326;POINT (13.7565174460047999 53.1607264761275005)', 'name': 'GeneratorDing0_MV_739_817523', 'v_level': 5, 'run_id': '20181017174124', 'weather_cell_id': None, 'id_db': 817523}, {'subtype': 'biogas', 'type': 'biomass', 'is_aggregated': False, 'nominal_capacity': 600.0, 'geom': 'SRID=4326;POINT (13.5625132599157006 53.2967558983443013)', 'name': 'GeneratorDing0_MV_739_817564', 'v_level': 5, 'run_id': '20181017174124', 'weather_cell_id': None, 'id_db': 817564}, {'subtype': 'biogas', 'type': 'biomass', 'is_aggregated': False, 'nominal_capacity': 637.0, 'geom': 'SRID=4326;POINT (13.6894924645932008 53.2241893736201988)', 'name': 'GeneratorDing0_MV_739_817577', 'v_level': 5, 'run_id': '20181017174124', 'weather_cell_id': None, 'id_db': 817577}  ... displaying 10 of 264 total bound parameter sets ...  {'subtype': 'wind_onshore', 'type': 'wind', 'is_aggregated': False, 'nominal_capacity': 600.0, 'geom': 'SRID=4326;POINT (13.7564623471836001 53.2611841312481005)', 'name': 'GeneratorFluctuatingDing0_MV_2275_817462', 'v_level': 5, 'run_id': '20181017174124', 'weather_cell_id': 1139099.0, 'id_db': 817462}, {'subtype': 'wind_onshore', 'type': 'wind', 'is_aggregated': False, 'nominal_capacity': 600.0, 'geom': 'SRID=4326;POINT (13.8147369403968003 53.2413659627798026)', 'name': 'GeneratorFluctuatingDing0_MV_2275_817463', 'v_level': 5, 'run_id': '20181017174124', 'weather_cell_id': 1139099.0, 'id_db': 817463})] (Background on this error at: http://sqlalche.me/e/9h9h)

This is because one dataset contains biogas_dry_fermentation as subtype which has 23 chars but the col
only allows 22:

ding0/ding0/io/db_export.py

Lines 278 to 292 in c04e8eb

	ding0_mv_generator = Table(DING0_TABLES['mv_generator'], METADATA,
	Column('id', Integer, primary_key=True),
	Column('run_id', BigInteger, ForeignKey(versioning.columns.run_id), nullable=False),
	Column('id_db', BigInteger),
	Column('name', String(100)),
	Column('geom', Geometry('POINT', 4326)),
	Column('type', String(22)),
	Column('subtype', String(22)),
	Column('v_level', Integer),
	Column('nominal_capacity', Float(10)),
	Column('weather_cell_id', BigInteger),
	Column('is_aggregated', Boolean),
	schema=ding0_schema,
	comment=prepare_metadatastring_fordb("mv_generator")
	)

Needs to be fixed too, let's take 30 or something..

[nesnoj]

And another issue: while exporting some hundred grids, there was at least one which crashed during export of lv_grids. The origin is here:

ding0/ding0/io/export.py

Line 443 in ca6ca4e

'geom': wkt_dumps(lv_district.geo_data),

The geom is a Polygon but the table column type is MultiPolygon. Without reviewing the code I have no clue why we have the wrong shapely geom type attached to .geom property.
I needed a fast fix so I just added

                        if isinstance(lv_district.geo_data, Polygon):
                            lv_district.geo_data = MultiPolygon([lv_district.geo_data])

prior to the dict assignment (MultiPolygon must be imported from shapely in the script header). This ugly fix must not be implemented in the branch, but it may helps you to avoid the export to crash for your run which I guess is quite time-critical too.

I can check why we have the wrong type here by end of Oct.

[jh-RLI]

Was not aware that this will cause trouble to the import. At the time I created the table definition I tried to use the geomtypes I found in the table description and ORM class definition. I already noticed that in some cases the "geom" Column types does not match the shaply geometrys for some reason. So maybe the mistake is not within the shaply.geom but in the Table definition?

[mltja]

Close because of upcoming major refactoring of the database interface #369, and then the using of saio.