1st Data Package: 01 HC Layers

[p-a-s-c-a-l]

Create Layers for Hazard Indices calculated by ZAMG and make them available in CSIS.

This encompasses the following steps (to be confirmed / updated by @clarity-h2020/data-processing-team):

• provide original HW hazard indexes for 1+3 time periods and 3 RCPs.
  • available here: sftp://5.79.69.49/clarityftp/europe/hazard_indices/heat/heat-wave-duration/Index_Tx75p/
• transformation to the harmonised Grid
• expose Layers (WCS/WMS) via GeoServer (currently located at METEOGRID Server)
  • available here: https://clarity.meteogrid.com/geoserver/web/wicket/bookmarkable/org.geoserver.web.demo.MapPreviewPage?2
• define some simple styling for Layers
  • not yet available
• Create Data Package Meta-Data in CSIS (see also 1st Data Package: 00 Background Layers - 2) Create Data Resource #7 and 1st Data Package: Context Entities #12)
  • not yet available
• Provide meta-data for Data Management Plan
  • not yet available

Questions:

• those layers are not used directly for impact calculation, right? Main purpose is visualisation in CSIS Map & Table View and input for Hazard Local Effects.
• is the grid transformation necessary or is the input data (NetCDF) already in the correct grid / projection?
• can the transformation / postgis import / geoserver layer generation be automated by some scripts? HC Layers need to be generated for Europe (screening), 4 DCs and when ZAMG has re-calculated the indices based on bias corrected data.
• who is responsible for this tasks? ATOS, METEOGRID, AIT?
• how and for which purpose is the data accessed by the impact model service (Emikat)? CSW or direct data base access (preferable)? Only together with a "differential layer" (from local effects) as inmput for HxExV?

Additionally, Map Visualisation, Table Visualisation and Report Generation have to be implemented.

[stefanon]

those layers are not used directly for impact calculation, right? Main purpose is visualisation in CSIS Map & Table View and input for Hazard Local Effects.

yes

is the grid transformation necessary or is the input data (NetCDF) already in the correct grid / projection?

for what I've seen ZAMG NetCDFs are in EPSG:3035 alias ETRS89 / ETRS-LAEA . The raster pixel size is way bigger of the base 500x500m grid.

can the transformation / postgis import / geoserver layer generation be automated by some scripts? HC Layers need to be generated for Europe (screening), 4 DCs and when ZAMG has re-calculated the indices based on bias corrected data.

Postgis import can be script automated, trasformations can be done in PostgreSQL/PostGIS PL/PgSQL procedures, Geoserver publication can be scripted by means of the Geoserver REST API.

[ghilbrae]

Answer in agreement with @luis-meteogrid

Create Layers for Hazard Indices calculated by ZAMG and make them available in CSIS.

Not only direct Hazard indexes but also, the CSIS needs to have the Local Effects in the Hazard indexes layers. As agreed in the Plenary Meeting, Meteogrid is responsible of doing this calculation.

This encompasses the following steps (to be confirmed / updated by @clarity-h2020/data-processing-team):

transformation to the harmonised Grid

We intend to do the calculations in the original grid and, as a last step, to transform everything to the agreed upon grid. ZAMG's original grid has EPSG 4326 instead of 3035, as expected by AIT. We can re-project and fit the results to the expected resolution.

import into PostgreSQL Database (currently located at ATOS Server)
expose Layers (WCS/WMS) via GeoServer (currently located at ATOS Server)

We were planning on using our instance of GeoServer located in the server in which we are going to perform the calculations for the Local Effects. Please, be aware that the Local Effects might need to be re-calculated on-the-fly if the user modifies the distribution of Urban Elements.
The layers will be exposed as raster files with 3035 projection using the reference grid provided by AIT. This includes both the re-projection of ZAMG's hazard indexes as well as the local effects. This will be done for other layers, such as Population Layer, that AIT might need.

style Layers according to Hazard Thresholds

This might be done at the visualization level or be attached to the layer if someone is able to provide the thresholds beforehand. We have a tool that already does this running on a django app.

Create Data Package Meta-Data in CSIS (see also https://github.com/clarity-h2020/data-package
/issues/7)
Provide meta-data for Data Management Plan
Questions:

those layers are not used directly for impact calculation, right? Main purpose is visualisation in CSIS Map & Table View and input for Hazard Local Effects.

From our POV, the main purpose is calculation of Local Effects as well as visualization. So, yes.

is the grid transformation necessary or is the input data (NetCDF) already in the correct grid / projection?

A transformation is necessary, the original data do not have the correct projection, grid nor the resolution.

can the transformation / postgis import / geoserver layer generation be automated by some scripts? HC Layers need to be generated for Europe (screening), 4 DCs and when ZAMG has re-calculated the indices based on bias corrected data.

The main goal is to automate most of the process, we'll have to see if it is achievable at a 100%. For the Local Effects calculations we are using our own framework to achieve this automation, both for the calculations, re-projection and interpolation.

who is responsible for this tasks? ATOS, METEOGRID, AIT?

We can take charge of the first steps, up until the Local Effects generation. AIT seems to be responsible from that step onward. If there's any problem with visualization we can agree on some sort of common solution.

how and for which purpose is the data accessed by the impact model service (Emikat)? CSW or direct data base access (preferable)? Only together with a "differential layer" (from local effects) as inmput for HxExV?
Additionally, Map Visualisation, Table Visualisation and Report Generation have to be implemented.

We understood that report generation was going to fall on Drupal's side.

As for the other two, it will depend on what this might entail, we already have some tools for map visualization, if they can be reused for this we could try and see how to provide some sort of communication between them, it would also depend on the ease of communication with drupal or/and emikat or any other tools.

Just to be clear, we are not going to develop on drupal, but we are willing to help with other tools to access data (tables), layers or maps.

[p-a-s-c-a-l]

Ok, Thanks!

In this issue, I care mainly about the original hazard Data, Local Effects will be addressed / discussed separately.

So AIT & ATOS are not directly involved in the "plain" HC task. The demo Hazard Layers on ATOS server will not be used in CSIS or for any HC-LE or RA/IA calculation. Agreed @bernhardsk @humerh & @negroscuro ?

In summary, the responsibilities are

• METEOGRID will do the reprojection of ZAMG's Hazard NetCDF files from WGS84 (EPSG:4326) 0.11 degree (EUR-11, ~12.5km) grid to ETRS89 LEA (EPSG:3035) 500x500m grid and make the layers available for visualisation via their own geoserver instance (https://clarity.meteogrid.com/geoserver/).
• CIS will create the Data Package Meta-Data in CSIS and show the layer on a (simple) Map. Of course, since no downscaling is performed yet, the 500x500m layer will still show just a 12x12km grid.
• CIS will add the respective Datasets (meta-data) to CKAN for the Data Management Plan

Open Issues:

• Atm, there is just one WMS Layer for Heatwave Duration Hazard for RCP2.6 and period 2011-01-01 - 2040-12-31 available and the layer is missing a proper style. For a realistic demo some simple styling must be defined and the Heatwave Layers for the remaining RCPs and time periods (@astridkainz should upload them to the sFTP server) should be added.
• For tabular visualisation we would need access to the actual values of all grid cells in the study area. How the values are aggregated and mapped to thresholds (good, average, bad) will be addressed in a separate Issue. Probably, this can be done on client-side, so access via e.g. WFS would be sufficient in the first place.

[RobAndGo]

Hello,
@astridkainz forwarded me the link to this discussion, so I'm just joining in now.

I have uploaded 2 zip files here:
[https://repository.atosresearch.eu/owncloud/index.php/apps/files?dir=/CLARITY/WP3%20Science%20Support/T3.2%20Climate%20Intelligence/Data]

Sorry if it is not the new way to upload files, but I didn’t have time to look into the CKAN/ATOS ftp server thingy.

These files are in the native netcdf EURO-CORDEX rotated grid projection. @luis-meteogrid you said it was no problem for you to convert it into a useable projection.

All of this data is to do with the climate index “Tx75p” which is a measure for heat wave duration. In detail, the index is:
“Maximum number of consecutive days with a daily maximum temperature ≥ 75th percentile of daily maximum temperatures in the period April – Sept (1971 – 2000).”

The file HazardMap_Tx75p.zip contains 9 layers for Tx75p for:

• RCPs 2.6, 4.5, 8.5
• Time periods 2011-2040, 2041-2070, 2071-2100
  These files show the hazard scale as 1=low, 2=medium, 3=high.

The file also contains a text file describing the rule I used to define the low/medium/high hazard levels.
“
The final signal is represented as an index of 1, 2, 3 (low, medium, high) according to:
1 - increase < 50 %
2 - increase >= 50 %, increase < 100 %
3 - increase >= 100 %
“
@p-a-s-c-a-l: is this what you mean by “style” in your comment of “some simple styling must be defined”?

The raw index values of Tx75p are in the file Index_Tx75p.zip. These values are the actual index values and would/could be used when implementing the method created by PLINIVS.

I hope this helps!

[astridkainz]

We've also prepared an updated example for the tabular visualisation of hazards:

In particular, we need to make sure that their is a clear distinction between the reference period (baseline) and the future scenarios, mainly because the respective hazard scales are defined differently.

A pull-down menu could be provided to let the user switch between different future periods, as suggested by Denis.

[p-a-s-c-a-l]

@p-a-s-c-a-l: is this what you mean by “style” in your comment of “some simple styling must be defined”?

Hm, the 'original' indexes range from 5,433333 to 41,63334 (days). IMHO we don't need the additional hazard maps with 'normalised' indexes (1 to 3, low, medium, high), but just the thresholds (e.g. x = 8 and y = 14) to perform the normalisation just in time when we render the maps and the tables.

@clarity-h2020/data-processing-team With help a of a styled layer descriptor it would be possible to map the original indexes (5,4 - 41,6) to 1,2,3 (green, yellow, red) when absolute values for x and y are know, right? On the other hand, I think we should show the original indexes in the map and the normalised indexes just in the table. Reason: The user can recognise slight differences between adjacent grid cells, especially when taking local effects in to account (500x500m grid). If we just show values ranging from 1-3 in the map, probably no difference between HC and HC-LE would be recognisable.

@clarity-h2020/mathematical-models-implementation-team: for impact calculation you would need the original indexes anyway?

So at the moment I don't see a need to store them twice in the database / geoserver.

Map comparison: orginal vs normalised:

@ghilbrae Could you please transform the ncfd files (sftp://5.79.69.49/clarityftp/europe/hazard_indices/heat/heat-wave-duration/Index_Tx75p/) to the normalised grid and make it available in your geoserver instance?

[RobAndGo]

IMHO we don't need the additional hazard maps with 'normalised' indexes (1 to 3, low, medium, high), but just the thresholds (e.g. x = 8 and y = 14)

Yes, that is true. It would also save some work from my side. :-)

@clarity-h2020/data-processing-team One point to add is that the thresholds I used weren't fixed values, but rather, values relative to the baseline (1971-2000). So, if at one place (e.g. Vienna) the baseline maximum heatwave length was 10 days and at another place (Naples) the baseline maximum heatwave length was 15 days, then a "high" hazard level for those places in the future would be >20 days and >30 days, respectively (i.e. increase >100% as I had defined it). This gets around the problem of latitudinal variations in the climate.

However, in keeping with @p-a-s-c-a-l 's idea, in this case one would just need to define the appropriate percentage values for low/medium/high and then have the software calculate e.g.
100 x [(future layer) - (baseline layer)] / (baseline layer)
Then find the points with values e.g. > 100% which would correspond to the high hazard level.
Where the baseline layer contains zeros and the future layer positive values, one could set those points to high - otherwise keep it at low.

The percentage values one uses to define low/medium/high would be different according to the index (e.g. for the flood hazard, the 5-day precipitation index would have 10% and 20% as a better choices for the different hazard levels).

At the moment my choice of these percentage values is not really scientifically based :-( but rather based on producing a map which is not completely everywhere "low" or everywhere "high".

[p-a-s-c-a-l]

Thanks for the clarification, yes defining the percentage relative to baseline makes more sense.
However, how do we define what's low/medium/high in the baseline period?

[RobAndGo]

how do we define what's low/medium/high in the baseline period?

@p-a-s-c-a-l Good question! To date I have just focused on the hazard scale for the future periods.
We are currently having some good discussions on how to define low/medium/high in the baseline period and I'll comment further on this when we have come to an agreement.

My personal opinion is that we omit this classification for the baseline period and just show an absolute value. My reason for this is following - what happens if a user sees that their hazard level for the current climate is "high", but life is going on without problem, buildings are not crumbling down, infrastructure is functioning, and everything is good. Then, if the hazard level for the future period (e.g. 2041-2070) period is also "high", how does the user react? I would guess they would be of the opinion that they don't need to do anything, as the current hazard level is "high" and nothing bad is happening.

[p-a-s-c-a-l]

at we omit this classification for the baseline period and just show an absolute value

Ok, agreed!

@luis-meteogrid
Could you please make the HW hazard (sftp://5.79.69.49/clarityftp/europe/hazard_indices/heat/heat-wave-duration/Index_Tx75p/) available in your geoserver instance (in normalised grid via CSW and WMS + simple styling?

[RobAndGo]

@p-a-s-c-a-l
OK, we (@mzuvela @astridkainz @claudiahahn) finally have a consensus.

1. The low/medium/high in the baseline period will/can be defined relative to all values of the index across Europe (i.e. a comparison across space). Such a method has also been used in the BRIGAID project (Deliverable 5.1, pages 15-16) in the assessment of their baseline climate.

One thing to note is that this will only work with indices defined with non-constant threshold values! For example, using our Tx75p example which uses a percentile value as a threshold is OK, as this value varies according to location. Here is a plot showing the absolute values of Tx75p. Just by eyeballing it, the colours deep-orange/red would be “high”, white-yellow “medium” and blues “low” hazard level.

However, using the index "number of summer days" will not work, since a summer day is defined as a day with the maximum temperature > 25°C (i.e. a fixed value) - in this case southern (northern) Europe will always register a high (low) hazard level.

2. If point 1 is implemented, then the table defined by @astridkainz in 1st Data Package: 01 HC Layers #8 (comment) will become complicated, as the terms low/medium/high for the future scenarios are not the same as the terms for low/medium/high for the baseline climate. To be precise, the future climate will need to be defined as "low change/medium change/high change" with the emphasis on the word "change" relative to the baseline climate.

3. Although we have a lot of different climate indices that we set out to calculate, not all need to be represented in a hazard table form. The idea is that we can calculate the hazard tables only for those indices that contain non-fixed threshold values, and keep the other indices for graphical plots or for processing which the methods of PLINIVS may require. The problem with calculating the hazard table for a lot of heat indices is that there is the risk of conflict between the results - e.g. if Tx75p gives a high change hazard level for 2041-2070 but summer days shows a low change hazard level, what should the user do?

[humerh]

I want to refer to my comment at:
https://github.com/orgs/clarity-h2020/teams/data-processing-team/discussions/1?from_comment=15#discussion-1-comment-15

If the hazard data are provided in the way, which were shown in the example "clarity:consecutive_max_heat" at the Geoserver "http://5.79.69.33:8080/geoserver" we are very lucky.
This example shows an WCS Interface and gives me the data in the raster size of 500m.

This example demonstarates also, how to extract only the cell data of the Project area.

I hope, that Atos and Meteogrid can provide all Hazard data layers in this way.

[p-a-s-c-a-l]

@humerh Please don't use the ATOS instance (5.79.69.33) anymore for HC & EE (we might still use it for "background layers" that are input for to HC-LE), please use the METEOGRID instance instead: https://clarity.meteogrid.com/geoserver/

@negroscuro Please remove the HC layers to avoid further confusion

@luis-meteogrid please provide the HC and EE layers on your geoserver instance

Thanks!

[ghilbrae]

Example of access via WCS (GetCapabilities):

https://clarity.meteogrid.com/geoserver/clarity/wcs?service=wcs&version=1.0.0&request=GetCapabilities&namespace=clarity

[RobAndGo]

No, there is some structure over Europe. When one plots the data from 0->(max value), one gets:

But limiting the range between (min value)->20, one can see that there is something meaningful there:

The problem with this index is that the values over water are higher than over land - the water temperature stays constantly warm in summer and is more stable, thus producing more days where the maximum temperature exceeds the 75th percentile. Thus the island coastal regions of Greece, Croatia, Italy not masked out will skew the colour scale. This was why I used the original mask to omit these problematic regions.

However, if it is possible to plot the colour scale to just 75% of the maximum value, maybe this could solve the problem.

[LauraMTG]

Ok @DenoBeno and @RobAndGo

The solution then is to create new legends with the following criteria:
1-More intervals for the set of low values.
2-New colors associated to each interval.
3- Lower legend maxima, e.g. 75% of the data range.

This may apply to several "type legends" which in turn apply to a set of layers. However, each layer cannot have a specific legend since it would be necessary to create 300 legends.

If this is a possible solution, I will proceed with it as follows

[DenoBeno]

Not sure what the story with 300 legends is, but it's IMO a good idea to keep the scale for a group of similar layers the same. Otherwise they can't be visually compared.

…

On Fri, 11 Oct 2019 at 14:36, LauraMTG ***@***.***> wrote: Ok @DenoBeno <https://github.com/DenoBeno> and @RobAndGo <https://github.com/RobAndGo> The solution then is to create new legends with the following criteria: 1-More intervals for the set of low values. 2-New colors associated to each interval. 3- Lower legend maxima, e.g. 75% of the data range. This may apply to several "type legends" which in turn apply to a set of layers. However, each layer cannot have a specific legend since it would be necessary to create 300 legends. If this is a possible solution, I will proceed with it as follows — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8?email_source=notifications&email_token=AAWTC7QKO2WTZ7M4X2ER3LTQOBXNLA5CNFSM4GHYS4O2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEA73MWQ#issuecomment-541046362>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAWTC7VZNVDJMKZZPXRE42DQOBXNLANCNFSM4GHYS4OQ> .

[RobAndGo]

@DenoBeno is right and is something I should have mentioned earlier

When comparing the scenarios one needs to use the same scale/legend. As a suggestion, one could fix the colour scale to that from the RCP45 2041-2070 scenario. This is approximately the middle of the possibilities, whereby the results from RCP26 and earlier time periods will appear lower, while those results from RCP85 and later time periods will appear higher.

[LauraMTG]

Ok @DenoBeno and @RobAndGo

We are in the process of correcting the mask of all HI layers. This can alter some values that are presented so I suggest to fix the legends later.
We will elaborate a standard legend for each RCP45 2041-2070 that will be applied to the rest of scenes and periods for each HI set and we will check that this solves the visualization of the data, if it seems suitable to them.

[DenoBeno]

I have opened a new issue for data color coding/map scale. Please see #53.

[p-a-s-c-a-l]

HC Heat Layers available for 1st data package.

[p-a-s-c-a-l]

We have to add Pluvial Flood HC resources. See clarity-h2020/local-effects#9 (comment)

[p-a-s-c-a-l]

A draft template resource for pluvial flood that uses variables is available here. Map Visualisation isn't 100% working yet as we had to introduce a lot of accidental complexity into the codebase due to incoherent variable meanings.

[p-a-s-c-a-l]

Including this one, there are now three issues regarding the Pluvial Floods Input layers (#54 and clarity-h2020/local-effects#9), so closing this one.