Transport Module integration description

[ghilbrae]

The approach is to embed all needed screens from the Transport Module into the CSIS. This means that each combination of Tab+Section, i.e., Hazard Characterization+Table, will need to access a different URL within the Transport Module but specifying the ID for the corresponding Study.

For example:
https://clarity.saver.red/en/simmer/simulation/ID/hazard_table
https://clarity.saver.red/en/simmer/simulation/ID/hazard_summary
https://clarity.saver.red/en/simmer/simulation/ID/exposure_evaluation/{}

01 - Add Study

Using the CSIS a new study can be created. This information has to be sent to the Transport Module (TM, from now on) to create the study there too. The TM will return an ID (integer).

Introduction

The text has already been changed but it might be updated (TBD).

Team

Comes from the CSIS

Context

Text was already changed.

Area

The area for the study will be selected. An embeded screen from the TM will go here to select the area.
For now the CSIS does not need to be aware of the selected area, but this can change in the future if there's time to do a better integration (via API, WMS, WFS, etc.).

Data

Comes from the DP in the CSIS.

Summary

As the CSIS will have no information regarding what's needed to create a Summary, there should be a request to the TM for a Summary at this stage (PDF?).
This will be replicated for all other Summaries from now on. Each Summary will be a portion of the final one pertaining to the section in which it has been requested.

02 - Hazard Characterization

Introduction

The text might change but still TBD.

Data

This comes from the Data Package. We'll have to ensure that these data is up to date with that on the TM, so nothing needs to be done.

Table

The TM will generate the table and it will be embedded in the CSIS, at least in the short term. Take into account that each of these embedded screens will be accesed through different urls that must include the ID for the study. Otherwise, all urls will be the same in each case.
Note that this may be changed in the future if there's time or after the project is finished.

Maps

Access the corresponding map on the TM via URL.

Twins

TBD
It may be the same as for the Urban Studies.

Summary

See description above.

Hazard Characterization - Local Effects

This has no use in the TM and has to be removed.

03 - Exposure Evaluation

Introduction

The text will change but still TBD.

Data

This comes from the Data Package. We'll have to ensure that these data is up to date with that on the TM, so nothing needs to be done.

Table

The TM will generate the table and it will be embedded in the CSIS, at least in the short term. Take into account that each of these embedded screens will be accesed through different urls that must include the ID for the study. Otherwise, all urls will be the same in each case.
Note that this may be changed in the future if there's time or after the project is finished.

Maps

Access the corresponding map on the TM via URL.

External

It no longer makes sense. All will be done via embedding.

Twins

TBD

Summary

See description above.

04 - Vulnerability Analisys

Introduction

The text might change but still TBD.

Data

This comes from the Data Package. We'll have to ensure that these data is up to date with that on the TM, so nothing needs to be done.

Table

The TM will generate the table and it will be embedded in the CSIS, at least in the short term. Take into account that each of these embedded screens will be accesed through different urls that must include the ID for the study. Otherwise, all urls will be the same in each case.
Note that this may be changed in the future if there's time or after the project is finished.

Maps

Access the corresponding map on the TM via URL.

Summary

See description above.

05 - Risk and Impact

Introduction

The text might change but still TBD.

Data

This comes from the Data Package. We'll have to ensure that these data is up to date with that on the TM, so nothing needs to be done.

Table

The TM will generate the table and it will be embedded in the CSIS, at least in the short term. Take into account that each of these embedded screens will be accesed through different urls that must include the ID for the study. Otherwise, all urls will be the same in each case.
Note that this may be changed in the future if there's time or after the project is finished.

Maps

Access the corresponding map on the TM via URL.

Scenario Analysis

TBD, we do not have a clear idea of what should be in here. How is it working for other studies??

Summary

See description above.

06 - Identify Adaptation Options

Introduction

The text might change but still TBD.

Table

This will come from the CSIS. We'll create the corresponding Adaptation options within the CSIS that can be associated to this kind of study.

Twins

TBD

Summary

This summary will come from the CSIS and will only contain the Adaptation Options.

07 - Final Summary

As stated on 01 - Add Study, the Summary will be a request for the full Summary to the TM, however the CSIS will need to provide an extra one containing the Summary for the Adaptation Options as an additional document.

[ghilbrae]

Just in case it's useful to you, here 04 CSIS - Transport module.zip you can see a bunch of screenshots with this workflow. When in doubt the description above takes precedence over the screenshots.

[patrickkaleta]

Ok, so what we need to do on CSIS-side:

• update the Study type Advanced Screening: Transport infratructure (remove the HC local effects step, remove "External" tab in EE, change the calculation method,...)
• add calculation method "Transport Module" (or some other name?)
• create a trigger (probably similar to the Emikat trigger?) for new studies of this type
• create "Extended Iframe" entities for the Maps and Table tabs (one for each might be enough, if we can add additional query parameters as explained here)
• create new Adaptation options for this Study type and link them to corresponding data packages
• find a way to easily create the summary with the input from the TM (not quite sure how to do this)

Biggest challenges will probably be 4) and 6).

For the Emikat trigger we're using REST. I hope that for the TM we can do the same. @ghilbrae who can I talk to in order to configure this new trigger (define endpoints, necessary data that needs to be transferred, etc)?


Regarding the summary:

As the CSIS will have no information regarding what's needed to create a Summary, there should be a request to the TM for a Summary at this stage (PDF?).
This will be replicated for all other Summaries from now on. Each Summary will be a portion of the final one pertaining to the section in which it has been requested.

@ghilbrae what did you have in mind? Do you want to create the final pdf report inside the TM and send it to the CSIS? I think it would be easier to leave the summary as it is. The users will take screenshots of the TM in the "Maps" and "Table" tabs and those screenshots will be added to the summary (just like for any other study). We can add a new text field (call it maybe "Commentary"?), where the TM could (via REST) add additional information. @DenoBeno requested such a field anyway, so that users could add their own texts to the final reports.

[p-a-s-c-a-l]

Regarding 4), I think we only need one extended iFrame entity if we can generate the the app's URI automatically from context information. Therefore we have to agree on an URL schema, e.g. https://clarity.saver.red/en/simmer/simulation/${id}/${step}/${component} with

• ${id} = CSIS studyId? CSIS studyUUID? or TA Application ID?
• ${step} = preferably the EU-GL taxonomy name
• ${component} = table, summary, map, etc. (the currently active tab)

[ghilbrae]

For the Emikat trigger we're using REST. I hope that for the TM we can do the same. @ghilbrae who can I talk to in order to configure this new trigger (define endpoints, necessary data that needs to be transferred, etc)?

The main developer of the TM is @rapto so he'll be the one to talk to.

Regarding the summary:

As the CSIS will have no information regarding what's needed to create a Summary, there should be a request to the TM for a Summary at this stage (PDF?).
This will be replicated for all other Summaries from now on. Each Summary will be a portion of the final one pertaining to the section in which it has been requested.

@ghilbrae what did you have in mind? Do you want to create the final pdf report inside the TM and send it to the CSIS? I think it would be easier to leave the summary as it is. The users will take screenshots of the TM in the "Maps" and "Table" tabs and those screenshots will be added to the summary (just like for any other study). We can add a new text field (call it maybe "Commentary"?), where the TM could (via REST) add additional information. @DenoBeno requested such a field anyway, so that users could add their own texts to the final reports.

That was exactly our idea so the CSIS would not need to compose anything, only get the PDF and store it or send it to the user. If you think it would be easier or less problematic going down the path of screenshots we are of course open to suggestions but how would that work? Is the CSIS able to take screenshots right now?

[p-a-s-c-a-l]

Is the CSIS able to take screenshots right now?

In theory yes, but this feature is rather unreliable and not compatible with all browsers. We had to implement a lot of workarounds. You can give it a try, but I wouldn't count on it.

[luis-meteogrid]

I think the best way to proceed by now is follow Pascal's idea, let agree on an URL schema. We are right now developing the new screens needed for vulnerability and risk, so I'll be back next week with the proposed schema.

[ghilbrae]

@patrickkaleta I'm attaching a short document with the text that needs to change as detailed on the first comment (#134 (comment)).

csis_tranpost infrastructure_rev.docx

Let us know if you have any questions.

[patrickkaleta]

@patrickkaleta I'm attaching a short document with the text that needs to change as detailed on the first comment (#134 (comment)).

csis_tranpost infrastructure_rev.docx

Let us know if you have any questions.

New texts are now shown instead of the default EU-GL taxonomy term texts. For that I updated the Study templates of the "Advanced Screening: Transport Infrastructure" Study type.

BTW: Custom Introduction texts can now also be added for other Study type templates as well. If the Step description field of a GL-template is filled with text, it will be displayed in the Study Introduction, Summary and PDF output instead of the default EU-GL taxonomy text.

[patrickkaleta]

The TM trigger has now been implemented in the csis-helpers module.

[patrickkaleta]

@luis-meteogrid & @ghilbrae
As discussed in the telco today, if you have the URL schemas available for the TM tables and maps, please post them here, so that we can configure the CSIS as described in Pascals comment.

[ghilbrae]

We have a bunch of URLS to share right now. Just a question first, is the creation of a new study already linked to the TM? We've created a new one but it does not seem to create the study in both sides.

The URLs that are working now are as follows:

Study -> Area -> https://clarity.saver.red/studies/reference/[CSIS Study ID]/

Hazard Characterization -> Table -> https://clarity.saver.red/study-hazard/reference/ID/elements_hazard/
Hazard Characterization -> Map -> https://clarity.saver.red/study-hazard/reference/ID/

Exposure Evaluation -> Table -> https://clarity.saver.red/study-exposition/reference/ID/elements_exposition/
Exposure Evaluation -> Map -> https://clarity.saver.red/study-exposition/reference/ID

Vulnerability Analysis -> Table -> https://clarity.saver.red/study-vulnerability/reference/ID/elements_vulnerability/
Vulnerability Analysis -> Map -> https://clarity.saver.red/study-vulnerability/reference/ID

Risk and Impact Analysis -> Table -> https://clarity.saver.red/study-risk/reference/ID/elements_risk/
Risk and Impact Analysis -> Map -> https://clarity.saver.red/study-risk/reference/ID

For Adaptation Options we will use the CSIS as is.

A working ID in the TM, that you can check if needed is 72.

Note that a couple of URLs may give some problem because we are still ironing out some details and working on them, but we think they are good to go to start testing the integration.

Summary and report stuff is an on going development, we will let you know when we have something working.

[p-a-s-c-a-l]

OK, so it seems that need just 2 Parameters to construct all of those URL:

• ${STUDY_ID}
• ${STEP_NAME} [vulnerability, exposition, risk]

Examples:

• https://clarity.saver.red/study-${STEP_NAME}/reference/${STUDY_ID}/elements_${STEP_NAME}/
• https://clarity.saver.red/study-${STEP_NAME}/reference/${STUDY_ID}/elements_${STEP_NAME}/

[luis-meteogrid]

@patrickkaleta don't forget about the study type field, it could be spain or europe, so we can create the study with the Spanish DataPackage or the European DataPackage.

[patrickkaleta]

@patrickkaleta don't forget about the study type field, it could be spain or europe, so we can create the study with the Spanish DataPackage or the European DataPackage.

Ok that's new to me. @rapto could you give me some details about that? So far all the Studies wege given the simulationtype = 118. I'm guessing that one is for Studies inside Spain. What number shall I give to Studies outside Spain?

Checking the API I can see two different Simulation types - CLARITY_STUDY_SPAIN and CLARITY_STUDY_EUROPE. But they don't have any numbers assigned to them.

[rapto]

@patrickkaleta The ids are exposed now at
https://clarity.saver.red/en/simmer/api/simulationtype/

        {
            "id": 118,
            "name": "Clarity Study - Spain",
            "locator": "CLARITY_STUDY_SPAIN"
        },
        {
            "id": 158,
            "name": "Estudio Clarity Europa",
            "locator": "CLARITY_STUDY_EUROPE"
        }

[patrickkaleta]

@patrickkaleta The ids are exposed now at
https://clarity.saver.red/en/simmer/api/simulationtype/

        {
            "id": 118,
            "name": "Clarity Study - Spain",
            "locator": "CLARITY_STUDY_SPAIN"
        },
        {
            "id": 158,
            "name": "Estudio Clarity Europa",
            "locator": "CLARITY_STUDY_EUROPE"
        }

Thanks @rapto. I've included the users parameter and made the simulationtype parameter depend on the selected country. It's not yet deployed, since I noticed a couple of things:

• our admins are not able to create TM studies or be included in them, since for them SSO is not enabled and therefore the TM doesn't recognize them as users. @fgeyer16 maybe we need to rethink how we handle admins and SSO. Would it be possible to give CSIS admins different roles in other systems, so that we could include them in the SSO?
• secondly, I think we should clean up the simulations inside the TM. I noticed that I get an error if I try to add a simulation with a reference field, that is apparently already used by another simulation (though I don't see it in the TM GUI using the api_access user, so I guess that it was created using a different user and now it's invisible to the api_access.

@rapto could you remove all those simulations in the TM that are not relevant? IMO only these simulations are relevant (listing the TM IDs):

• 93
• 94
• 92
• 91
• 66
• 77

[rapto]

I have deleted most of them, but left those we use for internal testing. I have set the reference to null so it does not interfere with the integration.

[luis-meteogrid]

Hi @patrickkaleta, have you been able to modify the CSIS and fix the problem? It is just because we are planning tests know. So if you want us to test anything just let us know.

[patrickkaleta]

Hi @patrickkaleta, have you been able to modify the CSIS and fix the problem? It is just because we are planning tests know. So if you want us to test anything just let us know.

Hi @luis-meteogrid. I deployed the changes - when transmitting Studies to the TM app, it now also sends the users and the simulation type is set depending on the selected country. Admin users are currently not included, since they are unknown to the TM (we will work on this issue together with @fgeyer16).

You can test the TM studies in the CSIS now - your users (luis, angela and meteogrid are not administrators, so they will have no problems).

Note

Since the selected country now plays an important role in determining the simulation type, new Studies are not sent over to the TM immediately upon their creation (as it was the case before), but only after a country has been selected in the Context tab of the Study.

[ghilbrae]

We've been testing the integration these past two days and we are stuck in the login screen both for existing studies or new ones. I created this one: https://csis.myclimateservice.eu/study/147/view/intro, just to ensure that there were no lingering issues from older studies. What I get is either the error screen coming from the error clarity.saver.red redirected you too many times. or a frame with an error from the TM, stating that there's an error but little else. Sometimes I've gotten a login screen from the TM on the frame but it is not consistent so I'm not yet certain on the conditions to replicate that outcome.

I've checked and I seem to be I'm logged in on both platforms with my user.

[luis-meteogrid]

@fgeyer16 It is seems that we have now another problem that may be related with the SSO,
https://profile.myclimateservices.eu/cas/login?service=https%3A%2F%2Fclarity.saver.red%2Faccounts%2Flogin%2F%3Fnext%3D%252Fstudies%252Freference%252F55%252F
can you please check if it is working properly?

[ghilbrae]

It seems that there's a problem with the URLs that are being used from the CSIS to call the TM. For example, the CSIS is using https://clarity.saver.red/studies/reference/145/ when it should be calling to
https://clarity.saver.red/study-hazard/reference/145/ (in case of hazard, but the same applies for the rest). The test study we have used is this one: https://csis.myclimateservice.eu/study/145/

Just a quick reminder of the pages that are connected to each of the EU-GL steps:

Maps:
https://clarity.saver.red/study-hazard/reference/777/
https://clarity.saver.red/study-exposition/reference/777/
https://clarity.saver.red/study-vulnerability/reference/777/
https://clarity.saver.red/study-risk/reference/777/

Tables:
https://clarity.saver.red/study-exposition/reference/777/elements_exposition/
https://clarity.saver.red/study-vulnerability/reference/777/elements_vulnerability/
https://clarity.saver.red/study-hazard/reference/777/elements_hazard/
https://clarity.saver.red/study-risk/reference/777/elements_risk/

[p-a-s-c-a-l]

What happens if you access the TM directly: https://clarity.saver.red/en/studies-spain/simulation/66/ ?
I get:

pascal.dihe No tiene autorización para ver esta página con las credenciales de acceso actuales. Por favor, salga del sistema y acceda con credenciales adecuadas.

Accessing the same page from within CSIS (https://csis.myclimateservice.eu/study/147/step/4333/view/maps) results in an infinite redirection loop of embedded the TM iFrame caused by CAS:

Apart from that my user is probably not authorised to access simulation No. 66, it seems that CAS redirection doesn't play well with iFrames?

[p-a-s-c-a-l]

The iFrame is correctly initialised with https://clarity.saver.red/study-hazard/reference/145/ and then redirected by TM to https://clarity.saver.red/study-hazard-spain/simulation/94/ and then again redirected by CAS (profiles.myclimateservices.eu) to https://clarity.saver.red/en/study-hazard-spain/simulation/94/ (not https://clarity.saver.red/study-hazard/reference/145/)

[p-a-s-c-a-l]

You can verify the original TM iFRame URL in the browser console:

[luis-meteogrid]

If you access directly either https://clarity.saver.red/studies/reference/55/ or https://clarity.saver.red/studies-spain/simulation/66/ you get:

[luis-meteogrid]

We have used this strategy in order that you don't have to change the url you access whether you are working in a Spanish-study or in an European one

[rapto]

The url https://clarity.saver.red/study-exposition/reference/_ID_/elements_exposition/ is now working

[rapto]

I am studying the auth loop. The cas auth ticket has lost its way (it shoud reuse the main frame cas ticket, but I have to read how that works).

[rapto]

OK; I have found that the stricter cookie policy in Django 2 sent Lax cookies. I have reset the policy to None, so now the session cookies are correctly allowed.

[rapto]

If there are any experts in the room, let me know if there is some way to reuse the csis login from the iframe. In the long run, it is clear that some rich api (that builds the interface) is the way to go.

[luis-meteogrid]

Great! Actually it is working although it is not yet connected to the desired pages in each section at least CSIS is already connecting to the transport module. I think we don't have much left. Come on, we're almost there.

[p-a-s-c-a-l]

although it is not yet connected to the desired pages in each section

O.K. Please let me know if any of the embedded iframes links to the wrong pages and let me know. I've configured them according to these links.

[luis-meteogrid]

It seems everything it is working properly. Let me check a few more cases before closing the issue. Thank you very much to you all.

[ghilbrae]

Hi @luis-meteogrid. I deployed the changes - when transmitting Studies to the TM app, it now also sends the users and the simulation type is set depending on the selected country. Admin users are currently not included, since they are unknown to the TM (we will work on this issue together with @fgeyer16).

We understand that the selection of the study type (Europe or Spain) should be done when creating a new study, an extra option along with the three already existing. This approach will also enable the users to create a study for Spain using the European variables available, as if it were any other country.

Since the selected country now plays an important role in determining the simulation type, new Studies are not sent over to the TM immediately upon their creation (as it was the case before), but only after a country has been selected in the Context tab of the Study.

We are testing the TM creation through the CSIS but we do not get any option to select the study type or the country. See https://csis.myclimateservice.eu/study/151/ as an example. Anyway, as stated above, we feel that it would be best to select the type beforehand. As an alternative an extra option could be added somewhere, like the Context tab you mentioned.

[p-a-s-c-a-l]

we do not get any option to select the study type or the country.

We've disabled changing the study type of existing studies as it introduced too many side effects. Country however can be changed in the study context:

[luis-meteogrid]

Fine, we have already checked this option and it is working

[luis-meteogrid]

One other issue, when you give access to other users to yours studies this information is not shared with the transport module, so the TM doesn't allow you to see the maps or tables related. It respond with a message signaling you are not allowed to access the study.

[patrickkaleta]

One other issue, when you give access to other users to yours studies this information is not shared with the transport module, so the TM doesn't allow you to see the maps or tables related. It respond with a message signaling you are not allowed to access the study.

Currently the TM only gets updated when changes directly inside the Study take place. Adding/changing the members of a Study are not considered as changes to the Study itself and therefore no update is pushed to the TM.

I'll have to look into it and see how we could trigger such an update in the members list. For the meantime, if you want to push an updated members list for a Study, just change the Study goal slightly and then the CSIS will push these changes (including the updated members list)

[luis-meteogrid]

One more thing, Is it possible to enlarge the embedded screen area for the TM? We are going to remove the menu bar but even doing that the map area remains still a bit small to work comfortably.