EPIC: As MaaS4Italy I would like to receive from the Open Data Hub a standard interface for planned and real-time mobility data (NeTEx and SIRI)

[rcavaliere]

In the scope of the new MaaS4Italy project the Open Data Hub is request to provide a standard interface for its mobility data, so that it can be integrated in a new MaaS architecture to be developed at national level. The following specification provides details between available Open Data Hub data and the requested fields.

240701_PianificazioneSviluppi.docx

The services to be considered are:

• static parking data, incl. e-charging stations (section 2.1.3)
• static sharing mobility data (section 2.3)
• real-time data (section 3.2)

Below the national specifications to be considered:

230711_Linee-guida-compilazione-NeTEx-IT-v.3.0.pdf

240502_SpecificaSIRI_v.1.0.3.pdf

DSRM-Architettura Target_Dati dinamici e sharing v1.2_signed.pdf
(relevant: paragraph 2.4)

NeTEx exports can be compared with an XSD before being published, available there https://github.com/5Tsrl/netex-italian-profile

The API is online at

• https://transmodel.api.opendatahub.testingmachine.eu
• https://transmodel.api.opendatahub.com

The URL schemes are :

• /netex
• /netex/parking
• /netex/sharing
• /siri-lite/fm
• /siri-lite/fm/parking
• /siri-lite/fm/sharing

Update 14.05.2024: STA has also requested to include in the exports bike parking services, since they are bookable and interesting for MaaS providers. I have amended in track change mode the internal specification document with indication on how to include this additional data. Please also note that there has been a new version of the SIRI national standard, some fields for parking have changed (removed).

[clezag]

After discussing this in person, some more details

• NeTEx and SIRI-FM services will be implemented as one or more custom REST endpoints, that produce the XML documents specified in the document on request
• The intended audience are STA backend services, the endpoints will not be advertised to the public (though accessible)
• No authentication will be necessary as everything is Open Data
• For NeTEx, we return the complete dataset as one document
• For SIRI-FM, we return only the document to one specific station/facility in the request.
• Where possible, we will validate against the XSD to guarantee formal correctness
• There is still lots of data missing in the Open Data Hub. The endpoints will follow the mapping in the specification as if the data existed, as it will be added later
• The provision that "all exports / end-points are made publicly available for all interested 3rd parties on the Open Data Hub" refers to a different, proxy-like service that will be implemented at a later time

[ohnewein]

The deadline for the first PoC is set to end of March 2024

[clezag]

@rcavaliere First PoC is up for static parking data:

https://sta-netex.opendatahub.testingmachine.eu/netex/parking

I've changed the ID a bit and added IT:OpenDataHub (it has to be globally unique), aside from that it should be faithful to the document.

Booleans currently default to false if no value in OpenDataHub is present, maybe we should implement ternary logic (e.g. leave it empty in netex where we don't have data).

XSD validation is also not implemented yet.

[rcavaliere]

@clezag that's look good, great work.
For the ID: this is not OK, since we need to put as prefix also the ISTAT code of the region. We could use OpenDataHub (in my view, not the proper solution, since the IDs should be related to the data, not to the platform), so something like it:ITH10:xxxx
I send you a technical specification for the IDs in the public transport, which is under implementation by STA.
230116_NeTExProfileSouthTyrol_GlobalID.pdf

[clezag]

I've updated the ID to the format

IT:ITH10:Parking:<sanitized scode>

where sanitized scode is the scode, with all characters that are not characters, numbers, hypens or underscores replaced with underscores.

e.g. station with scode=me: Karl-Wolf02 becomes IT:ITH10:Parking:me__Karl-Wolf02

This is how I understand the specification of NeTEx (4.2.1), and also matches with their examples.

The document from STA is not consistent with the NeTEx spec:

• country code must be capitalized, but in the examples is not
• the parens of the (edp) prefix are not actually part of the prefix, e.g. in the NeTEx spec it's just edp:IT:xxx
• the edp prefix is only required for frame identifiers, not for entities themselves

Origin is not necessary, since our scode is unique for station type. Each parking station is guaranteed to have a unique scode, and this way we avoid chaos when we deprecate/change origin in the future

[rcavaliere]

@clezag this is fine for me! Thanks for the input for the STA IDs

[clezag]

A PoC for sharing services (currently bike sharing BZ only) is now available:
https://sta-netex.opendatahub.testingmachine.eu/netex/sharing

A PoC for SIRI-FM is available here:
https://sta-netex.opendatahub.testingmachine.eu/siri/fm/115
the last part of the URL is the Open Data Hub scode, currently for Parking stations only

I think now would be a good time to talk both internally and with STA to define the next steps.

On our side, we mainly need to figure out how to integrate all the missing data (e.g. bike/car models, Operator address, zone polygons etc.).

With STA I'd like to finalize the workflow, format and interfaces of the API, and possible validation, as soon as possible

[rcavaliere]

@clezag wonderful work! I will analyze your work more in deep starting next week. I agree with you, we can start sharing this work with STA and define with them the next steps. I will contact them and put you in CC!

[rcavaliere]

Tasks to be completed:

• insertion of metadata (by @rcavaliere) -> https://noibz-my.sharepoint.com/:x:/r/personal/c_zagler_noi_bz_it/Documents/NeTEx%20missing%20data.xlsx?d=w0e6fbb54b4774d50b9a41e133893f3b5&e=4%3a3f920e90f1504a5491ead2526d7cb2b5&sharingv2=true&fromShare=true&at=9
• integration tests in main NeTEx export (by @clezag)

[rcavaliere]

@clezag I have started to insert the metadata. Parking areas and operators are completely filled.

[rcavaliere]

@clezag a remark: I would suggest to have at the end three different CompositeFrames:

• parking
• bike sharing (with all operators / services included, like for parking)
• car sharing

[rcavaliere]

@clezag an update on the SIRI interfaces. I have got an informal information that we would need to provide these interfaces using the "Lite" approach foreseen in the standard, which is much simpler. At the end it is exactly to provide end-points that a 3rd party can access with simple HTTP requests. Check as examples https://developer.entur.org/pages-real-time-api
So, perfectly in line with what you implemented. Let's wait for the formal specification to complete this part. It's probable that they will ask to implement in the API call some filtering options

[rcavaliere]

@clezag I have modified the user story description. The request is to add bike parking data (static -> NeTEx and dynamic -> SIRI) in the interfaces. Please note that we have also a new SIRI profile, some fields have been removed.
In case of questions let me know!

240514_PianificazioneSviluppi.docx
240502_SpecificaSIRI_v.1.0.3.pdf

[rcavaliere]

@clezag I have completed the insertion of metadata, including for bike parking. Let me know if there are some doubts, I would suggest to include this data as metadata for the reference station types, and then include in the mapping with the NeTEx export. Let me know when you are done so that I can check if in the NeTEx exports we have all fields available with the proper values.

[rcavaliere]

@clezag I have had also a look at the current exports. In general, it looks very good! There is something I would change.

• In the parking data of Skidata, in which we have the complexity of the ParkingFacility, we have the issue that names are not unique, so we have three parkings named as "Parcheggio Stazione Bressanone". I would change the Data Collector so to append here the information contained in the metadata field FacilityDescription (seems to be new, was not present when the Data Collector was implemented...)
• you already told me, please ensure to add all bike sharing services and not just the one of the Municipality of Bolzano
• in the mobility constraint zone, you just put a "fake" polygon, isn't it? We should create it here, probably on OSM we could easily extract such geographical data.

[clezag]

Just writing down what we discussed in person in addition to the points above:

Metadata

• Parking will be added by data collectors . Where already implemented, supplied via spreadsheet
  • parking-offstreet-meranobolzano
  • parking-tn (trento + rovereto)
  • (blocked due to endpoint not working) parking-offstreet-sta
  • parking-offstreet-a22
  • bike-boxes
• Operators will be added by the netex endpoint on the fly
• Bike models will be added by data collector (hardcoded)
• Car models will be added by data collector (configurable via csv or similar)

Bike Parking

• Bike parking station type & mapping to be added to parking export

[rcavaliere]

@clezag I have modified the user story description. The request is to add bike parking data (static -> NeTEx and dynamic -> SIRI) in the interfaces. Please note that we have also a new SIRI profile, some fields have been removed. In case of questions let me know!

240514_PianificazioneSviluppi.docx 240502_SpecificaSIRI_v.1.0.3.pdf

@clezag I have seen that in the document the integration modalities for car sharing data in the SIRI interface was missing. I have updated the document.

240521_PianificazioneSviluppi.docx

[rcavaliere]

@clezag an important update in relation to the implementation of the SIRI interfaces. We have received the attached documentation, available also in the description of this epic.
Relevant here is at the beginning of paragraph 2.4: we have to provide two end-points, one for parking and one for sharing services. You can see the names of the end-points we have to implement and what kind of input parameters we have to support.
Let me know in case of issues!

DSRM-Architettura Target_Dati dinamici e sharing v1.2_signed.pdf

[clezag]

@rcavaliere Just some progress updates:

static parking is now fully implemented:

• bikeparking implemented
• missing metadata added to data collectors and mapped in the export
• operator details configurable via config file in repo
• operator FBK is split into FBK-Trento and FBK-Rovereto on basis of smetadata.municipality
• skidata is excluded due to data collector not working

In general I think it's best to limit the dataset to what we know works, and only add new providers/origins explicitly.
To that end I have added a static config file in the repo where you have to explicitly add new origins in the future.
I think this will be better than having the export fail every few months due to new data and someone having to go in there and fix it.

[rcavaliere]

@clezag agree. I have just checked, I inserted all metadata here: https://noibz-my.sharepoint.com/:x:/r/personal/c_zagler_noi_bz_it/_layouts/15/Doc.aspx?sourcedoc=%7B0E6FBB54-B477-4D50-B9A4-1E133893F3B5%7D&file=NeTEx%20missing%20data.xlsx&fromShare=true&action=default&mobileredirect=true

Are you checking this file, isn't it? Let me know if there is something missing.

Do you have some new NeTEx exports that I can check?

Thanks for your great work here

[clezag]

@rcavaliere
Yes I've used this file.

• Bike sharing bz information about the locks is missing. I've set it to false for now
• For bike sharing merano I have assumed they are all muscular (judging from the images) with a basket, lamp and lock

On the static parking export, I've implemented all the missing points, so from my side I consider it complete. Only skidata is missing, which IMO is a "wontfix" until the data collector works again or better is in production.
You can take a look here:
https://sta-netex.opendatahub.testingmachine.eu/netex/parking

Sharing is here:
https://sta-netex.opendatahub.testingmachine.eu/netex/sharing
Bike sharing bolzano, which is the only one implemented, should now be complete.
I've implemented the missing metadata (bicycles, bike models, operators, service constraint polygon).

Still missing are the other bike sharing providers, the car sharing, and then the new SIRI developments

[rcavaliere]

@clezag let's try to close this first implementation iteration.

I have looked at the NeTEx parking export, in my opinion it's nearly done. I would just one thing: I would consider as names for the operators not their origin, but a "standard name". I have made a proposal in the shared Excel file, see here in red:

Can you match the correspondent fields with these values?

[clezag]

@rcavaliere I've added the mapping, but I'm not completely sure I understood:
Should I also use the operator name as ID/key? e.g. should "skidata" and "bicincitta" refer to the same single operator with ID = STA?. Or should I maintain two operators, who happen to have the same name and address?

[rcavaliere]

@clezag thanks! I would also change the IDs accordingly, so that we have consistency in the data.Yes, skidata and bicincittà have to refer to the same operator STA

[leonardehrenfried]

Hi, I'm the OpenTripPlanner developer that implements the consumer for parking.

I echo @clezag's doubt about Siri-Lite meaning that it's JSON.

Entur also publishes their SX, VM and ET feeds as what they call "HTTP GET (Standard SIRI 2.0 Lite)" and they are using XML.

Example: https://api.entur.io/realtime/v1/rest/sx

I would love to see some documentation that explains the mapping from XML to JSON. Is it available?

[leonardehrenfried]

@rcavaliere has posted this document https://github.com/user-attachments/files/16069269/240620_RAP.-.Specifiche.interfacce.dati.dinamici.pdf which says

Le risposte previste da questo endpoint anche in questo caso possono essere sia in
formato XML che in formato JSON e la decisione viene presa in base all’ header “Accept”
fornito nella richiesta: il DSRM predilige il secondo formato tendenzialmente più snello e
meno verboso, ma il contenuto informativo è assolutamente corrispondente al SIRI profilo
italiano descritto nelle

Google Translate suggests that this means you have a choice between XML and JSON.

Why do I prefer XML? Because OTP already loads the complete Siri XML schema and makes converting it to Java trivial.

[rcavaliere]

@leonardehrenfried @clezag I think we could offer without issues also an XML export. The indication to implement JSON was given during several meetings with the implementers of this national platform.

[clezag]

@leonardehrenfried
The JSON implementation has been done in a best effort way, because the only reference I could find was some French implementation from 2016.
It should be fairly easy however to provide a XML version again (I'll base it on the Accept header as outlined in the documentation you quoted).

Have you looked into the content itself? Is it conforming the the "SIRI standard" in OTP? From the little experience I've had with this SIRI/Netex stuff, the word "standard" is to be understood loosely, to say the least.

@rcavaliere

add an authentication layer. They expect to get from us a client-id and client-secret to make the pull requests

Do they know it's open data and we're not doing any authorization anyway? This is a lot of effort for something that doesn't do anything. I suggest we create them some keycloak credentials, they can then request and supply the authentication token, but we don't implement anything that looks at the token and validates it. If further down the line we need authentication/authorization, we can add the authentication code.

it's better to split the end-points in two, one for parking (siri-lite/fm/parking... and siri-lite/fm/sharing)

It already is implemented this way. The only thing I will change is making the path siri-lite instead of siri for all our siri endpoints

we should be able to implement all possible filtering possibilities by making the API call, as we do for our mobility API

To be clear, only the filters outlined in the document?

[leonardehrenfried]

Have you looked into the content itself? Is it conforming the the "SIRI standard" in OTP? From the little experience I've had with this SIRI/Netex stuff, the word "standard" is to be understood loosely, to say the least.

This is also my impression. NeTEx/Siri are really (very big) XML schemas and it's up to the implementers to fill it with life and agree on a subset of it. Most of them call it a profile, so we have EPIP, Nordic profile, UK profile. There appears to be huge room for interpretation, which is a shame.

What I miss most is some form of community where you can ask "Here is my XML, am I using it how it is intended?".

To answer your question: OTP doesn't use the JSON version of Siri so it's hard to say if it conforms to a schema but if I translate it back to XML in my head it looks correct on first glance. However, I would prefer to try it out with a proper parser and then give you feedback.

[clezag]

@leonardehrenfried
All good, I'll include the XML output in the next round of implementations and ping you here once it's up

[rcavaliere]

Do they know it's open data and we're not doing any authorization anyway? This is a lot of effort for something that doesn't do anything. I suggest we create them some keycloak credentials, they can then request and supply the authentication token, but we don't implement anything that looks at the token and validates it. If further down the line we need authentication/authorization, we can add the authentication code.

@clezag thanks for your feedback. OK, let's push our message that since all this data is licensed CC0 we won't implement any security mechanism in front of the end-points

[rcavaliere]

To be clear, only the filters outlined in the document?

@clezag, yes only these filters are really necessary!

[leonardehrenfried]

@clezag I tried out the Netex parking feed today and it conformed flawlessly to the schema (that OTP uses). If the Siri XML is of the same quality, I expect it no problems there as well.

[clezag]

@rcavaliere
Fixed the parking space metadata issue

@leonardehrenfried
I've implemented the SIRI XML feed again.

URL scheme is:

siri-lite/fm
siri-lite/fm/sharing
siri-lite/fm/parking

It defaults to JSON, you can force XML via Accept: application/xml or by setting (and I would leave this an undocumented feature for testing convenience) query string ?format=xml

Let me know if you run into any issues with the XML

[rcavaliere]

@clezag both are siri-lite, the difference with SIRI "standard" is that there is this simple URL, without a publish / subscribe mechanism at the beginning between data producer and data consumer

[clezag]

@rcavaliere I've edited my comment above, the Accept header and URL scheme are now implemented according to the spec document

[rcavaliere]

@clezag can you please put in the main description of this user story the consolidated end-points for the NeTEx and SIRI exports? I have now a bit confusion about what are the correct URLs :-) For the SIRI end-points, I would suggest, if you already didn't fix this, to change "sta-netex" at the beginning and just put "siri.opendatahub...." as mentioned above. Thanks a lot!

[clezag]

@rcavaliere I'm implementing the filters, and I'm not sure what "datasetId" is supposed to be. To which fields does it map (either in OpenDataHub or netex)

[rcavaliere]

@rcavaliere I'm implementing the filters, and I'm not sure what "datasetId" is supposed to be. To which fields does it map (either in OpenDataHub or netex)

@clezag can you please tell me exactly where this kind of filter is mentioned? I can not find it...

[clezag]

@rcavaliere I'm implementing the filters, and I'm not sure what "datasetId" is supposed to be. To which fields does it map (either in OpenDataHub or netex)

@clezag can you please tell me exactly where this kind of filter is mentioned? I can not find it...

https://github.com/user-attachments/files/16069269/240620_RAP.-.Specifiche.interfacce.dati.dinamici.pdf
2.5

[rcavaliere]

@clezag I would implement just the facilityRef filter. The datasetId would be useful if we have in the SIRI feed a field specifying the data provider for certain records, which would correspond to our "sorigin" field. But it's not clear to me how we could implement this...

[clezag]

@rcavaliere

I've implemented the filters (aside from datasetId).
Due to implementation details, and because we also have a combined endpoint for parking and sharing, all filters work on all SIRI endpoints (e.g. you can filter realtime parking data with gps coordinates and by operator, and have maxSize everywhere).

Regarding the endpoint URL, since the service is running both netex and siri at the same time, what do you think of making the endpoint transmodel.api.opendatahub.com?
You would then do transmodel.api.opendatahub.com/netex/... and transmodel.api.opendatahub.com/siri-lite/...

I can also set it up like you suggested, and have two URLs,

• netex.api.opendatahub.com/parking/...
• siri.api.opendatahub.com/lite/fm/parking

but it's a more complicated setup with custom proxy configuration, separate swagger definitions and so on, because we will fake a single service being two separate ones

I would prefer one single URL, and then have the /netex/ and /siri-lite/ paths to differentiate between services, but the decision is yours

[rcavaliere]

@clezag thanks for your work. I agree, let's have one single end-point called transmodel.api.opendatahub.com
Can you set it up today, both in a testing and productive mode? Testing end-point should therefore be transmodel.api.opendatahub.testingmachine.eu

[clezag]

@clezag can you please put in the main description of this user story the consolidated end-points for the NeTEx and SIRI exports? I have now a bit confusion about what are the correct URLs :-) For the SIRI end-points, I would suggest, if you already didn't fix this, to change "sta-netex" at the beginning and just put "siri.opendatahub...." as mentioned above. Thanks a lot!

@rcavaliere
I've added the new endpoints that are now online to the main story.
I've also changed this repo's name accordingly

I'm still working on providing swagger etc. which I will also link in the main story (should just be the / base URL as in our other APIs)

One minor thing: please note that in production, using the SIRI maxDistance filter currently results in an error. This is because it relies on a new ninja API feature that has not gone in to production yet (should do so early next week)

In the sprint meeting today, we also emphasized to not forget to link and document this API on out main channels and websites once it has reached some level of stability (when it goes into testing with the NAP).

[clezag]

@leonardehrenfried
the API has been migrated to

https://transmodel.api.opendatahub.com (production)
https://transmodel.api.opendatahub.testingmachine.eu (testing)

and the old URL will be dismissed shortly

[rcavaliere]

@clezag can you please put here some examples on how you can make some API calls with active filters? I was not able alone to reproduce this. Thanks!

[clezag]

@rcavaliere
There was a bug so that latitude and longitude were reversed, now it's fixed.
The change to the ninja API are now in production as well, so all filters should work:

• limit result to 10 records: https://transmodel.api.opendatahub.com/siri-lite/fm/maxSize=10
• Filter for a specific facility: https://transmodel.api.opendatahub.com/siri-lite/fm/parking?facilityRef=IT:ITH10:Parking:TRENTO_areaexsitviacanestrinip1
• This filter can be repeated to filter for two facilities (not required by spec): https://transmodel.api.opendatahub.com/siri-lite/fm/parking?facilityRef=IT:ITH10:Parking:TRENTO_areaexsitviacanestrinip1&facilityRef=IT:ITH10:Parking:ROVERETO_mart
• Filter within 3 km from a point (in Merano): https://transmodel.api.opendatahub.com/siri-lite/fm/sharing?lat=46.667439&lon=11.149468&maxDistance=3000
• Filter for operator bikesharing bolzano: https://transmodel.api.opendatahub.com/siri-lite/fm/sharing?operators=IT:ITH10:Operator:Municipality_of_Bolzano_bikesharing
• Filter for operators merano + bolzano (list syntax in spec): https://transmodel.api.opendatahub.com/siri-lite/fm/sharing?operators=IT:ITH10:Operator:Municipality_of_Bolzano_bikesharing,IT:ITH10:Operator:Municipality_of_Merano_bikesharing
• Filter for operators merano + bolzano (repeated parameter syntax, not required by spec): https://transmodel.api.opendatahub.com/siri-lite/fm/sharing?operators=IT:ITH10:Operator:Municipality_of_Bolzano_bikesharing&operators=IT:ITH10:Operator:Municipality_of_Merano_bikesharing

[leonardehrenfried]

Note to myself, the Netex parking feed is now at: https://transmodel.api.opendatahub.testingmachine.eu/netex/parking

[clezag]

Note to myself, the Netex parking feed is now at: https://transmodel.api.opendatahub.testingmachine.eu/netex/parking

Just FYI, this is the testing link, production is https://transmodel.api.opendatahub.com/netex

[clezag]

Openapi spec is now up:

https://transmodel.api.opendatahub.com/

Feel free to suggest additional documentation we could include.

[leonardehrenfried]

@clezag I'm currently working on consuming the Siri feed. What does the following mean?

		<FacilityCondition>
				<FacilityRef>IT:ITH10:Parking:IT_RBK_EBZ002_3</FacilityRef>
				<FacilityStatus>
					<Status>notAvailable</Status>
				</FacilityStatus>
				<MonitoredCounting>
					<CountingType>presentCount</CountingType>
					<CountedFeatureUnit>devices</CountedFeatureUnit>
					<Count>1</Count>
				</MonitoredCounting>
			</FacilityCondition>

This is a parking lot with RechargingAvalable=true, so presumably that's a charger for an electric car. But does <Status>notAvailable</Status> mean that the status is unknown or that the charger is currently not available, so cars cannot park there?

[clezag]

@clezag I'm currently working on consuming the Siri feed. What does the following mean?

		<FacilityCondition>
				<FacilityRef>IT:ITH10:Parking:IT_RBK_EBZ002_3</FacilityRef>
				<FacilityStatus>
					<Status>notAvailable</Status>
				</FacilityStatus>
				<MonitoredCounting>
					<CountingType>presentCount</CountingType>
					<CountedFeatureUnit>devices</CountedFeatureUnit>
					<Count>1</Count>
				</MonitoredCounting>
			</FacilityCondition>

This is a parking lot with RechargingAvalable=true, so presumably that's a charger for an electric car. But does <Status>notAvailable</Status> mean that the status is unknown or that the charger is currently not available, so cars cannot park there?

@leonardehrenfried

The specs of the mapping are [here, section 3.2.x] (https://github.com/noi-techpark/transmodel-api/blob/main/documentation/240603_PianificazioneSviluppi.docx)

TL;DR: In general, the status is to be interpreted as the state of parking, not charging availability. In your case the one parking space is occupied (presentCount=1, capacity=1), so parking is 'notAvailable'

For the specific parking station you are looking at, we happen to deduce it from the charging state, because that's all we have, but a SIRI consumer wouldn't know about that.
Having recharging can be a feature of a parking area, but it's not the thing we are reporting about via SIRI.

@rcavaliere correct me if I'm wrong here, you wrote the spec 😉

[rcavaliere]

@clezag the feedback is correct. At the end the e-charging is location is not available in the sense that a person can not make a charging operation, or is not available because somebody else is charging the car. In both cases the parking is to be considered not available, and that's the reason why, at the moment, we do not differentiate these situations. Also for routing purposes, such parking shouldn't be considered in such situations.