New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NMBS endpoint changed: Error 500 when querying connections #319

Closed
Bertware opened this Issue Oct 16, 2017 · 7 comments

Comments

Projects
None yet
2 participants
@Bertware
Contributor

Bertware commented Oct 16, 2017

PHP message: PHP Warning:  SimpleXMLElement::__construct(): Entity: line 9: parser error : Opening and ending tag mismatch: hr line 7 and body in /var/www/api.irail.be/api/data/NMBS/connections.php on line 198
PHP message: PHP Warning:  SimpleXMLElement::__construct():

Cause by HTML

Not Found

The requested URL /jp/sncb-nmbs-routeplanner/extxml.exe was not found on this server.
Apache/2.2.15 (Red Hat) Server at www.belgianrail.be Port 80

@Bertware Bertware self-assigned this Oct 16, 2017

@Bertware Bertware changed the title from Error 500 when querying connections to NMBS endpoint changed: Error 500 when querying connections Oct 16, 2017

@Bertware

This comment has been minimized.

Show comment
Hide comment
@Bertware

Bertware Oct 16, 2017

Contributor

Example alternative: using the mobile API enpoint. We'd need to decipher it though.
This is a route Halle -> Gent Sint Pieters:

POST http://www.belgianrail.be/jp/sncb-nmbs-routeplanner/mgate.exe HTTP/1.1
User-Agent: SNCB/302132 (Android_5.0.2) Dalvik/2.1.0 (Linux; U; Android 5.0.2; HTC One Build/LRX22G)
Content-Type: text/plain;charset=UTF-8
Host: www.belgianrail.be
Connection: Keep-Alive
Accept-Encoding: gzip
Content-Length: 807


{"auth":{"aid":"sncb-mobi","type":"AID"},"client":{"id":"SNCB","name":"NMBS","os":"Android 5.0.2","type":"AND","ua":"SNCB/302132 (Android_5.0.2) Dalvik/2.1.0 (Linux; U; Android 5.0.2; HTC One Build/LRX22G)","v":302132},"lang":"nld","svcReqL":[{"cfg":{"polyEnc":"GPA"},"meth":"TripSearch","req":{"arrLocL":[{"lid":"A=1@O=Gent-Sint-Pieters@X=3710675@Y=51035896@U=80@L=008892007@B=1@p=1429490515@","name":"Gent-Sint-Pieters","type":"S"}],"depLocL":[{"lid":"A=1@O=Halle@X=4240634@Y=50733931@U=80@L=008814308@B=1@p=1481329402@n=ac.1=GA@","name":"Halle","type":"S"}],"jnyFltrL":[{"mode":"BIT","type":"PROD","value":"11101111111111"}],"outDate":"20170315","outTime":"192345","economic":false,"extChgTime":-1,"getIST":false,"getPasslist":true,"getPolyline":true,"liveSearch":false}}],"ver":"1.11","formatted":false}
HTTP/1.1 200 OK
Date: Wed, 15 Mar 2017 18:23:54 GMT
Server: Apache/2.2.15 (Red Hat)
Content-Length: 12276
Connection: close
Content-Type: application/json; charset=utf-8

{
  "ver": "1.11",
  "lang": "nld",
  "id": "29k64uyqwwmjxic4",
  "svcResL": [
    {
      "id": "",
      "meth": "TripSearch",
      "err": "OK",
      "res": {
        "common": {
          "locL": [
            {
              "lid": "A=1@O=Halle@X=4240634@Y=50733931@U=80@L=8814308@",
              "type": "S",
              "name": "Halle",
              "icoX": 1,
              "extId": "8814308",
              "crd": {
                "x": 4240634,
                "y": 50733931,
                "z": 0
              },
              "pCls": 68,
              "rRefL": [
                0,
                1,
                2
              ]
            },
            {
              "lid": "A=1@O=Gent-Sint-Pieters@X=3710675@Y=51035897@U=80@L=8892007@",
              "type": "S",
              "name": "Gent-Sint-Pieters",
              "icoX": 1,
              "extId": "8892007",
              "crd": {
                "x": 3710675,
                "y": 51035897,
                "z": 0
              },
              "pCls": 100,
              "rRefL": [
                0,
                1,
                2
              ]
            },
            {
              "lid": "A=1@O=Brussel-Zuid@X=4336531@Y=50835707@U=80@L=8814001@",
              "type": "S",
              "name": "Brussel-Zuid",
              "icoX": 1,
              "extId": "8814001",
              "crd": {
                "x": 4336531,
                "y": 50835707,
                "z": 0
              },
              "pCls": 69,
              "rRefL": [
                0,
                1,
                2
              ]
            }
          ],
          "prodL": [
...
          ],
          "opL": [
            
          ],
          "remL": [
            {
              "type": "A",
              "code": "GA",
              "prio": 999,
              "icoX": 0,
              "txtN": "Belgium stations"
            },
            {
              "type": "A",
              "code": "GH",
              "icoX": 0,
              "txtN": ""
            },
            {
              "type": "A",
              "code": "LI",
              "icoX": 0,
              "txtN": ""
            }
          ],
          "icoL": [
            {
              "res": "INFO"
            },
            {
              "res": "STA_SNCB"
            },
            {
              "res": "IC"
            }
          ]
        },
        "outConL": [
          {
            "cid": "C-0",
            "date": "20170315",
            "dur": "005300",
            "chg": 1,
            "sDays": {
              "sDaysR": "Ma - Vr",
              "sDaysI": "niet 17. Apr, 1., 25. Mei, 5. Juni, 21. Juli, 15. Aug, 1. Nov",
              "sDaysB": "7CF9F3E7CF9F387CF9A3E7CF9F3E7CF8F3E3CF9F3A7C79F3E7CF9F3C7CF9F2E7CF9F3E7CF9F3E7CF9B3E7CF9F3E0"
            },
            "dep": {
              "locX": 0,
              "dPlatfS": "4",
              "dTimeS": "192700",
              "dTimeR": "192700",
              "dProgType": "PROGNOSED"
            },
            "arr": {
              "locX": 1,
              "aPlatfS": "12",
              "aTimeS": "202000",
              "aTimeR": "202100",
              "aProgType": "PROGNOSED"
            },
            "secL": [
              {
                "type": "JNY",
                "icoX": 2,
                "dep": {
                  "locX": 0,
                  "dPlatfS": "4",
                  "dTimeS": "192700",
                  "dTimeR": "192700",
                  "dProgType": "PROGNOSED"
                },
                "arr": {
                  "locX": 2,
                  "aPlatfS": "15",
                  "aTimeS": "193600",
                  "aTimeR": "193700",
                  "aProgType": "PROGNOSED"
                },
                "jny": {
                  "jid": "1|869|0|80|15032017",
                  "prodX": 0,
                  "dirTxt": "Luik-Guillemins",
                  "isRchbl": true,
                  "stopL": [
                    {
                      "locX": 0,
                      "idx": 18,
                      "aTimeS": "192600",
                      "aProgType": "PROGNOSED",
                      "dProdX": 0,
                      "dTimeS": "192700",
                      "dTimeR": "192700",
                      "dProgType": "PROGNOSED"
                    },
                    {
                      "locX": 2,
                      "idx": 23,
                      "aProdX": 0,
                      "aTimeS": "193600",
                      "aTimeR": "193700",
                      "aProgType": "PROGNOSED",
                      "dTimeS": "193900",
                      "dProgType": "PROGNOSED"
                    }
                  ],
               
           ... 
Contributor

Bertware commented Oct 16, 2017

Example alternative: using the mobile API enpoint. We'd need to decipher it though.
This is a route Halle -> Gent Sint Pieters:

POST http://www.belgianrail.be/jp/sncb-nmbs-routeplanner/mgate.exe HTTP/1.1
User-Agent: SNCB/302132 (Android_5.0.2) Dalvik/2.1.0 (Linux; U; Android 5.0.2; HTC One Build/LRX22G)
Content-Type: text/plain;charset=UTF-8
Host: www.belgianrail.be
Connection: Keep-Alive
Accept-Encoding: gzip
Content-Length: 807


{"auth":{"aid":"sncb-mobi","type":"AID"},"client":{"id":"SNCB","name":"NMBS","os":"Android 5.0.2","type":"AND","ua":"SNCB/302132 (Android_5.0.2) Dalvik/2.1.0 (Linux; U; Android 5.0.2; HTC One Build/LRX22G)","v":302132},"lang":"nld","svcReqL":[{"cfg":{"polyEnc":"GPA"},"meth":"TripSearch","req":{"arrLocL":[{"lid":"A=1@O=Gent-Sint-Pieters@X=3710675@Y=51035896@U=80@L=008892007@B=1@p=1429490515@","name":"Gent-Sint-Pieters","type":"S"}],"depLocL":[{"lid":"A=1@O=Halle@X=4240634@Y=50733931@U=80@L=008814308@B=1@p=1481329402@n=ac.1=GA@","name":"Halle","type":"S"}],"jnyFltrL":[{"mode":"BIT","type":"PROD","value":"11101111111111"}],"outDate":"20170315","outTime":"192345","economic":false,"extChgTime":-1,"getIST":false,"getPasslist":true,"getPolyline":true,"liveSearch":false}}],"ver":"1.11","formatted":false}
HTTP/1.1 200 OK
Date: Wed, 15 Mar 2017 18:23:54 GMT
Server: Apache/2.2.15 (Red Hat)
Content-Length: 12276
Connection: close
Content-Type: application/json; charset=utf-8

{
  "ver": "1.11",
  "lang": "nld",
  "id": "29k64uyqwwmjxic4",
  "svcResL": [
    {
      "id": "",
      "meth": "TripSearch",
      "err": "OK",
      "res": {
        "common": {
          "locL": [
            {
              "lid": "A=1@O=Halle@X=4240634@Y=50733931@U=80@L=8814308@",
              "type": "S",
              "name": "Halle",
              "icoX": 1,
              "extId": "8814308",
              "crd": {
                "x": 4240634,
                "y": 50733931,
                "z": 0
              },
              "pCls": 68,
              "rRefL": [
                0,
                1,
                2
              ]
            },
            {
              "lid": "A=1@O=Gent-Sint-Pieters@X=3710675@Y=51035897@U=80@L=8892007@",
              "type": "S",
              "name": "Gent-Sint-Pieters",
              "icoX": 1,
              "extId": "8892007",
              "crd": {
                "x": 3710675,
                "y": 51035897,
                "z": 0
              },
              "pCls": 100,
              "rRefL": [
                0,
                1,
                2
              ]
            },
            {
              "lid": "A=1@O=Brussel-Zuid@X=4336531@Y=50835707@U=80@L=8814001@",
              "type": "S",
              "name": "Brussel-Zuid",
              "icoX": 1,
              "extId": "8814001",
              "crd": {
                "x": 4336531,
                "y": 50835707,
                "z": 0
              },
              "pCls": 69,
              "rRefL": [
                0,
                1,
                2
              ]
            }
          ],
          "prodL": [
...
          ],
          "opL": [
            
          ],
          "remL": [
            {
              "type": "A",
              "code": "GA",
              "prio": 999,
              "icoX": 0,
              "txtN": "Belgium stations"
            },
            {
              "type": "A",
              "code": "GH",
              "icoX": 0,
              "txtN": ""
            },
            {
              "type": "A",
              "code": "LI",
              "icoX": 0,
              "txtN": ""
            }
          ],
          "icoL": [
            {
              "res": "INFO"
            },
            {
              "res": "STA_SNCB"
            },
            {
              "res": "IC"
            }
          ]
        },
        "outConL": [
          {
            "cid": "C-0",
            "date": "20170315",
            "dur": "005300",
            "chg": 1,
            "sDays": {
              "sDaysR": "Ma - Vr",
              "sDaysI": "niet 17. Apr, 1., 25. Mei, 5. Juni, 21. Juli, 15. Aug, 1. Nov",
              "sDaysB": "7CF9F3E7CF9F387CF9A3E7CF9F3E7CF8F3E3CF9F3A7C79F3E7CF9F3C7CF9F2E7CF9F3E7CF9F3E7CF9B3E7CF9F3E0"
            },
            "dep": {
              "locX": 0,
              "dPlatfS": "4",
              "dTimeS": "192700",
              "dTimeR": "192700",
              "dProgType": "PROGNOSED"
            },
            "arr": {
              "locX": 1,
              "aPlatfS": "12",
              "aTimeS": "202000",
              "aTimeR": "202100",
              "aProgType": "PROGNOSED"
            },
            "secL": [
              {
                "type": "JNY",
                "icoX": 2,
                "dep": {
                  "locX": 0,
                  "dPlatfS": "4",
                  "dTimeS": "192700",
                  "dTimeR": "192700",
                  "dProgType": "PROGNOSED"
                },
                "arr": {
                  "locX": 2,
                  "aPlatfS": "15",
                  "aTimeS": "193600",
                  "aTimeR": "193700",
                  "aProgType": "PROGNOSED"
                },
                "jny": {
                  "jid": "1|869|0|80|15032017",
                  "prodX": 0,
                  "dirTxt": "Luik-Guillemins",
                  "isRchbl": true,
                  "stopL": [
                    {
                      "locX": 0,
                      "idx": 18,
                      "aTimeS": "192600",
                      "aProgType": "PROGNOSED",
                      "dProdX": 0,
                      "dTimeS": "192700",
                      "dTimeR": "192700",
                      "dProgType": "PROGNOSED"
                    },
                    {
                      "locX": 2,
                      "idx": 23,
                      "aProdX": 0,
                      "aTimeS": "193600",
                      "aTimeR": "193700",
                      "aProgType": "PROGNOSED",
                      "dTimeS": "193900",
                      "dProgType": "PROGNOSED"
                    }
                  ],
               
           ... 
@Bertware

This comment has been minimized.

Show comment
Hide comment
@Bertware

Bertware Oct 16, 2017

Contributor

A temporary implementation is in place. Following functions are not available at this moment:

  • Changed platform indication
  • Service alerts
  • Arrival/Departure selection

Anyone who wants to help out debugging the new implementation is welcome to do so!

Contributor

Bertware commented Oct 16, 2017

A temporary implementation is in place. Following functions are not available at this moment:

  • Changed platform indication
  • Service alerts
  • Arrival/Departure selection

Anyone who wants to help out debugging the new implementation is welcome to do so!

@derhuerst

This comment has been minimized.

Show comment
Hide comment
@derhuerst

derhuerst Oct 16, 2017

FYI I wrote two mobile HAFAS API clients in JavaScript, so maybe the parsing logic is helpful to you. The client doesn't parse everything the API provides, but most of it, and has been running fine for the past 2 years.

Specifically, look at the following files:

Let me know if I can assist you with HAFAS-mobile-API-related problems, as long as they're not specific to Belgian Rail.

derhuerst commented Oct 16, 2017

FYI I wrote two mobile HAFAS API clients in JavaScript, so maybe the parsing logic is helpful to you. The client doesn't parse everything the API provides, but most of it, and has been running fine for the past 2 years.

Specifically, look at the following files:

Let me know if I can assist you with HAFAS-mobile-API-related problems, as long as they're not specific to Belgian Rail.

@Bertware

This comment has been minimized.

Show comment
Hide comment
@Bertware

Bertware Oct 16, 2017

Contributor

@derhuerst Thanks! I've looked around for a bit, and it seems really helpful (especially the implementations for vbb/db give me some more insights on how to use it)

Do you happen to know how we can detect a platform change? Platforms are in aPlatfS, but is there also something like aPlatfR for 'real' data in case it changed? PlatfR stands for realtime data!

Have you seen the search parameter liveSearch before? Or do you happen to know how to search journeys by time of arrival (instead of departure time)

Contributor

Bertware commented Oct 16, 2017

@derhuerst Thanks! I've looked around for a bit, and it seems really helpful (especially the implementations for vbb/db give me some more insights on how to use it)

Do you happen to know how we can detect a platform change? Platforms are in aPlatfS, but is there also something like aPlatfR for 'real' data in case it changed? PlatfR stands for realtime data!

Have you seen the search parameter liveSearch before? Or do you happen to know how to search journeys by time of arrival (instead of departure time)

@Bertware

This comment has been minimized.

Show comment
Hide comment
@Bertware

Bertware Oct 16, 2017

Contributor

According to traffic by the official NMBS app:

  • The vehicle field should be 01101111000111 for only trains
  • When searching by "arrive at", the following field should be added: "outFrwd": false
  • Example output when there are 'alerts' : https://pastebin.com/bfq2trTz

This should be enough to resolve all our issues with the new parser

  • Show all trains, but no buses etc
  • Allow searching by arrival time
  • Show alerts when requested using the &alerts=true parameter
Contributor

Bertware commented Oct 16, 2017

According to traffic by the official NMBS app:

  • The vehicle field should be 01101111000111 for only trains
  • When searching by "arrive at", the following field should be added: "outFrwd": false
  • Example output when there are 'alerts' : https://pastebin.com/bfq2trTz

This should be enough to resolve all our issues with the new parser

  • Show all trains, but no buses etc
  • Allow searching by arrival time
  • Show alerts when requested using the &alerts=true parameter
@derhuerst

This comment has been minimized.

Show comment
Hide comment
@derhuerst

derhuerst Oct 16, 2017

Have you seen the search parameter liveSearch before? Or do you happen to know how to search journeys by time of arrival (instead of departure time)

I haven't, sorry. What you can do though is use e.g. mitmproxy to redirect traffic of the common apps through your machine and record all requests. By doing this, you can reverse-engineer the parameters.

derhuerst commented Oct 16, 2017

Have you seen the search parameter liveSearch before? Or do you happen to know how to search journeys by time of arrival (instead of departure time)

I haven't, sorry. What you can do though is use e.g. mitmproxy to redirect traffic of the common apps through your machine and record all requests. By doing this, you can reverse-engineer the parameters.

@Bertware

This comment has been minimized.

Show comment
Hide comment
@Bertware

Bertware Oct 16, 2017

Contributor

@derhuerst Just figured it out how to search on arrival time using fiddler :) Still wondering what that liveSearch parameter is, it's always included, always false, and doesn't seem to affect anything. I've set it to true atm for our application in the hopes that it will return fresher data or anything.

Contributor

Bertware commented Oct 16, 2017

@derhuerst Just figured it out how to search on arrival time using fiddler :) Still wondering what that liveSearch parameter is, it's always included, always false, and doesn't seem to affect anything. I've set it to true atm for our application in the hopes that it will return fresher data or anything.

Bertware added a commit that referenced this issue Oct 18, 2017

Bertware added a commit that referenced this issue Oct 19, 2017

Improve new connections parser: Fix alerts, use arrived and left as k…
…eywords for status, parse intermediate stops but don't include them in API output yet #319

Bertware added a commit that referenced this issue Oct 22, 2017

Bertware added a commit that referenced this issue Oct 24, 2017

Bertware added a commit that referenced this issue Oct 24, 2017

Bertware added a commit that referenced this issue Oct 24, 2017

Bertware added a commit that referenced this issue Oct 26, 2017

Support routes with a small walking part. Include remarks, urls for r…
…emarks, urls for alerts, fix error message causing incorrect json #319 #325

@Bertware Bertware closed this in 32505c0 Nov 4, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment