JSONDecodeError after command=displayChallengeQuestion

[adeverteuil]

Hello,

I haven't imported my statements since July, so I'm not sure when the problem started.

I checked issue #2 and the problem happens at a different stage.

When sending the POST with command=PersonalCIF and ACN=XXXX parameters, the response contains:

<SCRIPT>                                                                            
location.replace("/web/Tangerine.html?command=displayLogin&fill=1");
                                                                                                          
</SCRIPT>

I checked the web browser console and the location.replace() parameter in the response should be "/web/Tangerine.html?command=displayChallengeQuestion".

Following this, when sending the GET with the command=displayChallengeQuestion parameter, the response is the same location.replace() script instead of the expected JSON document.

Here is the full DEBUG output:

INFO:MainLoop:Starting main loop
INFO:MainLoop:Logging in
INFO:tangerine.login:Initiating Tangerine logging flow
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): secure.tangerine.ca
DEBUG:urllib3.connectionpool:https://secure.tangerine.ca:443 "GET /web/InitialTangerine.html?command=displayLogout&device=web&locale=en_CA HTTP/1.1" 200 182
DEBUG:tangerine.login:
{

"MessageHeader" : {

"View" :
"Logout"
,
"Webtrends" : {

"WT.cg_n" :
"Auth"
,
"WT.cg_s" :
"Logout"
,
"WT.si_n" :
"Auth_Logout"
,
"WT.si_p" :
""
,
"locale" :
"en_CA"
,
"device" :
"metro"
,
"flavour" :
"metro"
,
"mode" :
"TPW"

}
,
"Reset" :
"true"

}
,
"MessageBody" : {

}

}

DEBUG:urllib3.connectionpool:https://secure.tangerine.ca:443 "GET /web/InitialTangerine.html?command=displayLoginRegular&device=web&locale=en_CA HTTP/1.1" 200 255
DEBUG:tangerine.login:
{

"MessageHeader" : {

"View" :
"ValidatePersonalCIF"
,
"Webtrends" : {

"WT.cg_n" :
"Auth"
,
"WT.cg_s" :
"Login"
,
"WT.si_n" :
"Auth_Login"
,
"WT.si_p" :
"ValidateWebCIF"
,
"locale" :
"en_CA"
,
"device" :
"metro"
,
"flavour" :
"metro"
,
"mode" :
"TPW"

}

}
,
"MessageBody" : {

"Action" :
"/web/Tangerine.html"
,
"Command" :
"PersonalCIF"
,
"Locale" :
"en_CA"
,
"Device" :
"web"
,
"GoButtonName" :
"Go"
,
"Fido2InitAuth" :
""

}

}

DEBUG:urllib3.connectionpool:https://secure.tangerine.ca:443 "POST /web/Tangerine.html HTTP/1.1" 200 108
DEBUG:tangerine.login:<SCRIPT>
location.replace("/web/Tangerine.html?command=displayLogin&fill=1");

</SCRIPT>
DEBUG:urllib3.connectionpool:https://secure.tangerine.ca:443 "GET /web/Tangerine.html?command=displayChallengeQuestion HTTP/1.1" 200 109
DEBUG:tangerine.login:<SCRIPT>
location.replace("/web/Tangerine.html?command=displayLogin&fill=1");

</SCRIPT>


Traceback (most recent call last):
  File "./fetch-tangerine-statements.py", line 196, in <module>
    MainLoop(options).main()
  File "./fetch-tangerine-statements.py", line 120, in main
    with self.client.login():
  File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/alex/.pyenv/versions/beancount/lib/python3.6/site-packages/tangerine/client.py", line 46, in login
    self.login_flow.start()
  File "/home/alex/.pyenv/versions/beancount/lib/python3.6/site-packages/tangerine/login.py", line 79, in start
    question = self._get_security_challenge()
  File "/home/alex/.pyenv/versions/beancount/lib/python3.6/site-packages/tangerine/login.py", line 63, in _get_security_challenge
    return r.json()['MessageBody']['Question']
  File "/home/alex/.pyenv/versions/beancount/lib/python3.6/site-packages/requests/models.py", line 892, in json
    return complexjson.loads(self.text, **kwargs)
  File "/home/alex/.pyenv/versions/beancount/lib/python3.6/site-packages/simplejson/__init__.py", line 518, in loads
    return _default_decoder.decode(s)
  File "/home/alex/.pyenv/versions/beancount/lib/python3.6/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/home/alex/.pyenv/versions/beancount/lib/python3.6/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[kevinjqiu]

Hi @adeverteuil

Thanks for the report. Someone has reported this before, as it appears that the new version of Tangerine implemented a fingerprinting mechanism on the client using Javascript. I haven't found the time to really tackle this issue yet. I might come back and look at it during the Christmas break, but any help is appreciated.

[adeverteuil]

Thanks for the quick response and for the status info!

[kevinjqiu]

Looked at it during the holidays... I'm not certain it's due to fingerprinting, since I get the same error if I supply pm_fp parameter (the fingerprint) that I got from the chrome devtool. Looks like there's now a separate step to obtain a user token when landing on the login page, but I haven't figured out how that user token is tied to the parameters in the cookie that's sent to the server.

[Connoropolous]

👋 hi folks.
I have been poking around this repo, haven't tried it yet, but I'm interested in similar goals as I imagine you have with this.

I discovered a great tool/system for debugging this kind of thing, in case you weren't aware of it. It's the POSTMAN app, and the fact that you can use the 'Request Interceptor' strategy which records the request history, which you can then review.
https://chrome.google.com/webstore/detail/postman-interceptor/aicmkgpgakddgnaphhhpliifpcfhicfo
https://learning.getpostman.com/docs/postman/sending-api-requests/capturing-http-requests/

I am a coder, but I'm not very familiar with python, so using this would be tricky for me

[kevinjqiu]

Thanks for your input @Connoropolous. I was using mitmproxy to inspect and record requests/responses.

[Connoropolous]

Nice. P.s. I'm Toronto based too, and also Florinapp looks interesting :)

I'm guessing these are "spare time" projects

[kevinjqiu]

Thanks :) It was an excuse for me to learn some react+redux :) With regard to this project, I kinda hit a wall now that they implemented browser fingerprints. I tried to use puppeteer (with an embedded chrome) to login and that didn't work either. If anyone has any ideas I'm all ears.

[Connoropolous]

What are these "browser fingerprints"?

[Connoropolous]

https://pixelprivacy.com/resources/browser-fingerprinting/ ?

[Connoropolous]

mitmproxy seems pretty amazing, was just checking it out.

@kevinjqiu for what its worth (which might not be much) I discovered the existence of https://mgateway.tangerine.ca today, doing some digging, by proxying network traffix of the mobile app. However, their security on the app seems very tight too, it quickly blocks the app saying it can't form a secure connection. Despite me using the mitmproxy root certificate setup that they document, for handling https traffic.

[Connoropolous]

Oh, also, looks like their app is just a web-wrapper HTML thing, so it has all the annoying things that the web version has.

[kevinjqiu]

Thanks for taking a look, @Connoropolous . FWIW, I didn't have a problem with using mitmproxy's root cert.

[francisjeanneau]

I am interested in this project but found out that it isn't working anymore sadly.

Has anyone investigated the use of requests_html which can interpret JavaScript by using Chromium under the hood?