FinTS after PSD2

[UweZiegenhagen]

As I learnt today, I won't be able to get my account statements for my Giro account from September on via HBCI since ING Diba will disconnect this service.

Will PSD2 have an impact on python-fints? Are you planning to implement PSD2?

[henryk]

As far as I can see, the primary change in PSD2 with regards to FinTS is Strong Customer Authentication (SCA) for more operations (e.g. login). And also that SCA now sometimes isn't necessary anymore (bank-specific, e.g. transfer to subaccounts or transfers of 30€ or less).
Basically you now can get a surprise "SCA required" response at any time, with no way to detect this before issuing the command.

When designing the new API I thought of that already: the SEPA transfer methods with either return a TransactionResponse (if the call went through without SCA requirement) or a NeedRetryResponse (if a TAN is required). This allows the library user to elegantly handle surprise SCA. What we still need is a way to do the same with login, where the dialog opening sometimes requires SCA.
For the sake of consistency I'd suggest caching the surprise SCA requirement in the dialogue and then returning a NeedRetryResponse on the first real operation call.

So, yeah, I think python-fints can still work with PSD2 without too many problems -- in the interactive use case. The non-interactive use case will probably involve sending an email to the user every 90 days to request a TAN from them. (Ahem, someone maybe can write a plugin module to automatically generate a ChipTAN from a connected smartcard reader and then just need to leave your Girocard with the server. Or use one of these new-fangled VoIP-providers with an API to receive SMS and automatically extract the smsTAN. Both are stellar options from a security point of view! Well done, PSD2!)

That being said, ING DiBa threatening to shut down their FinTS endpoints is an overreaction/decision by the bank, there's no hard technical reason for that. And nothing we can do about it. For my own company use I'm going to investigate how hard it would be to set up EBICS, because that would be the alternative.
(Or, you know, going back to the dark ages of screen scraping. Just with a new requirement to enter a TAN on each login.)

[raphaelm]

Yes, as @henryk said, PSD2 does not necessarily kill FinTS, but some banks probably will. Since many banks out their use FinTS as the backing protocol of e.g. their smartphone apps and desktop applications, I assume that most major banks will still support it for years to come, though.

My plan with regards to python-fints is not to put to much work into it over the summer, then evaluate the situation in September when we really know whats happening and work around it.

Keep in mind that PSD2 mandates banks to have some kind of API, but while some first standard-ish things emerge, it's still pretty hard to find reliable information on which bank will give what kind of access to what kind of API user.

[jahir]

Postbank and Consorsbank enabled SCA over the Weekend, i.e. you now need a TAN for login. While Consorsbank FinTS is still broken, Postbank seems to allow subsequent logins after one successful SCA. This works with my desktop software (Banking4), but how would I do this with python-fints? As far as I understand the docs, it should as me for a TAN if an operation requires one, but a get_sepa_accounts() returns this:

Dialog response: 9050 - Teilweise fehlerhaft.
Dialog response: 9999 - Dialog bereits geschlossen.
Dialog response: 9800 - Dialog ungueltig/unbekannt.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jh/.local/lib/python3.7/site-packages/fints/client.py", line 433, in get_sepa_accounts
    for seg in response.find_segments(HISPA1, throw=True):
  File "/home/jh/.local/lib/python3.7/site-packages/fints/types.py", line 293, in find_segments
    'The bank\'s response did not contain a response to your request, please inspect debug log.'
fints.exceptions.FinTSNoResponseError: The bank's response did not contain a response to your request, please inspect debug log.

There are two TAN mechanism available, while 920 seems to be the default.

OrderedDict([('920',
              fints.formals.TwoStepParameters5(security_function='920', tan_process='2', tech_id='BestSign', zka_id='BestSign', zka_version=None, name='BestSign', max_length_input=6, allowed_format=fints.formals.AllowedFormat.ALPHANUMERIC, text_return_value='BestSign', max_length_return_value=999, number_of_supported_lists=0, multiple_tans_allowed=False, tan_time_dialog_association=fints.formals.TANTimeDialogAssociation.NOT_ALLOWED, tan_list_number_required=fints.formals.TANListNumberRequired.NO, cancel_allowed=False, sms_charge_account_required=fints.formals.SMSChargeAccountRequired.MUST_NOT, principal_account_required=fints.formals.PrincipalAccountRequired.MUST, challenge_class_required=False, challenge_structured=True, initialization_mode=fints.formals.InitializationMode.CLEARTEXT_PIN_NO_TAN, description_required=fints.formals.DescriptionRequired.MUST, supported_media_number=9)),
             ('930',
              fints.formals.TwoStepParameters5(security_function='930', tan_process='2', tech_id='mobileTAN', zka_id='mobileTAN', zka_version=None, name='mobileTAN', max_length_input=6, allowed_format=fints.formals.AllowedFormat.ALPHANUMERIC, text_return_value='mobileTAN', max_length_return_value=999, number_of_supported_lists=0, multiple_tans_allowed=False, tan_time_dialog_association=fints.formals.TANTimeDialogAssociation.NOT_ALLOWED, tan_list_number_required=fints.formals.TANListNumberRequired.NO, cancel_allowed=False, sms_charge_account_required=fints.formals.SMSChargeAccountRequired.MUST_NOT, principal_account_required=fints.formals.PrincipalAccountRequired.MUST, challenge_class_required=False, challenge_structured=True, initialization_mode=fints.formals.InitializationMode.CLEARTEXT_PIN_NO_TAN, description_required=fints.formals.DescriptionRequired.MUST, supported_media_number=9))])

[henryk]

Outstanding. I have a Postbank account I can test with. You're right, the API is designed so that methods should return a NeedTANResponse if necessary, but this isn't fully implemented yet, and not at all for "login". I'm now working on it in https://github.com/henryk/python-fints/tree/psd2

[raphaelm]

Nice! I don't have full access to a Postbank account, so I'll need to wait for the Fiducia-based banks to move before I can really test around myself.

[jahir]

Update: Consorsbank managed to get their FinTS system up again. The OB software on my desktop (Banking4) is again able to retrieve transactions (sadly, for some stupid reason, they require a TAN for every retrieval).
The behaviour in fints is a little different to Postbank: get_sepa_accounts() works as before, but get_balance(account) fails:

Dialog response: 9050 - Nachricht teilweise fehlerhaft.
Dialog response: 9010 - Verarbeitung nicht möglich.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jh/.local/lib/python3.7/site-packages/fints/client.py", line 512, in get_transactions
    'HIKAZ'
  File "/home/jh/.local/lib/python3.7/site-packages/fints/client.py", line 453, in _fetch_with_touchdowns
    rm = dialog.send(seg)
  File "/home/jh/.local/lib/python3.7/site-packages/fints/dialog.py", line 133, in send
    self.client.process_response_message(self, response, internal_send=internal_send)
  File "/home/jh/.local/lib/python3.7/site-packages/fints/client.py", line 230, in process_response_message
    self._process_response(dialog, segment, response)
  File "/home/jh/.local/lib/python3.7/site-packages/fints/client.py", line 1228, in _process_response
    raise FinTSClientError("Error during dialog initialization, could not fetch BPD. Please check that you "
fints.exceptions.FinTSClientError: Error during dialog initialization, could not fetch BPD. Please check that you passed the correct bank identifier to the HBCI URL of the correct bank.

It seems that even fetching the BPD requires SCA (which would explain why Banking4 requested two TANs when I synchronized the account data).

[henryk]

@jahir Oh great, I feared something like that. I have been working through Sicherheitsverfahren PIN/TAN section B.4.3 and they say something like this about bootstrapping:

• Open an anonymous dialogue, to fetch BPD. Find out if HITANS6 is present (indicates SCA capability).
• Open a dialogue with security function 999 (one-step TAN) and PIN, fetch allowed security functions from response code 3920.
• May be necessary: Fetch TAN media descriptions: Open a dialogue with a two-step security function and pin but HKTAN6 with a dummy TAN media value and the special segment value "HKTAB", and add a HKTAB segment. Get HITAB response with SCA exception.
• Finally: Open a dialogue with the selected security function and PIN, and HKTAN6, setting the TAN media name to a valid name. Get HITAN6 response, ask user for TAN, retry dialogue initialisation with HKTAN6 and the provided TAN. Get UPD and open dialogue.

This four step process is bad enough (and trying to put that into control flow that gracefully skips steps if cached information is present is what I spent last night thinking about). But now the fun part:
Postbank provides the BPD during our current security-function-999-dialogue, so doesn't need an anonymous dialogue. From what @jahir says it seems that Consors requires the anonymous dialogue. On the other hand I'm not sure all banks allow an anonymous dialogue at all.
More fun: Without any indication of having done so Postbank implicitly closes the dialogue after their 3920 response, so we can't query HKTAB in the same dialogue.

[henryk]

One more thought: One very useful use case for this library is automated fetches of statements in the background. With SCA and one of the "push" type TAN mechanisms such a scheduled process would trigger a push to the user, which is both annoying and indistinguishable from an attack.
Many banks are implementing the allowed exception of only requiring one SCA every 90 days.

Ideally we would be able to probe whether an SCA is required before triggering it. My idea is to switch us into SCA-unaware mode by not transmitting HKTAN6 in the dialogue opening (same as now) and then maybe getting a 9075 response. (A library user that encounters this in a scheduled job would then alert the user via out-of-band means (e.g. e-mail) to trigger an SCA once manually.)

Implementation idea: FinTS3PinTanClient gets a new mode parameter with these possible values:

• offline: Never send anything to the bank. Only used for inspecting cached parameters (with the from_data= mechanism.)
• sca_unaware: Current behaviour. Do not send HKTAN6 in dialogue establishment. Raise 9075 response as SCARequiredError. If no SCA is required, the client will work as normal.
• sca_aware: New behaviour. The user will need to use a new method perform_sca() with a standing dialogue.

API wise I think something like this should be the least deviation from current state, and easiest for library users to integrate:

Var A Do not risk pushing a TAN to the user, use this in background

try:
  with FinTS3PinTanClient(..., mode=FinTSClientMode.SCA_UNAWARE) as client:
    txns = client.get_transactions(...)
except FinTSSCARequiredError:
  # notify user

Var B With possible TAN input, use this in normal UI

with FinTS3PinTanClient(..., mode=FinTSClientMode.SCA_AWARE) as client:
  # May call query methods to query and select security mechanism and tan medium
  tan_media = client.get_tan_media()
  client.set_tan_medium(tan_media[0])
  
  r = client.perform_sca()
  if isinstance(r, NeedRetryResponse):
    # Need to request and input TAN. This may involve a Flicker code or similar
    tan = input("TAN")
    client.send_tan(r, tan)
  # The dialogue is now open and can be used as normal
  txns = client.get_transactions(...)

(Web apps can split this flow by issuing .pause_dialog() and .deconstruct() in the case of NeedRetryResponse)

Caveat: I'm not sure all banks will allow Var A going forward. Library users should make sure to make scheduled polling optional (and strive to keep dialogues open as long as possible and reuse them whenever possible).

The current code tries to be as unobstrusive as possible and will only send to the bank if really necessary. This would change with SCA_AWARE: For proper function we'll have to open multiple dialogues and gather several kinds of data, the library will become comparatively noisy (well, not as noisy as version 1, but still :). Library users are now even more strongly encouraged to use .deconstruct()/from_data= to cache this data between sessions.

[raphaelm]

Great thoughts! Some comments:

sca_unaware: Current behaviour. Do not send HKTAN6 in dialogue establishment. Raise 9075 response as SCARequiredError. If no SCA is required, the client will work as normal.
sca_aware: New behaviour. The user will need to use a new method perform_sca() with a standing dialogue.

If we introduce a "high-level mode of operation" for this use-case, do we really want to make it an "SCA" mode? In the end, SCA is just one very specific instance of the more general problem that the library sometimes is run with user attention and sometimes is run in the background and needs to handle things differently in these cases. Maybe a better description wouldn't be "SCA (un)aware" but "interactive"/"non-interactive"?

To be clear, this is mostly about naming for now, but we don't know that the future holds for us...

API wise I think something like this should be the least deviation from current state, and easiest for library users to integrate:

That's always great, but I don't think it's the most important thing to consider in this case. My impression is that 90% of the users of this library use it in very simple scripts that pull data or init a transfer and are easy to migrate to API changes.

Caveat: I'm not sure all banks will allow Var A going forward. Library users should make sure to make scheduled polling optional (and strive to keep dialogues open as long as possible and reuse them whenever possible).

Well, there's only one way to find out – but even if only some banks will allow it, it would still be useful.

[jahir]

Next to follow: Fiducia (service provider for Volksbank/Raiffeisenbank). At least they provide the real reason why the dialog fails. (Last example, makes probably no sense to post every single bank here)

>>> accounts = f.get_sepa_accounts()
Dialog response: 9050 - Die Nachricht enthält Fehler.
Dialog response: 9075 - Starke Kundenauthentifizierung notwendig.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jh/.local/lib/python3.7/site-packages/fints/client.py", line 433, in get_sepa_accounts
    for seg in response.find_segments(HISPA1, throw=True):
  File "/home/jh/.local/lib/python3.7/site-packages/fints/types.py", line 293, in find_segments
    'The bank\'s response did not contain a response to your request, please inspect debug log.'
fints.exceptions.FinTSNoResponseError: The bank's response did not contain a response to your request, please inspect debug log.

[j-ittner]

Outstanding. I have a Postbank account I can test with. You're right, the API is designed so that methods should return a NeedTANResponse if necessary, but this isn't fully implemented yet, and not at all for "login". I'm now working on it in https://github.com/henryk/python-fints/tree/psd2

Many thanks Henryk, this is great. I have a Postbank account as well (and no more fints access since the weekend) - happy to volunteer as tester!