Problem parsing a sample file

[raphaelm]

Hi,

in my project, I test the mt940 feature (that is based on your package) against the sample files of multiple banks.

Since version 4.0, the sample file from this document (page 34) can no longer be parsed.

I tried to find the problem myself but wasn't very successful until now. Do you see it?

The file:

:20:STARTUMS
:25:1222333444
:28:1/1
:NS:22Test GmbH
23Testkonto
240,800
25010102311202
3037010000
3190000022
:60F:C020315DEM0,00
:61:0203170320CM5000,00S05168790452
:NS:01Verwendungszweck 1
02Verwendungszweck 2
15Empfänger
17Buchungstext
1812345
191000
204711
:61:020322CM20000,00NCHG
:61:020322CM20000,00S051
:61:020322CM20000,00S051
:61:020322CM20000,00S051
:61:020322CM20000,00S051
:62M:C020315105000,00
:20:STARTUMS
:25:1222333444
:28:1/1
:NS:223037010000
:60M:C020315DEM105000,00
:61:020322CM20000,00S051
:61:020322CM20000,00S051
:62F:C020315145000,00
:20:STARTUMS
:25:3346780111
:28:2/1
:NS:22Meyer + Schneider
23Testkonto
3037010000
3187132101
:60F:C020324DEM145000,00
:61:020324DM50000,00S051
:NS:01bekannt
1812345
:62F:C02032495000,00

I get the following error:

RuntimeError: ('Unable to parse "<mt940.tags.Statement object at 0x7f96544eae10>" from "0203170320CM5000,00S05168790452\n:NS:01Verwendungszweck 1\n02Verwendungszweck 2\n15Empfänger\n17Buchungstext\n1812345\n191000\n204711"', <mt940.tags.Statement object at 0x7f96544eae10>, '0203170320CM5000,00S05168790452\n:NS:01Verwendungszweck 1\n02Verwendungszweck 2\n15Empfänger\n17Buchungstext\n1812345\n191000\n204711')

With the following output:

DEBUG 2016-05-04 16:32:57,596 mt940.tags.TransactionReferenceNumber tags matched (8) "STARTUMS" against "(?P<transaction_reference>.{0,16})", got: {'transaction_reference': 'STARTUMS'}
DEBUG 2016-05-04 16:32:57,596 mt940.tags.AccountIdentification tags matched (10) "1222333444" against "(?P<account_identification>.{0,35})", got: {'account_identification': '1222333444'}
DEBUG 2016-05-04 16:32:57,596 mt940.tags.StatementNumber tags matched (76) "1/1
:NS:22Test GmbH
23Testkonto
240,800
25010102311202
3037010000
3190000022" against "
    (?P<statement_number>\d{1,5})  # 5n
    (?:/(?P<sequence_number>\d{1,5}))?  # [/5n]
    ", got: {'sequence_number': '1', 'statement_number': '1'}
DEBUG 2016-05-04 16:32:57,596 mt940.tags.FinalOpeningBalance tags matched (14) "C020315DEM0,00" against "^
    (?P<status>[DC])  # 1!a Debit/Credit
    (?P<year>\d{2})  # 6!n Value Date (YYMMDD)
    (?P<month>\d{2})
    (?P<day>\d{2})
    (?P<currency>.{3})  # 3!a Currency
    (?P<amount>[0-9,]{0,16})  # 15d Amount (includes decimal sign, so 16)
    ", got: {'year': '02', 'month': '03', 'amount': '0,00', 'currency': 'DEM', 'day': '15', 'status': 'C'}
INFO 2016-05-04 16:32:57,596 mt940.tags.Statement tags matching (127) "0203170320CM5000,00S05168790452
:NS:01Verwendungszweck 1
02Verwendungszweck 2
15Empfänger
17Buchungstext
1812345
191000
204711" against "^
    (?P<year>\d{2})  # 6!n Value Date (YYMMDD)
    (?P<month>\d{2})
    (?P<day>\d{2})
    (?P<entry_month>\d{2})?  # [4!n] Entry Date (MMDD)
    (?P<entry_day>\d{2})?
    (?P<status>[A-Z]?[DC])  # 2a Debit/Credit Mark
    (?P<funds_code>[A-Z])? # [1!a] Funds Code (3rd character of the currency
                            # code, if needed)
    (?P<amount>[\d,]{1,15})  # 15d Amount
    (?P<id>[A-Z][A-Z0-9]{3})?  # 1!a3!c Transaction Type Identification Code
    (?P<customer_reference>.{0,16})  # 16x Customer Reference
    (//(?P<bank_reference>.{0,16}))?  # [//16x] Bank Reference
    (\n?(?P<extra_details>.{0,34}))?  # [34x] Supplementary Details
                                             # (this will be on a new/separate
                                             # line)
    $"

[wolph]

Hi @raphaelm

Sorry for breaking the code in your case. Apparently there's another implementation of the MT940 standard I wasn't aware of. Can you test if the hotfix/#23 branch solves your problem?
https://github.com/WoLpH/mt940/tree/hotfix/%2323

[raphaelm]

Hi,

no worries, as I said thats only an example I use for testing, no real users harmed. Thanks for maintaining this high-quality library!

Yes, your fix removes the exception. Thank you! Just so you now, the output differs slightly from 3.8:

{'customer_reference': '68790452', 'transaction_reference': 'STARTUMS', 'id': 'S051', 'funds_code': 'M', 'currency': 'DEM', 'entry_date': Date(2002, 3, 20), 'extra_details': ':NS:01Verwendungszweck 1', 'bank_reference': None, 'status': 'C', 'amount': <5000.00 DEM>, 'date': Date(2002, 3, 17)}
...
{'customer_reference': '', 'transaction_reference': 'STARTUMS', 'funds_code': 'M', 'currency': 'DEM', 'id': 'S051', 'extra_details': ':NS:01bekannt', 'bank_reference': None, 'status': 'D', 'amount': <-50000.00 DEM>, 'date': Date(2002, 3, 24)}

In 3.8 the extra_details fields have been empty, but that is totally okay to me.

On a side node, it is surprisingly hard to pip install from a git branch that includes a / and a # ;)

[wolph]

Hrm... I forgot that you actually need to escape the # in this case. I suppose the easiest way would be a manual checkout followed by a pip install -e directory
Next time I'll omit the # :)

I'll take a look at the modified output soon, the mt940 parsing is a bit fragile due to all the (partially) incorrect implementations around

[wolph]

Took a bit of time to investigate further, but I found the problem. This example uses extra (non-swift, indicated by :NS: data) which wasn't supported yet.

Since the entire code assumes that all tags are always numeric, this was a bit of an issue but I believe it will work correctly now :)