Bunq API gives HTTP 500 error (Insufficient authentication)

[nickstrijbos]

The stack last update was on 2022-05-12 08:29:28 UTC+0200. It stopped working on the 22nd of september. So not sure why the API suddenly stopped working. It worked for more than a full year.

I just created a new Bunq API key, updated the stack again and it works. So not sure if we can add something where it mails us when de lambda function gives a 500 error on API authentication.

log output from the function when it didn't update.:

timestamp	message
1695665133169	INIT_START Runtime Version: python:3.8.v26 Runtime Version ARN: arn:aws:lambda:eu-north-1::runtime:81f077a44b32842209c33a8a4ecbe13f17dabc0964eaae632cb27846f04bf85e
1695665133684	START RequestId: 4df58b78-fdb9-4a1e-a558-bc76d39050bf Version: $LATEST
1695665133685	2023-09-25 18:05:33,685
1695665135113	2023-09-25 18:05:35,113
1695665135396	2023-09-25 18:05:35,396
1695665135396	2023-09-25 18:05:35,396
1695665135396	2023-09-25 18:05:35,396
1695665135553	2023-09-25 18:05:35,553
1695665135553	Traceback (most recent call last):
1695665135553	File "/var/task/lambda_function.py", line 51, in lambda_handler
1695665135553	sync.populate()
1695665135553	File "/var/task/lib/sync.py", line 59, in populate
1695665135553	self.bunq_accounts = list(bunq_api.get_accounts())
1695665135553	File "/var/task/lib/bunq_api.py", line 109, in get_accounts
1695665135553	for u in [first_value(u) for u in bunq.get("v1/user")]:
1695665135553	File "/var/task/lib/bunq.py", line 206, in get
1695665135553	return call('GET', method)
1695665135553	File "/var/task/lib/bunq.py", line 197, in call
1695665135553	raise Exception(result["Error"][0]["error_description"])
1695665135553	Exception: Insufficient authentication.

[wesselt]

Goed idee, een mail als het token verlopen is. Ik ga kijken hoe ik dat kan toevoegen.

[nickstrijbos]

Het stomme is dat Bunq niet laat zien wanneer de token verlopen is. Weet jij toevallig vanuit de API docs of er een verloopdatum op zit? Als ik moet mee kijken in AWS om iets te kunnen mailen moet je het maar even laten weten!

[wesselt]

Volgens mij is de duur van een bunq token niet gedocumenteerd en kunnen zij dit veranderen.

Branch / pull request aangemaakt voor email: #50

Hoe werkt deze voor jou?

[wesselt]

Sending an email when an error occurs is now supported

[nickstrijbos]

Hey Wesselt! Thanks voor het toevoegen. Moet ik nog iets verder testen in AWS? had een beetje hectische week op werk.

[wesselt]

Hoi Nick, de AWS versie staat op: https://serverlessrepo.aws.amazon.com/applications/eu-west-1/991880991323/bunq-ynab-aws-lambda

Javy de Koning werkte deze bij, maar ik denk dat hij geen bunq gebruiker meer is. @javydekoning hoe kan ik de repo bijwerken naar een nieuwe versie?

[nickstrijbos]

@wesselt Waarschijnlijk de template yaml aanpassen met extra parameters. Ik zat zelf nog te denken dat een poort keuze ook wel handig kan zijn omdat sommige lokaal op 25 al draaien. En ik gebruik bijvoorbeeld google workspace waarbij ik een relay server heb en username password niet perse nodig zijn. Hier voorbeeld toevoeging van parameters. Deze moeten dan nog wel gedefinieerd worden in de juiste config files in die template.yml

Parameters:
  BunqApiToken:
    Type: String
    Description: Your BUNQ API Token
  YnabAccessToken:
    Type: String
    Description: Your YNAB access token
  YnabBudget:
    Type: String
    Description: Name of your YNAB budget (use * for all budgets)
  SMTPServer:
    Type: String
    Description: Your SMTP server, for example smtp.google.com
  SMTPPort:
    Type: string
    Description: Port for the SMTP Server, examples are 25, 465 or 587. Relay servers don't require user or password input most of the time. 
  SMTPUser:
    Type: String
    Description: Your SMTP username
  SMTPPassword:
    Type: String
    Description: Your SMTP password
  SMTPFrom:
    Type: String
    Description: From where you get the mail from bunq2ynab@some.where
  SMTPTo:
    Type: String
    Description: Your SMTP to adress, example@gmail.com

[javydekoning]

Yes, like @nickstrijbos says those parameters would need to be added.

You'd also need to update the config json here:

https://github.com/wesselt/bunq2ynab/blob/master/template.yaml#L98-L112

Indeed, I'm no longer at Bunq (most dutch banks sync natively with YNAB through Truelayer).

I'd be happy to publish a new version, but I won't be able to test it...

If someone wants to take over, you can publish to the SAR here: https://us-east-1.console.aws.amazon.com/serverlessrepo/home?locale=en&region=us-east-1#/published-applications

[nickstrijbos]

@javydekoning I will fork the repo and make changes to the template, test them and merge them to your repo. And can publish the SAR if that's okay with you too @wesselt

[wesselt]

Sounds good! Does that mean the new version gets a new URL?

@javydekoning What options are there to give other people rights to update the SAR entry at https://serverlessrepo.aws.amazon.com/applications/eu-west-1/991880991323/bunq-ynab-aws-lambda?

[javydekoning]

Yes, the new version will get a new URL if someone takes over publishing. The 991xxx part of the url is the AWS Account ID.

[wesselt]

@nickstrijbos Commit cf91414 supports specifying STMP port. If the port is 25, it starts with plain SMTP, and only switches to TLS when a password is provided.

[MagicLegend]

@nickstrijbos Are you able to keep the AWS application updated by any chance? :)

[nickstrijbos]

@nickstrijbos Are you able to keep the AWS application updated by any chance? :)

Hey @MagicLegend, had some personal issues the last couple of months, and no energy in side projects. But I want to pick this back up. Don't pin me down on it, but I'm going to do my best to commit this as soon as possible.

[nickstrijbos]

@MagicLegend, I've reached out to @javydekoning and resolved the deployment part. I'm currently in the process of testing the solution with the latest commit to ensure everything functions as expected. Once confirmed, I'll proceed with deploying it to AWS.

@wesselt I can update the Lambda function every time you commit something new but then we need to use versioning

[wesselt]

@nickstrijbos Checked out versioning, that looks easy enough. I've added a release: https://github.com/wesselt/bunq2ynab/releases/tag/release How does that work for your purpose?

[MagicLegend]

Thank you for picking it up @nickstrijbos !

Perhaps a bit of documentation on how to update an already running lambda would be useful as well (I did not know that I had to redeploy the lambda from the Application repository until I did some reading...).

And perhaps development should be done outside of master if we are going to auto-deploy to the AWS repo? Otherwise it would be possible to deploy something that still needed some fixes :-)

[nickstrijbos]

@wesselt The next preferable tag should be 1.4.0 in semantic versioning. So the template can take that over when I build, package and deploy the new version.

@MagicLegend, appreciate your suggestion! I'm currently in the process of refining the README to make it more applicable and useful in the context of AWS. On the topic of auto-deployment to AWS, it's not feasible at this moment due to the necessity of including personal access keys and secrets in the packaging. However, the repository's setup allows for the testing of new updates or functions without the need to directly commit to the AWS Serverless Application Repository.

[wesselt]

@nickstrijbos Just to double check, you're saying that 0.4.1 is preferable to 24022001? How about 24.2.20?

[nickstrijbos]

Alright update from my side. Got some issues with the new SMTP Variables. They can't be empty. So for the time being will update the AWS template to only work without email. So I can patch and update all the functions @wesselt added in the last couple of days. So I will push a 1.4.0 later this weekend a 1.5.0 for the mail part. Will also open up a new issue.

[nickstrijbos]

Hey all,

Updated and published the new template. It can be found here: bunq-ynab-aws-lambda — version 1.4.4

The SMTP functions are still not visible. Same steps apply (test the deployment to start automatic syncing). We still need to fix some of the errors that are showing up. But in my case it syncs again. Open issues if you find something so we can patch it. Thanks for the patience.

[MagicLegend]

Hi @nickstrijbos

I've updated to 1.4.4, but syncing gives the following error:

2024-02-26 07:40:38,803 | INFO | lambda_function.py:13 | add_callbacks | SSM callback = "https://asdfasdfasdf.execute-api.location.amazonaws.com/bunq2ynab-lambda"
2024-02-26 07:40:38,803 | INFO | lambda_function.py:16 | add_callbacks | Adding callbacks...
2024-02-26 07:40:38,803 | ERROR | lambda_function.py:62 | lambda_handler | Exception occurred
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 56, in lambda_handler
add_callbacks(sync)
File "/var/task/lambda_function.py", line 18, in add_callbacks
callback_marker = config.get("callback-marker") or "bunq2ynab_autosync"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/task/lib/config.py", line 96, in get
raise Exception(f"Use underscore instead of dash when "
Exception: Use underscore instead of dash when getting config value callback-marker
END RequestId: 44826099-asdf-asdf-asdf-8015f0ed900b
REPORT RequestId: 44826099-asdf-asdf-asdf-8015f0ed900b	Duration: 24854.26 ms	Billed Duration: 24855 ms	Memory Size: 512 MB	Max Memory Used: 102 MB	Init Duration: 639.40 ms

When changing the given line to use callback_marker instead it gives the following error:

2024-02-26 07:43:18,864 | INFO | lambda_function.py:13 | add_callbacks | SSM callback = "https://asdf.execute-api.location.amazonaws.com/bunq2ynab-lambda"
2024-02-26 07:43:18,864 | INFO | lambda_function.py:16 | add_callbacks | Adding callbacks...
2024-02-26 07:43:18,864 | ERROR | lambda_function.py:62 | lambda_handler | Exception occurred
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 56, in lambda_handler
add_callbacks(sync)
File "/var/task/lambda_function.py", line 19, in add_callbacks
bunq_api.add_callback(uid, callback_marker, url)
File "/var/task/lib/bunq_api.py", line 14, in add_callback
raise Exception("URL should end with end-of-url marker")
Exception: URL should end with end-of-url marker
END RequestId: f1b23a00-asdf-asdf-asdf-121cb4d6374c
REPORT RequestId: f1b23a00-asdf-asdf-asdf-121cb4d6374c	Duration: 17619.31 ms	Billed Duration: 17620 ms	Memory Size: 512 MB	Max Memory Used: 103 MB	Init Duration: 623.16 ms

Which I suspect might be because I already had config values from the old deployment, and the code is not handeling that correctly?

I'm afraid it'll need another update :) Thanks!

Edit:

Ah, I figured it out. This line causes the error: https://github.com/wesselt/bunq2ynab/blob/master/lambda_function.py#L18

bunq2ynab/lambda_function.py

Line 18 in 5a0b1be

callback_marker = config.get("callback_marker") or "bunq2ynab-autosync"

The default ending is not bunq2ynab-autosync but bunq2ynab-lambda. The fix is either to change the code, or to add a value to the configuration object so the callback_marker config value actually exists (because it's not in the template). E.g.:

{
  "api_token": "asdfasdf",
  "personal_access_token": "asdfasdf",
  "accounts": [
    {
      "ynab_budget_name": "asdfasdf"
    }
  ],
  "callback_marker": "bunq2ynab-lambda"
}

I guess the template should either include the callback_marker with this default value, or the code should be updated to use the default value created by AWS. Note that the state gets initialized with this line ending in the template as well:

bunq2ynab/template.yaml

Line 126 in 5a0b1be

"aws_callback": "https://${HttpApi}.execute-api.${AWS::Region}.amazonaws.com/bunq2ynab-lambda"

Which leads me to suspect that even a fresh setup would be broken due to this.

[wesselt]

Tried to check it out by deploying on AWS. The monitoring tab shows:

There was an error while making a request to StartQuery
Log group '/aws/lambda/serverlessrepo-bunq-ynab-aws-lam-Bunq2YnabFunction-gjX4OByrbnfU' does not exist for account ID '122776901776' (Service: AWSLogs; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: d28abbb0-5859-4172-bd29-9bda90b088a2; Proxy: null)

I wonder what this means and how I can fix it.

[javydekoning]

Looks like you've manually deleted the CloudWatch log group in the past. Either recreate it or delete the stack and redeploy it.

[wesselt]

That makes sense, I switched to my Rasperry Pi and deleted everything on AWS, and I probably deleted the log group then.

This was a fresh install though. And I just I deleted and redeployed the stack and get the same error (with different ids.)