Skip to content
This repository has been archived by the owner on Oct 29, 2023. It is now read-only.

Transactions timestamp is wrong #110

Open
gsalvatori opened this issue Jan 25, 2022 · 35 comments
Open

Transactions timestamp is wrong #110

gsalvatori opened this issue Jan 25, 2022 · 35 comments
Labels

Comments

@gsalvatori
Copy link

gsalvatori commented Jan 25, 2022

Hi all,

just to report that I'm noticing that the transactions timestamps retrieved via api, don't match the actual timestamps on the app. For example, if you take the first transaction from this list (via CLI)

on macbook ~ took 12s
➜ n26 transactions
Output is limited to 5 entries.
Date               Amount      From    To          Message         Recurring
-----------------  ----------  ------  ----------  --------------  -----------
**01/25/22 07:29:28  -8.81 EUR   You     NATURASI'   ATM Withdrawal**
01/24/22 14:28:44  -29.9 EUR   You     PAGOPA      ATM Withdrawal
01/24/22 14:28:44  -9.99 EUR   You     SPOTIFY     ATM Withdrawal
01/24/22 14:28:44  -73.1 EUR   You     PASCAL SRL  ATM Withdrawal
01/24/22 14:28:44  -11.95 EUR  You     EXCELSA     ATM Withdrawal

reports that the transaction was carried out on January 25 but in the app I get Saturday, January 22 (which is the correct date) :

image

Same example via API:

- id: ..
  userId: ..
  type: PT
  amount: -8.81
  currencyCode: EUR
  originalAmount: -8.81
  originalCurrency: EUR
  exchangeRate: 1
  merchantCity: ROMA
  visibleTS: 1643092168637
  mcc: 5499
  mccGroup: 7
  merchantName: NATURASI'
 ...

visibleTS is 1643092168637 which converted turns out to be Tuesday 25 January 2022 06:29:28.637

maybe something has changed on the API side ?

Thanks

@markusressel
Copy link
Collaborator

I think the API contains two date fields, one when the transaction was created and another when it was actualy fully processed. Currently we only show one of them and never switch between the two.
Maybe we should?

@gsalvatori
Copy link
Author

gsalvatori commented Jan 26, 2022

Hi @markusressel,

unfortunately I can only speak for my use case which is running a cronjob every day that gets all the transactions I've made during the day and bring them to another application. So, for me, the reference timestamp is what I can see on the app and I think it's the "created" one.

By the way, all the timestamps I get now on the dictionary returned by APIs are "wrong", in relation to what I see on the app and I'm quite sure that approximately one week ago it was working well

@tranb3r
Copy link

tranb3r commented Jan 28, 2022

The app is using a new endpoint for transactions : /api/feed/v2 instead of /api/smrt/transactions.
For each transaction item in the response, there is a single timestamp field, which is the one you see in the app, and which corresponds to the visibleTS that was used before.
I think it's a pretty straightforward change to implement, but I'm no python expert...

@markusressel
Copy link
Collaborator

It might be possible to add support for this endpoint, but its output is pretty limited by default, contains other stuff beside transactions, and I am not sure how filtering works (the old parameters don't seem to have any effect). I think for this to be useful we would need at least some kind of working date filtering. Is there anything out there that we could build off of?

@tranb3r
Copy link

tranb3r commented Feb 1, 2022

You can use /api/feed/accounts/{account_id}/transactions/search for search and filters.

Example of filter:

{
    "filterCriteria": {
        "filters": [{
                "criteria": {
                    "from": 1641942000000,
                    "to": 1643670000000
                },
                "type": "DATE_RANGE"
            }
        ]
    },
    "searchText": "amazon"
}

@markusressel
Copy link
Collaborator

Not sure where you get your infos from, but we need more of them if we want to get this to work 😄

Querying https://api.tech26.de/api/feed/v2 with the existing code works and returns a lot of stuff, but as commented earlier, its unclear to me what the data is constrained to. For an API we need control over that.

result = self._do_request(GET, BASE_URL_DE + f'/api/feed/v2')

Querying https://api.tech26.de/api/feed/accounts/uuid-i-got-from-accoun-info-endpoint/transactions/search gives me a 502 Bad Gateway error.

result = self.get_account_info()
account_id = result["id"]

filters = []
if from_time is not None and to_time is not None:
    filters.append(
        {
            "criteria": {
                "from": from_time,
                "to": to_time
            },
            "type": "DATE_RANGE"
        }
    )

result = self._do_request(GET, BASE_URL_DE + f'/api/feed/accounts/{account_id}/transactions/search', json={
    "filterCriteria": {
        "filters": filters
    },
    "searchText": text_filter
})

@tranb3r
Copy link

tranb3r commented Feb 1, 2022

Not sure where you get your infos from, but we need more of them if we want to get this to work 😄

Inspecting the http traffic from the android app.

Querying https://api.tech26.de/api/feed/accounts/uuid-i-got-from-accoun-info-endpoint/transactions/search gives me a 502 Bad Gateway error.

It's a POST, not a GET.

@markusressel
Copy link
Collaborator

Still fails, now with a 500 Internal Server Error.

But I had a look at the TIMELINE output from the feed/v2 endpoint, and the dates also don't really match what I am seeing in the app or on the website. F.ex. a transaction displayed in the app as: January 31, 2022, 12:48 PM is displayed as: 1 Feb in from the API (see json below), and when interpreting the timestamp field "manually" it results in: GMT: Tuesday, February 1, 2022 5:53:23.158 PM or Your time zone: Tuesday, February 1, 2022 6:53:23.158 PM both of which are completely wrong.

{
  'id': 'redacted',
  'title': 'Steam',
  'subtitle': {
    'template': '1 Feb  ·  Online'
  },
  'tintedSubtitle': {
    'elements': [
      {
        'value': '1 Feb  ·  Online'
      }
    ]
  },
  'amount': -6.78,
  'amountStyle': 'NONE',
  'currency': 'EUR',
  'timestamp': 1643738003158,
  'type': 'TRANSACTION',
  'category': 'micro-v2-media-electronics',
  'highlight': 'NONE',
  'icon': {
    'type': 'MERCHANT',
    'url': 'https://cdn.number26.de/feed/merchants/5102.png'
  },
  'balance': {
    'title': 'Balance on {{date}}',
    'amount': redacted,
    'currency': 'EUR'
  },
  'deeplink': 'number26://main/detail/redacted',
  'gestures': {
    'swipeLeft': {
      'description': 'Pay back from a Space',
      'deeplink': 'redacted',
      'tracking': {
        'action': 'feed.gesture_swipe.left',
        'category': 'engagement',
        'property': 'payback'
      }
    }
  }
}

Not sure whats going on here, but there is probably more to it than just a different endpoint.

@tranb3r
Copy link

tranb3r commented Feb 1, 2022

Timestamp issue: make sure you have all these headers properly set:

n26-timezone-identifier: Europe/Paris
x-n26-platform: android
x-n26-app-version: 3.73
n26-app-build-number: 202203000

@tranb3r
Copy link

tranb3r commented Feb 1, 2022

For the 500 error: my guess is that it's either the missing headers, or something is wrong with your json content.

@markusressel
Copy link
Collaborator

Adding both of the x-n26 headers results in the date beeing correct when querying the feed API

"x-n26-platform": "android",
"x-n26-app-version": "3.73",

The old API does still not show the correct dates (which is not a problem if we get the new one to work).

However, even adding all 4 headers doesn't fix the POST call, still 500.
This is the json body that I sent (as in code commented earlier):

{
  "filterCriteria": {
    "filters": [
      {
        "criteria": {
          "from": 1643628699863,
          "to": 1643747219000
        },
        "type": "DATE_RANGE"
      }
    ]
  },
  "searchText": "Music"
}

Maybe the account-uuid is not the correct/same as in the old API?

@tranb3r
Copy link

tranb3r commented Feb 1, 2022

The old API does still not show the correct dates

This is precisely why I suggested using the new endpoints.

However, even adding all 4 headers doesn't fix the POST call, still 500.

I don't get a 500 when sending the json content you've pasted.

Maybe the account-uuid is not the correct/same as in the old API?

Yes, it's the same. At least for me.

There must be something wrong in your headers (content-type ?)

@markusressel
Copy link
Collaborator

These are all the headers that I tried:

USER_AGENT = ("Mozilla/5.0 (X11; Linux x86_64) "
              "AppleWebKit/537.36 (KHTML, like Gecko) "
              "Chrome/59.0.3071.86 Safari/537.36")

_headers = {
            'Authorization': 'Bearer {}'.format(access_token),
            "n26-timezone-identifier": "Europe/Paris",
            "x-n26-platform": "android",
            "x-n26-app-version": "3.73",
            "n26-app-build-number": "202203000",
            "User-Agent": USER_AGENT,
            "Content-Type": "application/json"
        }

I also tried just the Content-Type, no luck.
If the app uses more, let me know.

@markusressel
Copy link
Collaborator

I also just tried the same in Hoppscotch, also doesn't work. I get the same 500 error code.

@tranb3r
Copy link

tranb3r commented Feb 1, 2022

What about device-token ?

@markusressel
Copy link
Collaborator

markusressel commented Feb 1, 2022

Still 500 😢
But this token is generated, its not coming from my phone. So if they have some kind of device-token handshake on this endpoint, I cannot replicate that.

@markusressel
Copy link
Collaborator

It might be easier for you to check on your end, which changes to the query (omission of headers etc) result in a 500 response. Unfortunately I currently don't have a setup (or time) to investigate the queries sent by the phone app, so I can only rely on the community.

@tranb3r
Copy link

tranb3r commented Feb 2, 2022

Sure, I'll try to do that.
Meanwhile, could you please copy/paste your raw request here ? (it might help reproduce the error)

@tranb3r
Copy link

tranb3r commented Feb 2, 2022

Using postman, I've tested removing pretty much everything except the authorization header, and it still works.

POST https://api.tech26.de/api/feed/accounts/xxxxxxxxxxxxxxxx/transactions/search HTTP/1.1
Authorization: Bearer xxxxxxxxxxxxxxxxxxxxxxx
Content-Type: application/json
User-Agent: PostmanRuntime/7.26.8
Accept: */*
Cache-Control: no-cache
Postman-Token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Host: api.tech26.de
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 231

{
  "filterCriteria": {
    "filters": [
      {
        "criteria": {
          "from": 1643628699863,
          "to": 1643747219000
        },
        "type": "DATE_RANGE"
      }
    ]
  },
  "searchText": "Music"
}

@markusressel
Copy link
Collaborator

I cannot get any closer without installing postman:

POST https://api.tech26.de/api/feed/accounts/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/transactions/search
Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Content-Type: application/json
User-Agent: PostmanRuntime/7.26.8
Accept: */*
Cache-Control: no-cache
Host: api.tech26.de
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 218

{
  "filterCriteria": {
    "filters": [
      {
        "criteria": {
          "from": 1643628699863,
          "to": 1643747219000
        },
        "type": "DATE_RANGE"
      }
    ]
  },
  "searchText": "Music"
}

Fails with 500.

Can you test if (somehow) using HTTP/1.0 breaks it?
According to this we might be using 1.0 although the posts is pretty old...

I also tried the same through the hotspot of my phone to make sure my local network isn't acting up, still 500.

Can anybody else verify if this only happens to me?

@tranb3r
Copy link

tranb3r commented Feb 4, 2022

I think it would be better to implement a basic client of this new endpoint (in a new branch) and let people (including you and me) test it using the lib instead of doing manual requests. It's the only way to make sure we're all using the same headers and protocol.
Do you think it's a good idea ?

@femueller
Copy link
Owner

@tranb3r @markusressel Does it maybe make sense to set up a workspace on Hoppscotch or Postman instead of having a branch for API testing?

I think in the long run API endpoint health questions will come up again and again. Maybe this could provide us more transparency without having to create new debugging branches?

@tranb3r
Copy link

tranb3r commented Feb 14, 2022

I don't know anything about workspaces on Hoppscotch or Postman, but if you think it could be useful, let's try it !
One question though: is it safe to use this kind of service, regarding the need to enter our password or access token ?

@femueller
Copy link
Owner

femueller commented Feb 22, 2022

is it safe to use this kind of service, regarding the need to enter our password or access token

That's the reason that's been holding me back so far from using it.

I don't think think that it's unsafe from the authentication perspective, since our phones are the second factor. But I am not sure at the moment how each of these handle the API responses and if they might store them (sensitive data like transactions etc.).

@rakicluca
Copy link

Hi @femueller , i'm using this package for a personal project. When I call the api_client.get_transactions() I can't see the Date columns. Is this feature currently disabled?

@markusressel
Copy link
Collaborator

The last release of python-n26 was released 8 months before this issue was opened, so no, we didn't disable anything 😄
It might very well be that the old API starts to break apart further and further while we have no working alternative in place. If you could read up on the comments within this issue and then try to reproduce the problems I had getting the new APi to work that could help us get it running.

@MaxGemini89
Copy link

@markusressel I had the error 500 like you because for the call to /api/feed/accounts/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/transactions/search I used userId (retrieved from /oauth2/token or /api/me response) instead of accountId (you can find it inside the /api/feed/v2 or /api/smrt/transactions response).
Try to switch this parameter and let me know if you resolve the issue

@markusressel
Copy link
Collaborator

markusressel commented Jan 30, 2023

For all of my tests I used the id property of the response of the /api/me endpoint. There seems to also be a shadowUserId property that I have not tried yet.

Requesting the /api/feed/v2 endpoint does seem to contain an accountId property in some nested items. This actually did result in the correct response, no error 🥳

However, I am not quite sure how the API is intended to be used correctly then, since I can't imagine you have to fetch the feed to get the accountId from one of the items in the list? 🤔

This just doesn't look right 😆

account_id = list(filter(lambda x: x["type"] == "TIMELINE", result["modules"]))[0]["items"][0]["accountId"]

@tranb3r Where did you get the account_id property from?

Also, would you be able to find out the schema of other filters, if there are any?
Things like category/tags, incoming/outgoing/both etc.

@tranb3r
Copy link

tranb3r commented Jan 31, 2023

I'm requesting /api/accounts first. It's easy to get the account id from the response.
Then you can use /api/feed/accounts/{account_id}/transactions/search for search and filters.
I don't have time right now to capture more search criteria, but I'll try to do it soon if this can help you.

@markusressel
Copy link
Collaborator

@tranb3r that would be great, so we can give users the most flexibility. I hope you can reuse the setup you used preciously and safe some time.

I will try /api/accounts, thx.

if that works out the next thing I am thinking about is how to proceed with our API client. Should this be a new method or should this replace the existing API? The latter would be a breaking change, although I don't think many people would be affected by this since the old API is broken anyway...

Same goes for the CLI client, although for that I think it doesn't make a lot of sense to support the old API at all.

@femueller any thoughts on that?

@tranb3r
Copy link

tranb3r commented Feb 1, 2023

You can now get account_id from /api/account/primary.

Here is the schema for transactions filter (/api/feed/accounts/{account_id}/transactions/search)

{
   "filterCriteria":{
      "filters":[
         {
            "criteria":{
               "from":1643628699863,
               "to":1643747219000
            },
            "type":"DATE_RANGE"
         },
         {
            "criteria":{
               "value":"INCOMING"
            },
            "type":"DIRECTION"
         }
      ]
   },
   "searchText":"Music"
}

For the DIRECTION filter type, values are either INCOMING or OUTGOING (remove this filter if you want both directions).

Hope this helps!

@markusressel
Copy link
Collaborator

markusressel commented Feb 1, 2023

Yes thats very helpful! ❤️

Also these scenarios would be of interest:

  • currently the list seems to be limited to 20 items, if you scroll down the list and a new badge of items are loaded, what does the request look like? any query parameters, headers or json body of interest?
    • using trial and error I figured out that a query parameter named limit=50 can be used to specify the amount of items that will be returned, it seems to be limited to 800 items max
    • I figured out that the paginationKey can be passed as a json body param to retrieve the "next badge" of results, using the paginationKey of the previous response
  • if ou click on a tag in the app to search for a specific "tag", any requests of interest?
  • if you go into the "Insights" View, are there any requests of interest?
  • if you go to "Recurring payements" is there a special request or also just a filter?
  • The response contains "searchSuggestions", that suggest that it might be possible to filter by "MERCHANTS", as well as "TAGS" did you see anything related?

Basically: Click on every single screen and button within the app once 🤣

Sorry for "abusing" you here, but if we can figure this out we can create a pretty nice API from it! 🤓

@tranb3r
Copy link

tranb3r commented Feb 2, 2023

For infinite scrolling, you can use the paginationKey value.
It's returned in the /api/feed/accounts/{account_id}/transactions/search response.
And then you pass it to the next request body: {"filterCriteria":{"filters":[]},"paginationKey":"blablabla"}.

Tags and merchants are just values you put in the searchText field in the transactions/search query.

Suggestions: POST /api/feed/accounts/{account_id}/transactions/search/suggestions.
Insights: GET /api/insights/dashboard.
Balance overview: GET /api/insights/balance-overview?page=1.
Recurring payments: GET /api/insights/recurring-payments.
Categories: GET /api/insights/balance-overview/expenses/{category}?period=2023-02 with category being either bars_and_restaurants or food_and_groceries or shopping or leisure_and_entertainment. There are probably more categories, that's just the ones visible in my app.
Scheduled payments: GET /api/scheduled-payments/overview.

@markusressel
Copy link
Collaborator

Awesome! I don't know when I will get to it, but this should provide a good basis for an implementation. I will keep you updated.

@femueller
Copy link
Owner

femueller commented Feb 5, 2023

#110 (comment)

I think it doesn't make a lot of sense to support the old API at all

Very much agreed on that. We could create a new Major release for that if we have any breaking changes.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

6 participants