cannot unmarshal object into Go struct field TransactionResponse.items.subAmount of type string #27

dhojayev · 2024-06-05T20:13:48Z

Describe the bug
Fetching timeline transactions fails on JSON unmarshalling stage because we never expected that subAmount field could containing something else but string.

Jun  5 21:27:33.866 [PANI] could not fetch transactions: could not unmarshal timelineTransactions response: json: cannot unmarshal object into Go struct field TransactionResponse.items.subAmount of type string
panic: (*logrus.Entry) 0xc0002b0a80

goroutine 1 [running]:
github.com/sirupsen/logrus.(*Entry).log(0xc0002b0150, 0x0, {0xc000338180, 0xb6})
        /go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:260 +0x491
github.com/sirupsen/logrus.(*Entry).Log(0xc0002b0150, 0x0, {0xc0000a5e60?, 0xc0001d1180?, 0xc0002a4ab0?})
        /go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x48
github.com/sirupsen/logrus.(*Logger).Log(0xc0001d1180, 0x0, {0xc0000a5e60, 0x1, 0x1})
        /go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/logger.go:204 +0x58
github.com/sirupsen/logrus.(*Logger).Panic(...)
        /go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/logger.go:253
main.main()
        /github/workspace/cmd/portfoliodownloader/public/main.go:44 +0x43b

To Reproduce
Waiting for the sample JSON

Expected behavior
It should not fail on this stage, especially because we do not even use this field anywhere within our code.

The text was updated successfully, but these errors were encountered:

tenjaa · 2024-06-06T08:47:52Z

Do you have a way to share the JSON privately? Or should I go through it and kill all personal data?

tenjaa · 2024-06-06T08:54:09Z

    {
      "action": {
        "payload": "48efbcad-f320-5979-8d64-<XXX>",
        "type": "timelineDetail"
      },
      "amount": {
        "currency": "EUR",
        "fractionDigits": 2,
        "value": -5.95
      },
      "badge": null,
      "eventType": "card_successful_transaction",
      "icon": "logos/merchant-d86e2356-3f2c-4e12-bb84-<XXX>/v2",
      "id": "48efbcad-f320-5979-8d64-<XXX>",
      "status": "EXECUTED",
      "subAmount": null,
      "subtitle": null,
      "timestamp": "2024-05-27T13:51:55.167+0000",
      "title": "<XXX>"
    },
    {
      "action": {
        "payload": "8334ce43-36e0-43cc-9a23-<XXX>",
        "type": "timelineDetail"
      },
      "amount": {
        "currency": "EUR",
        "fractionDigits": 2,
        "value": -5.36
      },
      "badge": null,
      "eventType": "benefits_spare_change_execution",
      "icon": "logos/<XXX>/v2",
      "id": "8334ce43-36e0-43cc-9a23-<XXX>",
      "status": "EXECUTED",
      "subAmount": null,
      "subtitle": "Round up",
      "timestamp": "2024-05-23T13:59:05.402+0000",
      "title": "<XXX>"
    },
    {
      "action": {
        "payload": "701e379c-a992-520a-b5c9-<XXX>",
        "type": "timelineDetail"
      },
      "amount": {
        "currency": "EUR",
        "fractionDigits": 2,
        "value": -157.93
      },
      "badge": null,
      "eventType": "card_successful_transaction",
      "icon": "logos/merchant-fallback-entertainment/v2",
      "id": "701e379c-a992-520a-b5c9-<XXX>",
      "status": "EXECUTED",
      "subAmount": {
        "currency": "CZK",
        "fractionDigits": 2,
        "value": -3900
      },
      "subtitle": null,
      "timestamp": "2024-05-23T11:37:27.519+0000",
      "title": "<XXX>"
    },

I had a look myself and here are two examples of transactions with a subAmount that is not a string.
Personal data and some parts of the ids are replaced by <XXX>.

dhojayev · 2024-06-06T12:14:12Z

Hey @tenjaa,
thanks for the data.
We could find a way to share the json privately, but I would still like all identifiable data to be removed. I can work with the examples you shared. I am already working on the fix for your case, it should be an easy hotfix for now. Will keep you posted.

dhojayev · 2024-06-06T13:54:46Z

Hey @tenjaa
Please download the binary using the link below and check that it doesn't fail anymore. I fixed the issue and will release it if you could confirm that it works.
https://github.com/dhojayev/traderepublic-portfolio-downloader/actions/runs/9402123472

Thanks

tenjaa · 2024-06-06T14:07:54Z

It starts downloading files successfully but then fails.

Jun  6 16:08:33.731 [INFO] [id:f8023427-d358-4263-9a18-17ff59a01ce7] Fetching transaction details (/home/runner/work/traderepublic-portfolio-downloader/traderepublic-portfolio-downloader/cmd/portfoliodownloader/app.go:59 github.com/dhojayev/traderepublic-portfolio-downloader/cmd/portfoliodownloader.App.Run)
Jun  6 16:08:33.731 [TRAC] [message:sub 74 {"id":"f8023427-d358-4263-9a18-17ff59a01ce7","token":"ey<XXX>","type":"timelineDetailV2"}] sent message (/home/runner/work/traderepublic-portfolio-downloader/traderepublic-portfolio-downloader/internal/api/websocket/reader.go:96 github.com/dhojayev/traderepublic-portfolio-downloader/internal/api/websocket.(*Reader).Read)
Jun  6 16:08:33.731 [TRAC] [message:73 C] received msg (/home/runner/work/traderepublic-portfolio-downloader/traderepublic-portfolio-downloader/internal/api/websocket/reader.go:104 github.com/dhojayev/traderepublic-portfolio-downloader/internal/api/websocket.(*Reader).Read)
Jun  6 16:08:33.759 [TRAC] [message:74 E {"errors":[{"errorCode":"NOT_FOUND","errorField":null,"errorMessage":"Could not find resource","meta":{"source":"PLATFORM"}}]}] received msg (/home/runner/work/traderepublic-portfolio-downloader/traderepublic-portfolio-downloader/internal/api/websocket/reader.go:104 github.com/dhojayev/traderepublic-portfolio-downloader/internal/api/websocket.(*Reader).Read)

Jun  6 16:06:56.718 [PANI] could not fetch transaction details: could not fetch timelineDetailV2: error state received: 75 E {"errors":[{"errorCode":"NOT_FOUND","errorField":null,"errorMessage":"Could not find resource","meta":{"source":"PLATFORM"}}]}
panic: (*logrus.Entry) 0xc00019ae00

goroutine 1 [running]:
github.com/sirupsen/logrus.(*Entry).log(0xc0003ac540, 0x0, {0xc0004f8380, 0xe0})
        /home/runner/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:260 +0x491
github.com/sirupsen/logrus.(*Entry).Log(0xc0003ac540, 0x0, {0xc0004dfe60?, 0xc0002e4e30?, 0xc0002506f0?})
        /home/runner/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x48
github.com/sirupsen/logrus.(*Logger).Log(0xc0001d1180, 0x0, {0xc0004dfe60, 0x1, 0x1})
        /home/runner/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/logger.go:204 +0x58
github.com/sirupsen/logrus.(*Logger).Panic(...)
        /home/runner/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/logger.go:253
main.main()
        /home/runner/work/traderepublic-portfolio-downloader/traderepublic-portfolio-downloader/cmd/portfoliodownloader/public/main.go:44 +0x43b

dhojayev · 2024-06-06T14:38:27Z

This is interesting, it is definitely something else. could you post the json for this ID please? I will meanwhile disable exiting when details cannot be fetched but rather skip the transactions for now. But your json still could help to understand and fix the problem

tenjaa · 2024-06-06T14:46:52Z

I don't think I can. I enabled to write responses but there is no file called f8...json in responses/timelineDetailV2 . Or do I have to look for something else?

dhojayev · 2024-06-06T15:00:51Z

It should be in responses/timelineTransactions with the filename like page-X.json. Just try searching by content please. responses/timelineDetailV2 cannot yet contain anything as it is exactly the problem you facing, that the app cannot fetch the details, Trade Republic says that details for that ID do not exist.

tenjaa · 2024-06-06T15:08:40Z

Found it

    {
      "action": {
        "payload": "f8023427-d358-4263-9a18-17ff59a01ce7",
        "type": "timelineDetail"
      },
      "amount": {
        "currency": "EUR",
        "fractionDigits": 2,
        "value": 13.67
      },
      "badge": null,
      "eventType": "ORDER_EXECUTED",
      "icon": "logos/timeline_person/v2",
      "id": "f8023427-d358-4263-9a18-17ff59a01ce7",
      "status": "EXECUTED",
      "subAmount": null,
      "subtitle": "Verkaufsorder",
      "timestamp": "2021-09-22T12:06:16.619+0000",
      "title": "<XXX>"
    },

dhojayev · 2024-06-06T15:17:20Z

This is super strange. You can see in the action section it has an ID to fetch timelineDetail, which we are doing in this case, but then TR responds that the object doesn't exist. I guess that it happens because this order is old it requires the outdated websocket message type to fetch the details. But then I am not sure if the response there will be similar to timelineDetailV2. So I will simply skip it for now. But it was really helpful that you found it.
Unfortunately, this means that this transaction won't be in your CSV, but we can work further together to see if implementing support for it is possible or not. Let me quickly finish skipping it for now and I will come back with the plan on what to do next.

dhojayev · 2024-06-06T15:19:47Z

@tenjaa If it would even be possible it would really help if you could find this transaction in TR mobile or web app (I know this means a lot of scrolling, and this is exactly the reason why our app exists :)) and check if you can press on it and see the details page. Thank you!

dhojayev · 2024-06-06T15:50:44Z

@tenjaa please download the new revision from here and this time I hope you will have your csv and docs :)
https://github.com/dhojayev/traderepublic-portfolio-downloader/actions/runs/9403979524

tenjaa · 2024-06-06T16:11:16Z

@tenjaa If it would even be possible it would really help if you could find this transaction in TR mobile or web app (I know this means a lot of scrolling, and this is exactly the reason why our app exists :)) and check if you can press on it and see the details page. Thank you!

Doesn't work. I found it but when clicking on it it seems to load indefinately. There is no timeout but it is in that state for over a minute now already.

tenjaa · 2024-06-06T16:14:20Z

@tenjaa please download the new revision from here and this time I hope you will have your csv and docs :) https://github.com/dhojayev/traderepublic-portfolio-downloader/actions/runs/9403979524

It still shows an error but does not crash and continues successfully!

could not fetch transaction details: could not fetch timelineDetailV2: error state received: 75 E {"errors":[{"errorCode":"NOT_FOUND","errorField":null,"errorMessage":"Could not find resource","meta":{"source":"PLATFORM"}}]}

dhojayev · 2024-06-06T16:16:23Z

@tenjaa If it would even be possible it would really help if you could find this transaction in TR mobile or web app (I know this means a lot of scrolling, and this is exactly the reason why our app exists :)) and check if you can press on it and see the details page. Thank you!

Doesn't work. I found it but when clicking on it it seems to load indefinately. There is no timeout but it is in that state for over a minute now already.

Thanks a lot for spending time and confirming my theory. Even TR app cannot not fetch the info, so maybe we could use the data from the first response only, I will create an issue for that later.

dhojayev · 2024-06-06T16:17:19Z

@tenjaa please download the new revision from here and this time I hope you will have your csv and docs :) https://github.com/dhojayev/traderepublic-portfolio-downloader/actions/runs/9403979524

It still shows an error but does not crash and continues successfully!
could not fetch transaction details: could not fetch timelineDetailV2: error state received: 75 E {"errors":[{"errorCode":"NOT_FOUND","errorField":null,"errorMessage":"Could not find resource","meta":{"source":"PLATFORM"}}]}

This is exactly how I wanted it to be: error is shown to you, but the app doesn't fail. So I can release it if it finished all transactions with no errors?

tenjaa · 2024-06-06T16:26:17Z

In my opinion yes.
There are still over 50% of documents skipped for me but I guess that is somewhat expected reading through the docs.
In case something important might be missing I would just open another issue :)

dhojayev · 2024-06-06T16:58:38Z

Great.
Which other documents you mean: account related documents (like Jahressteuerbericht, etc) or other transaction-related documents? I can prioritise implementation if there is request.

In my opinion yes. There are still over 50% of documents skipped for me but I guess that is somewhat expected reading through the docs. In case something important might be missing I would just open another issue :)

tenjaa · 2024-06-06T17:21:50Z

In the end my goal is to put all trades into portfolio performance.

There are two ways to import data: csv and parsing the pdf directly (see here).
So my first thought was to download all pdfs and import them. I'll report back if something is missing or not working for my flow.
Thanks!

dhojayev · 2024-06-06T18:33:45Z

In the end my goal is to put all trades into portfolio performance.

There are two ways to import data: csv and parsing the pdf directly (see here). So my first thought was to download all pdfs and import them. I'll report back if something is missing or not working for my flow. Thanks!

I will have a look into it. Was not familiar with that app. Maybe it would make sense to consider exporting csv for it in future. Thank you for your assistance.

martinAtGitHub · 2024-06-07T04:48:49Z

In my opinion yes. There are still over 50% of documents skipped for me but I guess that is somewhat expected reading through the docs. In case something important might be missing I would just open another issue :)

for me it is the credit card transactions that are skipped as unknown transactions

dhojayev · 2024-06-07T07:03:44Z

In my opinion yes. There are still over 50% of documents skipped for me but I guess that is somewhat expected reading through the docs. In case something important might be missing I would just open another issue :)

for me it is the credit card transactions that are skipped as unknown transactions

Hey @martinAtGitHub,
Is it "unknown" or "unsupported"? If the latter, then this is by design as we only wanted investment related transactions to be in the csv file. There was an idea to create a separate csv file for card transactions, but it has not been planned properly till now. Plus, card transactions contain no documents attached to them, so there is not much data to extract from there.

martinAtGitHub · 2024-06-13T05:03:40Z

Hey @martinAtGitHub, Is it "unknown" or "unsupported"? If the latter, then this is by design as we only wanted investment related transactions to be in the csv file. There was an idea to create a separate csv file for card transactions, but it has not been planned properly till now. Plus, card transactions contain no documents attached to them, so there is not much data to extract from there.

Thanks for the response! It is "Unsupported". So as of now it is the intended behaviour.
Since there is not that much reporting functionality in traderepublic it would be quite convinient, to have the credid card transactions in a CVS file.

dhojayev · 2024-06-13T20:11:49Z

Hey @martinAtGitHub, Is it "unknown" or "unsupported"? If the latter, then this is by design as we only wanted investment related transactions to be in the csv file. There was an idea to create a separate csv file for card transactions, but it has not been planned properly till now. Plus, card transactions contain no documents attached to them, so there is not much data to extract from there.

Thanks for the response! It is "Unsupported". So as of now it is the intended behaviour. Since there is not that much reporting functionality in traderepublic it would be quite convinient, to have the credid card transactions in a CVS file.

If that helps I can implement including those transactions optionally. But there are a few questions that would be good to clarify from the very beginning:

Shall we include those transactions in the same csv file?
If yes - shall we simply write them under the "credit" and "debit" columns?

martinAtGitHub · 2024-06-16T06:50:57Z

If that helps I can implement including those transactions optionally. But there are a few questions that would be good to clarify from the very beginning:
* Shall we include those transactions in the same csv file?
* If yes - shall we simply write them under the "credit" and "debit" columns?

I would take the option that causes the least effort for you.

same CSV or different CSV would be both fine for me
and yes, just populating the columns with the available data would be perfect

dhojayev · 2024-06-16T21:00:42Z

I created a dedicated issue for that. We can definitely implement it if it is helpful to at least one person. We already receive all necessary information from TR, so it is not that much of a hassle. However, I still don't know if writing those transactions into the same csv file is a good idea. I am thinking about having a separate file for that. I guess we all agree that it is not part of the portfolio but rather the "bank side" of Trade Republic.

martinAtGitHub · 2024-06-23T09:25:19Z

a separate file would be perfectly fine for me, too.

dhojayev self-assigned this Jun 5, 2024

dhojayev mentioned this issue Jun 5, 2024

Do not exit with fatal error on unknown transactions #26

Closed

dhojayev linked a pull request Jun 6, 2024 that will close this issue

27 cannot unmarshal object into go struct field transactionresponseitemssubamount of type string #28

Merged

dhojayev closed this as completed in #28 Jun 6, 2024

dhojayev added the bug Something isn't working label Jun 6, 2024

dhojayev mentioned this issue Jun 16, 2024

Implement writing TR card transactions into CSV file #36

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cannot unmarshal object into Go struct field TransactionResponse.items.subAmount of type string #27

cannot unmarshal object into Go struct field TransactionResponse.items.subAmount of type string #27

dhojayev commented Jun 5, 2024 •

edited

Loading

tenjaa commented Jun 6, 2024

tenjaa commented Jun 6, 2024 •

edited

Loading

dhojayev commented Jun 6, 2024

dhojayev commented Jun 6, 2024

tenjaa commented Jun 6, 2024 •

edited

Loading

dhojayev commented Jun 6, 2024

tenjaa commented Jun 6, 2024

dhojayev commented Jun 6, 2024

tenjaa commented Jun 6, 2024

dhojayev commented Jun 6, 2024

dhojayev commented Jun 6, 2024

dhojayev commented Jun 6, 2024

tenjaa commented Jun 6, 2024

tenjaa commented Jun 6, 2024

dhojayev commented Jun 6, 2024

dhojayev commented Jun 6, 2024

tenjaa commented Jun 6, 2024

dhojayev commented Jun 6, 2024

tenjaa commented Jun 6, 2024

dhojayev commented Jun 6, 2024

martinAtGitHub commented Jun 7, 2024

dhojayev commented Jun 7, 2024

martinAtGitHub commented Jun 13, 2024

dhojayev commented Jun 13, 2024

martinAtGitHub commented Jun 16, 2024

dhojayev commented Jun 16, 2024

martinAtGitHub commented Jun 23, 2024

cannot unmarshal object into Go struct field TransactionResponse.items.subAmount of type string #27

cannot unmarshal object into Go struct field TransactionResponse.items.subAmount of type string #27

Comments

dhojayev commented Jun 5, 2024 • edited Loading

tenjaa commented Jun 6, 2024

tenjaa commented Jun 6, 2024 • edited Loading

dhojayev commented Jun 6, 2024

dhojayev commented Jun 6, 2024

tenjaa commented Jun 6, 2024 • edited Loading

dhojayev commented Jun 6, 2024

tenjaa commented Jun 6, 2024

dhojayev commented Jun 6, 2024

tenjaa commented Jun 6, 2024

dhojayev commented Jun 6, 2024

dhojayev commented Jun 6, 2024

dhojayev commented Jun 6, 2024

tenjaa commented Jun 6, 2024

tenjaa commented Jun 6, 2024

dhojayev commented Jun 6, 2024

dhojayev commented Jun 6, 2024

tenjaa commented Jun 6, 2024

dhojayev commented Jun 6, 2024

tenjaa commented Jun 6, 2024

dhojayev commented Jun 6, 2024

martinAtGitHub commented Jun 7, 2024

dhojayev commented Jun 7, 2024

martinAtGitHub commented Jun 13, 2024

dhojayev commented Jun 13, 2024

martinAtGitHub commented Jun 16, 2024

dhojayev commented Jun 16, 2024

martinAtGitHub commented Jun 23, 2024

dhojayev commented Jun 5, 2024 •

edited

Loading

tenjaa commented Jun 6, 2024 •

edited

Loading

tenjaa commented Jun 6, 2024 •

edited

Loading