Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: api can't download remote file #1394

Closed
1 task done
toebbel opened this issue Jul 24, 2023 · 11 comments · Fixed by #1397
Closed
1 task done

[Bug]: api can't download remote file #1394

toebbel opened this issue Jul 24, 2023 · 11 comments · Fixed by #1397
Labels
API Issues with the @actual-app/api package bug Something isn't working

Comments

@toebbel
Copy link

toebbel commented Jul 24, 2023

Verified issue does not already exist?

  • I have searched and found no existing issue

What happened?

I try to write a script that interacts with a remote server via @actual/api like described here.

Server version: v23.7.2
@actual/api version: 6.1.1

The script I use:

const api = require('@actual-app/api');

(async () => {
  await api.init({
    //dataDir: './',
    serverURL: '<https url to a fly VM>',
    password: '<server password>',
  });
  console.log(api);

  await api.downloadBudget('< my sync id >', { password: '< file password>' } );

  let accounts = await api.getAccounts();
	for (account in accounts) {
		console.log(account.name);
	}
  await api.shutdown();
})();

I execute the script via node v20.2.0.

I tried the following:

  • use the budget name instead of the id --> I see an error that the sync ID can't be found
  • use another password --> I see an error that the encryption password is wrong
  • create a new budget file on the server, without password protection --> same error as before

What error did you receive?

Error: Something went wrong trying to download that file, sorry! Visit https://actualbudget.org/contact/ for support. (fileId: a96b84c9-c873-41a2-9314-da454c5bc386)
at handlers.api/download-budget (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:32814:31)

Where are you hosting Actual?

Fly.io

What browsers are you seeing the problem on?

Other

Operating System

Mac OSX

@toebbel toebbel added the bug Something isn't working label Jul 24, 2023
@j-f1 j-f1 added the API Issues with the @actual-app/api package label Jul 24, 2023
@j-f1
Copy link
Contributor

j-f1 commented Jul 24, 2023

Can you try downloading the beta API package from this build (the actual-api artifact)? Once you’ve downloaded that zip, uncompress it to get a tgz file. Then pass that to npm install /path/to/actual-api.tgz to install it into your project. Can you try again and let me know what error you get (please send the full error log)? That build has some improved error messaging for this situation that will be included in the next version, but if we can also fix your issue directly that would be great to do too.

@toebbel
Copy link
Author

toebbel commented Jul 24, 2023

Thanks @j-f1 for getting back on this matter so quickly.

I've unzipped the actual-api build artefact and pointed the package.json file to the unzipped path.

This is the error log I see now:

Full error details {
  type: 'FileDownloadError',
  reason: 'internal',
  meta: { fileId: '4a0719ca-fbba-<retracted>' }
}
/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:32223
                        throw new Error((0, _shared_errors__WEBPACK_IMPORTED_MODULE_2__.getDownloadError)(result.error));
                              ^

Error: Something went wrong trying to download that file, sorry! Visit https://actualbudget.org/contact/ for support. (reason: internal, fileId: 4a0719ca-fbba-<retracted>)
    at handlers.api/download-budget (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:32223:31)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

@j-f1
Copy link
Contributor

j-f1 commented Jul 24, 2023

Thanks for checking that! I now see where the error happened. Can you try out this build? That adds a bit more logging. My guess is that somehow the file ID is incorrect, since that’s the main way this could fail (assuming your connection to the server is correct)

@toebbel
Copy link
Author

toebbel commented Jul 24, 2023

Ahh. The error shows something odd

Error fetching file info FetchError: request to https://<redacted>/sync/get-user-file-info failed, reason: socket hang up
    at ClientRequest.<anonymous> (/Users/tobi/projects/actual-to-google-sheets/node_modules/node-fetch/lib/index.js:1505:11)
    at ClientRequest.emit (node:events:511:28)
    at TLSSocket.socketOnEnd (node:_http_client:519:9)
    at TLSSocket.emit (node:events:523:35)
    at endReadableNT (node:internal/streams/readable:1367:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  type: 'system',
  errno: 'ECONNRESET',
  code: 'ECONNRESET'
}
Full error details {
  type: 'FileDownloadError',
  reason: 'internal',
  meta: { fileId: '4a0719ca-fbba-<redacted>' }
}
/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:32223
                        throw new Error((0, _shared_errors__WEBPACK_IMPORTED_MODULE_2__.getDownloadError)(result.error));
                              ^

Error: Something went wrong trying to download that file, sorry! Visit https://actualbudget.org/contact/ for support. (reason: internal, fileId: 4a0719ca-fbba-<redacted>)
    at handlers.api/download-budget (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:32223:31)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Node.js v20.2.0

When I open the path https://<redacted>/sync/get-user-file-info in Safari I see this response

{"status":"error","reason":"unauthorized","details":"token-not-found"}

Same, when I use curl

% curl -v https://<redacted>/sync/get-user-file-info
*   Trying <redacted>:443...
* Connected to <redacted> (<redacted>) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=*.fly.dev
*  start date: Jun  9 23:43:51 2023 GMT
*  expire date: Sep  7 23:43:50 2023 GMT
*  subjectAltName: host "<redacted>" matched cert's "*.fly.dev"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /sync/get-user-file-info]
* h2h3 [:scheme: https]
* h2h3 [:authority: <redacted>]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x12000a800)
> GET /sync/get-user-file-info HTTP/2
> Host: <redacted>
> user-agent: curl/7.88.1
> accept: */*
> 
< HTTP/2 401 
< x-powered-by: Express
< access-control-allow-origin: *
< ratelimit-limit: 500
< ratelimit-remaining: 499
< ratelimit-reset: 42
< content-type: application/json; charset=utf-8
< content-length: 70
< etag: W/"46-arBLVEwoDaCK5mPJtWkb6+x8MW0"
< date: Mon, 24 Jul 2023 20:38:26 GMT
< server: Fly/a0b91024 (2023-06-13)
< via: 2 fly.io
< fly-request-id: 01H64SZTEQ4PMME0SNCJKWFP0H-arn
< 
* Connection #0 to host <redacted> left intact
{"status":"error","reason":"unauthorized","details":"token-not-found"}%       

First thing I tested: change the password that is sent to the server to some gibberish. I do get a different error response:

Error: Could not get remote files

I double-checked the password in my original script: it is the one I use to authenticate to the remote-server.

Second thing: I looked at the request headers that are passed with the /sync request from the web app to the server. I see that a header X-ACTUAL-TOKEN is passed. I attempted to call /sync/get-user-file-info with this header via
curl -H "X-ACTUAL-TOKEN: <redacted>" https://<redacted>/sync/get-user-file-info.

I am also puzzled, because the web app makes requests to sync/list-user-files instead of /sync/get-user-file-info; but I don't have much experience with the code-base or endpoints.

I'll be AFK until tomorrow morning CEST. Thanks for the quick support :)

@j-f1
Copy link
Contributor

j-f1 commented Jul 25, 2023

Yep… you’re actually not the first person with that particular issue. (See https://discord.com/channels/937901803608096828/1123193332697419776/1123193332697419776). Something weird must be happening on Fly’s end. The only thing I can think of trying is changing what headers we send to see what the difference is between the browser/curl and Node?

Did some research, and it looks like it may be this bug which was just fixed & released: node-fetch/node-fetch#1735

Can you try again with this build? It has the updated version of node-fetch.

@toebbel
Copy link
Author

toebbel commented Jul 25, 2023

Oh wow - great find!

I attempted to test the build, but I get this error:

/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/index.js:33
const node_fetch_1 = __importDefault(require("node-fetch"));
                                     ^

Error [ERR_REQUIRE_ESM]: require() of ES Module /Users/tobi/projects/actual-to-google-sheets/node_modules/node-fetch/src/index.js from /Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/index.js not supported.
Instead change the require of /Users/tobi/projects/actual-to-google-sheets/node_modules/node-fetch/src/index.js in /Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/index.js to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/index.js:33:38)
    at Object.<anonymous> (/Users/tobi/projects/actual-to-google-sheets/index.js:1:13) {
  code: 'ERR_REQUIRE_ESM'
}

Node.js v18.17.0

I'm not sure, but it seems like the dist folder contains javascript that has some invalid require statements?

I have first tried with node version 20.2.0, then 20.5.0 and now I switched to the LTS version of node. My package.json only contains this:

{
  "dependencies": {
    "@actual-app/api": "file:/Users/tobi/Downloads/actual-api"
  }
}

@j-f1
Copy link
Contributor

j-f1 commented Jul 25, 2023

Yep, that’s my fault. Can you give the build from here a shot? I’ve updated the code so that it uses import() to import the node-fetch module (I was misled because our source code uses ESM syntax, but the built code is in CJS for compatibility). I also removed the reference to node-fetch from the bundle.api.js file.

@toebbel
Copy link
Author

toebbel commented Jul 27, 2023

I'm always puzzled by web pack options. Thanks for looking into this!
Unfortunately, I get a different error now:

/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:28559
            const fetch = self.fetch;
                          ^

ReferenceError: self is not defined
    at ./packages/loot-core/src/platform/server/fetch/index.api.ts (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:28559:27)
    at __webpack_require__ (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:45954:48)
    at ./packages/loot-core/src/server/post.ts (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:38398:92)
    at __webpack_require__ (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:45954:48)
    at ./packages/loot-core/src/server/sync/index.ts (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:40342:76)
    at __webpack_require__ (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:45954:48)
    at ./packages/loot-core/src/server/db/index.ts (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:36879:75)
    at __webpack_require__ (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:45954:48)
    at ./packages/loot-core/src/server/accounts/sync.ts (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:30106:73)
    at __webpack_require__ (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:45954:48)

Node.js v18.17.0

@j-f1
Copy link
Contributor

j-f1 commented Jul 27, 2023

Sorry about that! Gave the code in this build a more through test and you should not be seeing that issue now.

@toebbel
Copy link
Author

toebbel commented Jul 28, 2023

awesome - this works like a charm!
Thank you very much @j-f1 for the great support :)

@andrewzolotukhin
Copy link

Hi! I'm still getting the same problem using the latest (6.3.0) version of API. Just wondering if the was released?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API Issues with the @actual-app/api package bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants