Replace request with fetch (and fix other bugs)

[qgustavor]

As issue #64 is stale I plan to at least replace request with fetch to remove the warning that NPM shows when installing the library.

Because the existent build process is old it was causing some issues, so I replaced Rollup with esbuild and updated some dev dependencies to reduce the number of reported vulnerabilities on them to zero.

But, because of that, now the distributed code is not ES5 but ES6 and it require more modern platforms, both Node and browser. Anyway, the important thing is making the library work, adding ES5 support, if someone wishes it, will come later.

Checklist:

• Fix backpressure in upload function which broke when fixing browser/Deno version;
  • Pausing a PassThrough stream, unlike npm's through, do not stop data from passing thought via a pipe;
  • Solution: implement a PassThrough stream that pauses using Transform;
• Fix other issues that can arise after fixing the above;
  • Uploading never calls end or complete functions
    • Solution: do not override Stream methods
  • "Promise resolution is still pending but the event loop has already resolved." in Deno
  • "spawn deno ENOENT" in CI
  • "spawn npx ENOENT" in CI
  • "Error: Dynamic require of "abort-controller" is not supported" in Node V14
• Add test case for don't throw error on empty file #83;
• Add types;
• Write new documentation (as the wiki do not allow versioning easily, docusaurus allow);
• Wait for Ship final version of 17.0.0 standard/eslint-config-standard#208 then fix issues found in files that could not be linted;
• Wait for Mem leak when keepalive is used node-fetch/node-fetch#1446 then fix eventual issues;
• Add promise support
• Release preview version and wait for feedback (remember to add tag);
• Release 1.0;

[qgustavor]

I updated the branch with some fixes, I tested in Node and I could download some files. Both browser and Deno are not working well, but I will put those aside for the moment.

There are still a important pending issue: downloading only works reliably when using single-connection downloading, trying to use chunked downing results in the server disconnecting (socket hang up).

I'm checking megatools source-code and the logic is pretty the same, but it retries failed chunks. How about creating something to handle chunk errors like megatools? If so, how? Hardcode a exponential backoff like megatools or make something more configurable?

Other possibility is trying to detect what is causing the server to drop the connection. In those changes I added a HTTP(S) agent to keep the connection alive and a default user agent (which should be used anyway), but the issue persists.

Edit: Travis finished, for some reason tests are failing in Node 14 with Error: Cannot find module '@babel/core'. Is it an issue in Ava? It it persists I will remove this version from the CI.

[qgustavor]

I will add a handleRetries arguments to .download() which accepts a function and defaults to the below:

function handleRetries (tries, error, cb) {
  if (tries > 8) {
    cb(error)
  } else {
    setTimeout(cb, 1000 * Math.pow(2, tries))
  }
}

The first argument is the current number of tries, the second is the last error and the third is a callback. Calling the callback with an argument will throw an error in the stream and stop the download. The default logic is similar to the one in megatools but instead of capping the max wait to 4 minutes, it stops the download.

By making it an argument library users can override this logic to match their needs without having to modify the library code. A similar argument can be added to the upload function too.

Edit 1: If someone thinks a function like that needs more arguments or, maybe, even replacing (tries, error, cb) with ({tries, error}, cb) please comment before I publish it.

Edit 2: The default retry handler can be overridden globally by replacing File.defaultHandleRetries with other function.

[qgustavor]

Tests are getting a lot of warnings because of this: node-fetch/node-fetch#1446 I will wait this issue to be fixed first.

Edit 1: I downgraded node-fetch to v2 in order to resume development of this library, I will return to v3 when the above issue gets fixed. Ava tests are passing without warnings now.

In the other hand Ava's tests use the mock server, not real servers, and for some reason they continue hanging up. Retrying chunks sure helped to reduce the number of failed downloads, but there are some cases when some chunks fail more than eight times, so while it was an improvement, it was not the solution. I will need to do more tests first to guess what's going on.

Edit 2: Node's own debugger do not support debugging HTTP(S) requests. Using a debugging proxy changes the behavior (it worked fine when I attached Fiddler). I will try to fix the browser build and debug using a browser.

[qgustavor]

The code now looks good for me. The only thing missing is node-fetch issue to be fixed. Would be great if someone test this branch.

Edit: I think that's better to try fixing more issues before publishing, like fixing those CI checks that are failing. The issue is that Ava's testing the source code, not compiled esbuild code, using Babel and Babel is failing on Node 14.

Is better to rewrite the tests to use only compiled esbuild code, Some internal functions are not exported, so I will need to make esbuild export a build just for testing that include them. Is better to remove the current browser test code as it's just testing the pure-JavaScript versions of crypto functions and add a test using real browsers later.

Edit 2: I think will be better to replace Travis with Github Actions. From my experience it works better. Also, while a test with real browsers may come later, testing with Deno seems to be a good thing as it's similar to browsers: the only things that change between Node and browser builds is crypto code and HTTP requesting code, and in those two cases the same code used for browsers also work in Deno.

Edit 3: Is not like someone really depends on this project or cares for it - except me - so I will take my own time to fix it.

[qgustavor]

Now, from the issues I want to fix, only Typescript related ones are missing. Nice. And I need to sleep.

Edit: Nevermind, I still need to fix the browser build and make Deno-based tests work.

Edit 2: Also need to add tests for the logout and verification methods. Also move documentation from the wiki to something else where versioning is supported.

[qgustavor]

I could not resist and I fixed the issue with Deno types. The documentation is updated. I did some simple tests and everything is working well, it's even type checking events, which is great.

[qgustavor]

Seems chunked download is broken: I just got two corrupted downloads. I think the issue is how the retry mechanism works since storage testing do not test it (mega-mock do not simulate a failed connection).

Edit: well, there is no test downloading the large file uploaded before, so the issue might be other thing. I will change this test to include some deterministic random looking data instead of zeros to detect if chunks are merged in the wrong order.

Edit 2: I'm modifying the streamed upload test to also test downloading... and it's failing.

Edit 3: streamed uploading and chucked downloading are broken.

Edit 4: for some reason source is switching to paused mode and the data stops. Calling read() multiple times make the data events emit again.

Edit 5: Checking documentation I checked the readableFlowing and it's false. Since neither .pause() nor .unpipe() are being called the only reason for that is backpressure. Seems the issue relies in the pauseStream implementation which is not handling backpressure correctly.

Edit 6: Seems it will be better to rewrite the upload function to not depend on neither pauseStream nor pumpify. 1.0 will have to wait.

Edit 7: I changed the upload function to not depend on pauseStream and it's now working in Chrome and Deno, but not in Node. Weird.

Edit 8: When I try to debug it using Node's debugger the bug vanishes. This is a Heisenbug!

[qgustavor]

Another issue: error handling in .upload() got messed because, since a promise is always returned, the error event is never called, instead the promise will be rejected. What to do? Check if the error event have a listener?

Edit: That will be the fix.

[qgustavor]

Looks like there is a race-condition in the download function which causes chunks to be read out of order. Weird.

[qgustavor]

storage › Should upload and download huge files in parts Promise returned by test never resolved

Looks like streaming upload and/or download are randomly failing. It needs more tests.