Plans for 2.0

[qgustavor]

1.0 was released but it was a release intended to be backyards compatible with 0.17 with few modifications. For 2.0 I want to rewrite the library to fix design issues:

API exports

Currently it works like this:

console.log(await import('megajs'))
// { Storage, File, API, file, decrypt, encrypt, verify  }

There are some issues: the File conflicts with the global File object, having to log-in using new Storage() is not intuitive and the usage of large classes make tree shaking less efficient as desirable. If you want to make a webpage that loads the name of the last uploaded file in a folder, something like "Last update: filename.ext", it will load a lot of unneeded methods, even using tree shaking.

Proposed exports:

console.log(await import('megajs'))
// { login, loadFromUrl, upload, download, share, createAccount, ... }

No more File conflicts, and export names would be more intuitive. Also, separating upload and download from the rest of the library will make tree-shaking way more efficient and will make the library more flexible, since the same upload function used when logged-in could be used to upload files in MEGAdrop folders.

The only issue is that switching from file.download(opts) to download(file, opts) may look weird: why not just class methods, right? Well, because those are huge and complicated methods. As for me is the same thing as when in Vue 3 some API methods were moved to be tree-shakable. As it explains well, because "[...] tree-shaking, which is a fancy term for 'dead code elimination.'"

Simplify the upload method

The current upload method is way to complicated: it accepts buffers, anything that Buffer.from accepts and Node readable streams, and also return a writable stream. Sure, internally it's just the writable stream and the input is just written to it, but, since Mega's API requires a file size before upload starts and also requires uploads to be chunked, is it the best way to handle that?

One way to handle it is using blobs since their interface make chunking way more simple: just slice the blob to the chunk start and end then send it. Chunk errors could easily be retried by just reading the same blob again. Since streams are mostly used to reduce memory consumption when dealing with huge files using blobs would be a good thing. In the other hand Blobs are still experimental in Node and don't looks like there is a way to create a blob that references to a file in Node and Deno without loading those entirely in memory at the moment.

It sure the state of blob will change in future, making it way more desirable.

Improve downloading

In order to download you must need to call Mega's API to get a download URL then download from this URL. At the moment if you want to read files with random access (like for streaming videos or reading archive files) the library do not allow doing that first API call once, instead it repeats that call for each read. That's issue #30.

One way to improve it is making download return not a Buffer or a stream, but a object that would store the download URL. Taking jimmywarting suggestion would be a good thing if this object works like blobs:

const file = await loadFromUrl('https://mega.nz/file/example#example')
const downloader = await download(file) // get "blob"
const slicedDownloader = downloader.slice(start, end) //  get a "blob slice"
const data = await downloader.arrayBuffer() // get arraybuffer
const text = await downloader.text() // get string
const stream = downloader.stream() // get stream

Other improvements

• Move internals to use promises and removing callbacks entirely;
• Replace buffers, streams and another Node built-ins with modern JavaScript features;
• Use TypeScript internally instead of manually writing the .d.ts files;
• Replace RSA functions with Web Crypto;
• Implementing account creation and file dropping (which were not implemented because RSA issues).

Finally, 2.0 don't need to be rushed. I prefer waiting for Blob and fetch support become at least a bit more stable in Node first.

If someone have suggestions for 2.0 - i.e. those that require major changes in the library - I will edit this post to include them. Suggestions for the next minor releases will be discussed in other threads.

[RealAlphabet]

According to the NodeJS documentation https://nodejs.org/api/buffer.html#class-blob.

The Blob object represents a blob, which is a file-like object of immutable, raw data.

I don't find the use of Blob desirable. I have another idea to share, why not just add an options parameter to the download function which takes a range property containing start and end. This would download the corresponding part of the file.

If users want the simplicity of blobs, there's nothing stopping us from creating a new BlobDownloader class that relies on the download function internally. This way, if users prefer a more "low level" way to download parts of the file, they are free to do so without having to use Blob. Going back to your three-shaking argument, that would be more desirable in my opinion.

[qgustavor]

MEGA files are immutable when downloading and uploading, right?

About adding the options parameter: that's how it works today. I don't know if you checked the download code and understood it - it's a mess! - but I think that it, in fact, will make it be more low level than it is today. One issue from the current code is that it disallows people getting the raw URL and I saw some forks adding a function to access it. The blob-like API idea allows that: it does not need to be a real blob, it can have a .rawDownloadUrl property that allows people accessing the raw URL. It would optimize non-sequential downloading: the current implementation requests a download URL for each request, in that implementation it would be cached.

About tree shaking I was thinking about Vue and, while in Vue 3 some methods moved from Vue global to named exports allowing them to be tree shackable, there's another pattern Vue uses to make builds lighter: app.use(), in other words, allowing classes to be extendable. Maybe it would require exposing some internals.

[RealAlphabet]

Would it be possible to obtain the link of the fork concerned?
The one to retrieve the raw download URL.

I guess you have to decrypt the file afterwards?
Unfortunately, I don't know enough about how MEGA works yet.

[qgustavor]

datnguyencse@01d30f4

When downloading you request the download URL here then you download chunks from it by requesting ${downloadURL}/${startByte}-${endByte} here.

Edit: about "I guess you have to decrypt the file afterwards?" I guess from other commits in the fork that the objective was using MEGA as a non-encrypted file host, which, as discussed in #108, don't works anymore.