[meta] Tracking issue for Node.js/Deno standard library inspired functions to include in @tauri-apps/api

[amrbashir]

I will be refactoring our js api and aiming to make the it more familiar to users coming from electron/js-land so it is easier to migrate.

Some of current functions will be deprecated and removed when new variants are released.

We want to gather some feedback from users and which functions they want to add from the Nodejs standard library, so comment below and I will add them to this list of new functions:

1. os module

   • EOL
   • platform()
   • version()
   • tempdir()
   • hostname

2. path module

   • join()
   • normalize()
   • resolve()
   • sep
   • delimiter
   • dirname()
   • basename()
   • extname()
   • isAbsolute()

3. fs module

   • create()
   • open()
   • close()
   • FsFile class
   • writeFile()
   • writeTextFile()
   • readFile()
   • readTextFile()
   • copyFile()
   • readDir()
   • mkdir()
   • remove()
   • rename()
   • exists()

4. net module

   • createConnection()

5. dns module

[Shotman]

Suggestion : get complete directory tree as JSON from a given folder, along with files with their metadata
getFileTree('dir')

[amrbashir]

Suggestion : get complete directory tree as JSON from a given folder, along with files with their metadata
getFileTree('dir')

Sorry, I forgot to mention I am accepting only function suggestion from Nodejs standard library for now.

Once I am done with this, I think I will make a plugin for extended api and will accept suggestion for anything.

[Verequies]

It would be great to have 'net.createConnection' from the Net API so that discord-rpc can work.
I would be happy to make an example project that can be tested with if needed.

[amrbashir]

It would be great to have 'net.createConnection' from the Net API so that discord-rpc can work.

I added createConnection to the list for now, but I will ask @nothingismagick , our security expert, if it is okay to include it or not since it will deal with sockets.

I would be happy to make an example project that can be tested with if needed.

I will try to do my best but I am not sure if this will be a drop in replacement for the NodeJS one.

[jvail]

Would be nice to have a synchronous api to access to files in a way that allows open/read/write/close (as mentions here #1025) to directly read/write large files without copying them into memory:

Something like https://github.com/emscripten-core/emscripten/blob/main/src/library_nodefs.js

[amrbashir]

Would be nice to have a synchronous api to access to files in a way that allows open/read/write/close (as mentions here #1025) to directly read/write large files without copying them into memory:

Something like https://github.com/emscripten-core/emscripten/blob/main/src/library_nodefs.js

A synchronous api is not possible, but I will add nodejs async function open which returns a FileHandle class

[jvail]

A synchronous api is not possible, but I will add nodejs async function open which returns a FileHandle class

Sorry, a bit off-topic but Is that a design decision or technically impossible?

I will be refactoring our js api and aiming to make the it more familiar to users coming from electron/js-land so it is easier to migrate

Since electron offers sync fs it will be difficult to migrate. Are there ways to customize/extend tauri to make a sync fs available in a webworker?

[amrbashir]

Sorry, a bit off-topic but Is that a design decision or technically impossible?

yeah it is a kinda impossible technically.

Since electron offers sync fs it will be difficult to migrate. Are there ways to customize/extend tauri to make a sync fs available in a webworker?

you can wrap your logic in an async function and await our api calls.
here is an example using an async IIFE.

(async () => {
	doSomething();
	let file = await fs.readFile();
	doSomethingWithFile(file);
})();

[jquesada2016]

Sorry, a bit off-topic but Is that a design decision or technically impossible?

yeah it is a kinda impossible technically.

Since electron offers sync fs it will be difficult to migrate. Are there ways to customize/extend tauri to make a sync fs available in a webworker?

you can wrap your logic in an async function and await our api calls.
here is an example using an async IIFE.

(async () => {
	doSomething();
	let file = await fs.readFile();
	doSomethingWithFile(file);
})();

Regarding sync function calls, I do understand the technical limitations on why promises must be used. But just in case, it is possible to emulate (though inefficiently) a blocking synchronous function in JS from an asynchronous function. This can be done as follows:

const syncFunctionNotDoneSymbol = Symbol(/* named, if you prefer */);
let valueToBeReturned = syncFunctionNotDoneSymbol;

export function syncFunction() {
    asyncFunctionCall().then(result => valueToBeReturned = result);

    while(valueToBeReturned === syncFunctionNotDoneSymbol) {
        // burn CPU cycles...
    }

    // Reset the value, so subsequent function calls still work.
    let val = valueToBeReturned;
    valueToBeReturned = syncFunctionNotDoneSymbol;

    return val;
}

Two important things to note on my primitive implementation above:

1. The functionality can be made into a simple helper function to convert any async function into a sync one (haven't looked in NPM to see if such a disgrace already exists...
2. I am not sure if my above implementation works if the function is called from worker threads, as I don't know if the symbol definition is re-evaluated when called from another thread context. Though it is definitely possible to make it work if it doesn't by default by using a simple array as a sort of concurrent queue for the symbols. If you would like me to show you an example, let me know.

It would also be a good idea, if such functionality is desired, to add a disclaimer in the sync function docs where you heavily discourage their use, due to them eating CPU cycles polling for the promise. But it might serve as a simple shim for large code bases until they are ported over.

[chippers]

Regarding sync function calls, I do understand the technical limitations on why promises must be used. But just in case, it is possible to emulate (though inefficiently) a blocking synchronous function in JS from an asynchronous function. This can be done as follows:

The impossible part isn't exposing a sync api, it was the other part of the request.

to directly read/write large files without copying them into memory

This is difficult in the current design because we use message passing to communicate with the WebView, meaning Rust loads the file and passes the bytes over to the webview into JS where it is consumed. To do that without copying into memory, there would need to be a way from JavaScript to directly load a file from the filesystem through the native WebView. Browsers don't have this, and AFAIK the native WebView libraries we use don't expose some way to do this.

As for your sync version of async methods, I'm not sure if we will expose pseudo sync functions. IMO it matters less to have them since all the supported browsers have await. If we don't expose them, it could still be a simple/contained package that wraps the async api. I also think that the JavaScript Generator API would more succinctly allow you to turn it into a sync call.

[jvail]

Thank you for your thoughts and insight, but ...

This is difficult in the current design because we use message passing to communicate with the WebView, meaning Rust loads the file and passes the bytes over to the webview into JS where it is consumed

excuse the heresy - this is pretty much what I can do already in a good old browser: Ask for a file, load it entirely into memory, process it and ask for a save-as dialog. This is fine for small files but let's say I'd like to process a 500MB geotiff then I'll soon hit the limitations of the browser (and tauri's).

The use-case I have in mind is running an emscripten compiled WebAssembly in a web worker to process local files. As of now I can not embed the wasm directly in rust (even if I could it has some advantages being able to run it within the webview). With electron I can run it in a worker and by setting nodeIntegrationInWorker: true the lib gets full read/write access to the filesystem.

I do not dare to ask for design changes but maybe there are ways to allow more customization on the rust level to inject an API (e.g. filesystem) in the worker context bypassing tauris async command (serde) api. Being able to combine WebAssembly, workers and tauri would create really interesting opportunities.

[chippers]

Thank you for your thoughts and insight, but ...

This is difficult in the current design because we use message passing to communicate with the WebView, meaning Rust loads the file and passes the bytes over to the webview into JS where it is consumed

excuse the heresy - this is pretty much what I can do already in a good old browser: Ask for a file, load it entirely into memory, process it and ask for a save-as dialog. This is fine for small files but let's say I'd like to process a 500MB geotiff then I'll soon hit the limitations of the browser (and tauri's).

The use-case I have in mind is running an emscripten compiled WebAssembly in a web worker to process local files. As of now I can not embed the wasm directly in rust (even if I could it has some advantages being able to run it within the webview). With electron I can run it in a worker and by setting nodeIntegrationInWorker: true the lib gets full read/write access to the filesystem.

I do not dare to ask for design changes but maybe there are ways to allow more customization on the rust level to inject an API (e.g. filesystem) in the worker context bypassing tauris async command (serde) api. Being able to combine WebAssembly, workers and tauri would create really interesting opportunities.

I agree with you, that would be a really interesting interface and I would love to see it someday soon. Electron is in a fortunate place regarding this due to controlling both the backend and a custom build of Chromium. We provide Wry by default in Tauri to support the "native" OS browser (iOS/macOS = WkWebView, Linux = webkit2gtk, Windows = webview2) which would require all those libraries to support such functionality, and then apply Tauri APIs over it.

As for regarding "this is pretty much what I can do already in a good old browser", that is relatively true. Browsers will let you open a local file from a prompt, some experimental APIs will even let you do it without a prompt, if the files are in the correct directory.

These types of things can likely be implemented by a Tauri plugin, but only if the underlying platform supports it. I haven't look into it at all, but if you are interested in it then the results may be fruitful.

Additionally, if you did want to support a custom runtime including things like preventing copying memory when reading large files, then you can implement Runtime from tauri-runtime as the backend of your Tauri application. It does take a decent chunk of work to implement your own runtime, so beware if you decide to do so.

[ghost]

I don't see it listed as a task (or maybe I misread), but adding file permissions when writing a file (not after the fact - race condition) would help tremendously for electron apps that needed to store sensitive data (e.g. passwords, secrets) that are not readable by groups or others.

Example:

// mode should be configurable, just showing as an example:
File::with_options().mode(600).open("somefile.txt");

[amrbashir]

@zoombinis it is listed under fs section, all current functions will have new overloads that is similar to nodejs standard library, with support for file permissions

Edit: updated the list for more transparency

[Lancear]

I really need the net.createConnection function for my app to create an IPC connection. How much work would it be to implement that? Could someone with little/basic Rust experience implement it? Would love to help with that if it means I can use Tauri instead of Electron for my app!

[amrbashir]

@Lancear we are in a code-freeze phase and there won't be any new features to Tauri v1, so any new features will land in Tauri v2 check #2060 for more info.

With that said, I gotta warn you, we might not be able to exactly replicate NodeJS' net.createConnection() but we will try our best. I also wanna note that fs module has higher priority than net module so it might be a while until we start on it.

Pull requests to the next branch are welcome though.

[KaKi87]

Hello,

I would also like to request the ability to read a file line by line, like the following in Node :

import {
    createReadStream
} from 'node:fs';
import {
    createInterface
} from 'node:readline';
for await (const line of createInterface({
    input: createReadStream(
        path,
        {
            flags: 'a+'
        }
    ),
    crlfDelay: Infinity
})){
    // TODO something with `line`
}

Not necessarily by implementing exactly the same modules and methods as long as it does the same thing.

Thanks

[amrbashir]

@KaKi87 thanks for the suggestion, it has now been implemented in the linked PR and will look like this:

const lines = await readTextFileLines('file.txt', { baseDir: BaseDirectory.App });
for await (const line of lines) {
  console.log(line);
}

Also, since each line will be lazily read as you advance through the iterator, it makes this function perfect for performance critical apps as you can manually advance the iterator by calling .next() whenever you want.

[lucasfernog]

Nice one @amrbashir

[KaKi87]

each line will be lazily read as you advance through the iterator, it makes this function perfect for performance critical apps

Exactly, thanks !

[SamuelScheit]

Any updates on the net api?

[jlarmstrongiv]

Does the fs module support streaming apis like:

• fs.createReadStream
• fs.createWriteStream
• fs.readableWebStream

I was looking for ways to read files without loading the entire file into memory, but found issues like #4133 and this one

Any workarounds for the current version of tauri?

[KaKi87]

Any workarounds for the current version of tauri?

If you need it that bad, then you could call an external tool (maybe cat) using shell.Command.

[jlarmstrongiv]

@KaKi87 that’s great for reading files! Do you have a workaround for writing files too?

[KaKi87]

echo.

[jlarmstrongiv]

Ahh, I was looking at the Command interface, not the Child interface. My bad. Thank you @KaKi87!

[jlarmstrongiv]

Actually, with echo, it isn’t possible to add the redirect to send the content to a file. Plus, with cross-platform scripts, it becomes even more challenging. It may be easiest to include a two sidecar binaries:

• one that reads a file, converts the binary to hex, and prints hex
• one that reads hex, converts it to binary, and writes it to a file

I hope that a streaming version of file reading/writing can be added to the tauri fs library

[KaKi87]

Actually, with echo, it isn’t possible to add the redirect to send the content to a file.

You can, using bash -c.

Plus, with cross-platform scripts, it becomes even more challenging.

Indeed, my proposal should work as much on Mac as it does on Linux, but different commands would for sure be required on Windows.

It may be easiest to include a two sidecar binaries

Of course. That, or writing Rust code.

[adminy]

any plans for child_process module support?

I find that whatever apps I write, I always end up using
child_process due to how easy it is to just CLI a solution
quickly and then Worry about writing proper library
bindings later if the app is actively developed and maintained.

How quickly I can sketch up an app is significantly faster with
just calling a cli whether its spawn or execSync to just
perform a command. Speed of development can be quite the deal breaker.

[amrbashir]

@adminy We already have this, see https://tauri.app/v1/api/js/shell

[amrbashir]

We decided to most of the JS APIs into its own plugins in https://github.com/tauri-apps/plugins-workspace, most of the APIs proposed here are already implemented there or waiting for merge. The fs changes will have to wait for #6107.

If anything is missing or doesn't have a PR open, feel free to open an issue in that repo.

[KaKi87]

Where's the documentation for the fs-extra plugin ?

[amrbashir]

fs-extra plugin has been removed and its functionality has been integrated into tauri-plugin-fs for v2. The documentation is still WIP.

[KaKi87]

So, concretely, where will the readTextFileLines method be available ?

[amrbashir]

That will be implemented in tauri-plugin-fs for v2 possibly with #6107

[KaKi87]

So it will never be available in v1 ?

[amrbashir]

Not planned atm but it can be implemented in your app:

1. The function in typescript

async function readTextFileLines(
  path: string
): Promise<AsyncIterableIterator<string>> {

  return Promise.resolve({
    path,
    rid: null as number | null,
    async next(): Promise<IteratorResult<string>> {
      if (!this.rid) {
        this.rid = await invoke<number>('read_text_file_lines', { path: this.path })
      }

      const [line, done] = await invoke<number>('read_text_file_lines_next', { rid: this.rid })

      // an iteration is over, reset rid for next iteration
      if (done) this.rid = null

      return {
        value: done ? '' : (line as string),
        done
      }
    },
    [Symbol.asyncIterator](): AsyncIterableIterator<string> {
      return this
    }
  })
}

2. The command in rust

#[tauri::command]
fn read_text_file_lines<R: Runtime>(app: AppHandle<R>, path: PathBuf) -> Result<u32, std::io::Error> {
    use std::io::{BufRead, BufReader};
    let file = File::open(path)?;
    let lines = BufReader::new(file).lines();

    let rid = 0 // generate a uniuqe ID for this request

	// Assuming you have a `ResourceTable` struct where you store your resources
	// and you stored it inside tauri state 
	// by calling `app.manage(ResourceTable::new())` in the `setup` function

	// grab the resource table
    let mut resource_table = app.state::<ResourceTable>().inner().lock().unwrap();
    // store the lines iterator inside the resource table
    resource_table.insert(lines);
    
    // return the rid
    Ok(rid)
}

#[tauri::command]
fn read_text_file_lines_next<R: Runtime>(app: AppHandle<R>, rid: u32) -> Result<(Option<String>, bool), std::io::Error> {
    use std::io::Lines;

	// grab the resource table
    let mut resource_table = app.state::<ResourceTable>().inner().lock().unwrap();
    // get the stored lines iterator for this rid
    let lines = resource_table.get(&rid).unwrap();

    let ret = lines
      .next()
      .map(|l| (l.ok(), false))
      .unwrap_or_else(|| {
        let _ = resource_table.remove(&rid).unwrap();
        (None, true)
      });

    Ok(ret)
  }

[amrbashir]

Most of the APIs in this issue have been implemented in path module in @tauri-apps/api and in tauri-apps-plugin-fs starting 2.0.0-alpha.5

If there is any missing APIs, feel free to open an issue in plugins-workspace repo.

[Artrix9095]

updates on net?

[amrbashir]

No work has been done on net module unfortunately, and it is not a priority atm.