Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

repl: allow await in REPL #13209

Closed
benjamingr opened this issue May 24, 2017 · 88 comments
Closed

repl: allow await in REPL #13209

benjamingr opened this issue May 24, 2017 · 88 comments

Comments

@benjamingr
Copy link
Member

@benjamingr benjamingr commented May 24, 2017

I've seen this requested a few times in StackOverflow now:

People have code that is asynchronous and would like to use values from it in the REPL:

// how do I access in the REPL? returns promise
Transaction.where('reference', '1').fetch().then((res) => { return res }); 

A common workaround is to store a global reference in the REPL, which is a little hacky and relies on timing, but can work for simple cases:

> Transaction.where('reference', '1').fetch().then((res) => out = res)
[Object Promise]
> out
   /* response available here*/

This works, but has timing issues and generally isn't great. We could however run the REPL in the context of an async function potentially - which would allow awaiting values in the REPL:

let res = await Transaction.where('reference', '1').fetch(); 

I think it could be an interesting enhancement but would like more feedback on the idea. @nodejs/collaborators

@not-an-aardvark
Copy link
Contributor

@not-an-aardvark not-an-aardvark commented May 24, 2017

👍

I looked into implementing this when Node 7.6.0 came out and async functions were originally added. At first glance it seemed like it would be a bit difficult because the REPL currently doesn't run in a function context at all. However, there might be a better solution that I'm not aware of.

@refack
Copy link
Member

@refack refack commented May 24, 2017

I'm +1.
But there are userland solutions, quick googling found https://github.com/skyrising/await-repl, https://github.com/StreetStrider/repl.js, and I'm sure there are more

@not-an-aardvark
Copy link
Contributor

@not-an-aardvark not-an-aardvark commented May 24, 2017

await-repl seems to only await a promise if the line starts with "await ", which is probably insufficient (e.g. it doesn't allow foo = await bar().

repl.js seems to auto-await all Promises that are returned from an eval, but that also probably isn't what we want (and it doesn't allow for nested await either).

@refack
Copy link
Member

@refack refack commented May 24, 2017

I'm not arguing that this is a beneficial feature. But IMHO the REPL is an exploratory tool, so those tools at least enable easier exploration 🤷‍♂️
On the other hand the cli -e option, again IMHO, is a more "important" feature to enable easy await usage for.
node -e "await new Promise.resolve(4)"

@vkurchatkin
Copy link
Contributor

@vkurchatkin vkurchatkin commented May 24, 2017

The question is, is it really possible? It seems like it's not

@refack refack added the cli label May 24, 2017
@not-an-aardvark
Copy link
Contributor

@not-an-aardvark not-an-aardvark commented May 24, 2017

It would definitely be possible in theory by reimplementing await semantics and parsing, but that seems like a lot of work. Ideally there would be some --allow-top-level-await flag in V8 that we can use.

@joyeecheung
Copy link
Member

@joyeecheung joyeecheung commented May 25, 2017

There was a feature request issue for this and there are some solutions proposed there: #8382 although it seems most of them are somewhat hacky

@benjamingr
Copy link
Member Author

@benjamingr benjamingr commented May 25, 2017

@vkurchatkin

The question is, is it really possible? It seems like it's not

Why can't we just wrap the content with an async function?

@ChALkeR
Copy link
Member

@ChALkeR ChALkeR commented May 25, 2017

@benjamingr, that will change the semantics of repl. We need to detect it somehow, or use a flag, or force this as a semver-major change for everyone.

@benjamingr
Copy link
Member Author

@benjamingr benjamingr commented May 25, 2017

@ChALkeR what semantics would it change? I figured that since the REPL is not synchronous anyway (being user input and all) the microtick of wrapping it in an async function - unwrapping it and giving the user back the result wouldn't be too bad.

Am I missing anything?

@targos
Copy link
Member

@targos targos commented May 25, 2017

@benjamingr do you have ideas about how to make this work:

> const a = Promise.resolve(42);
undefined
> const b = await a;
undefined
> b
42

If we wrapped the code in an async function, b would be local to this function. How to unwrap it?

@benjamingr
Copy link
Member Author

@benjamingr benjamingr commented May 25, 2017

@targos you're absolutely right! I totally missed the scoping issue.

Parsing the line and figuring out if it's a variable declaration seems easier than detecting an async function, it would still be problematic for things like f(); const a = 5; which are legit.

I'm wondering if we could have an escape hatch at the V8 level perhaps? Optimally something that lets V8 know it should evaluate the code passed as an async function?

Essentially, we'd need a way for Script to take an async function (well, the Script::Compile method).

I'm wondering if that's possible - @nodejs/v8 @fhinkel ?

@bergus
Copy link

@bergus bergus commented May 26, 2017

This also needs some kind of cancellation option - like Ctrl+C - to stop waiting. Something like

await new Promise(() => {}) // forever pending

should not break the REPL

@Fishrock123
Copy link
Member

@Fishrock123 Fishrock123 commented May 29, 2017

This seems like an odd idea to me, what does the repl do during an await? Does it just stall and not respond to further user input? How do you know when the await is completed otherwise?

Either way seems confusing and poor design to me, maybe I'm not considering something?

I suppose it's not much different that running a long sync operation although that is probably less common.

@Qard
Copy link
Member

@Qard Qard commented May 29, 2017

Yeah, I think stalling is the desired behaviour. The idea is to be able to inspect the return values of a promise API and use them for subsequent operations in a more friendly way. Currently you have to attach then() handlers to write values into global s and check repeatedly to see when they become available. It's very unfriendly to promise users.

@refack
Copy link
Member

@refack refack commented May 29, 2017

Does it just stall and not respond to further user input? How do you know when the await is completed otherwise?

Maybe show a spinner?

@addaleax
Copy link
Member

@addaleax addaleax commented May 29, 2017

Does it just stall and not respond to further user input? How do you know when the await is completed otherwise?

Maybe show a spinner?

Sounds good. As long as you can see that something is happening and you can abort it with Ctrl+C, we should be good. :)

@vkurchatkin
Copy link
Contributor

@vkurchatkin vkurchatkin commented May 29, 2017

Maybe show a spinner?

Sounds inappropriate for REPL. Aborting is a must, though. That said, I think it's a waste of time to discuss such details until we figure out a way to do this.

@bmeck
Copy link
Member

@bmeck bmeck commented May 30, 2017

at some point people might want to think of foreground and backgrounding await if you need to test how 2 async things interact... like Jobs in shells

@refack
Copy link
Member

@refack refack commented May 30, 2017

foreground and backgrounding await

Isn't there a key for that in bash? CTRL-Z? than also jobs and bg and fg? (not a bash expert)
We already have .* REPL commands, so .jobs / .fg / .bg doesn't sound absurd...

@RReverser
Copy link
Member

@RReverser RReverser commented May 30, 2017

It would definitely be possible in theory by reimplementing await semantics and parsing, but that seems like a lot of work.

FWIW that's what regenerator already did (as used by Babel for transpiling generators and async-await to ES5).

So for a problematic example above, wrapped into an async function:

(async function() { 
 const b = await a;
})();

it would generate

(function _callee() {
  var b;
  return regeneratorRuntime.async(function _callee$(_context) {
    while (1) {
      switch (_context.prev = _context.next) {
        case 0:
          _context.next = 2;
          return regeneratorRuntime.awrap(a);

        case 2:
          b = _context.sent;

        case 3:
        case "end":
          return _context.stop();
      }
    }
  }, null, this);
})();

From here, we could easily unwrap the IIFE and execute body, which would solve the scope propagation issue as all the modified variables would be visible after the body is executed.

However I'm not sure if we would want to integrate such, even minimal, third-party transpiler into Node.js REPL.

But if we do, I could try to make this happen :)

@chicoxyzzy
Copy link

@chicoxyzzy chicoxyzzy commented May 30, 2017

IMO REPL should be predictable. await is valid identifier outside of async function. So it should work as identifier.

@addaleax
Copy link
Member

@addaleax addaleax commented May 30, 2017

However I'm not sure if we would want to integrate such, even minimal, third-party transpiler into Node.js REPL.

If you can make this happen and the code doesn’t make the node binary turn huge, I would be in favour of doing that.

@ebraminio
Copy link
Contributor

@ebraminio ebraminio commented May 30, 2017

I think you should consult some of v8 guys for help and taking best strategy here as the same feature would be nice for Chrome console as well so a similar approach could be used for node.js I guess.

(BTW I am very happy that this is proposed here as the idea of top-level await itself seems is rejected but all I personally liked to have was a console one)

@ebraminio
Copy link
Contributor

@ebraminio ebraminio commented May 30, 2017

Filed as http://crbug.com/727924 Edited: already available on http://crbug.com/658558

@ebraminio
Copy link
Contributor

@ebraminio ebraminio commented May 30, 2017

According to this comment and my test, this is already supported on Safari so I guess Chrome team should/will soonish pick up the task to fill the gap up with the other browser.

@Krinkle
Copy link
Contributor

@Krinkle Krinkle commented May 2, 2018

For the record:

  1. Yes, this was landed in master in December 2017, and has been released in Node 10.
  2. But, it has since been put behind a CLI flag: --experimental-repl-await, per #19604.

Node 10 with --experimental-repl-await provides a working REPL, today, that supports await!

🎉

@esteban-uo
Copy link

@esteban-uo esteban-uo commented May 15, 2018

thanks for supporting it!. I don't know if it is just me, but in Node 10.1, it just hangs out while using REPL await... anyone else experiencing the same?

@vsemozhetbyt
Copy link
Contributor

@vsemozhetbyt vsemozhetbyt commented May 15, 2018

@esteban-uo It seems you need --experimental-repl-await for this now.

@esteban-uo
Copy link

@esteban-uo esteban-uo commented May 15, 2018

@vsemozhetbyt yes and actually it works just fine 👍 Seems to be it happens when an exception occurs, which is very confusing sometimes, since you could think that the promise can't be resolved at all or a timeout is happening

@domainoverflow
Copy link

@domainoverflow domainoverflow commented May 29, 2018

Forgive me if I am misreading but according to the above I would be able to CLI node app.js where app.js could contain awaitS placed outside functions ( ex: copy and paste from puppeteersandbox.com examples without wrapping and cli running it ) ?

@Krinkle
Copy link
Contributor

@Krinkle Krinkle commented May 29, 2018

@domainoverflow Thank you for asking!

This issue is only about using "await" in the special "REPL" mode of Node.js. If you run node as CLI command without JS file, that is the REPL mode. This mode behaves similarly to the DevTools console in a web browser.

REPL is a different from how programs run normally. And for JS programs, it is not possible to use await outside async functions because the JavaScript programming language is specified in a way that does not allow this. There is a proposal from TC39 to allow "await" outside functions in normal programs. More about that at https://github.com/tc39/proposal-top-level-await. This was relatively easy to do in REPL for debugging and testing, but is less easy for normal programs. If the proposal is approved, then the JavaScript engine for Node.js (Google V8) may implement the change, after which it may eventually become available in Node.js.

@techsin
Copy link

@techsin techsin commented Sep 4, 2018

+1

@jiayuzhang
Copy link

@jiayuzhang jiayuzhang commented Feb 3, 2019

@esteban-uo It seems you need --experimental-repl-await for this now.

Any way to specify the experimental flag programmatically (instead of command line)? since I'm using repl.start api

@vsemozhetbyt
Copy link
Contributor

@vsemozhetbyt vsemozhetbyt commented Feb 3, 2019

@mariapryimak
Copy link

@mariapryimak mariapryimak commented Apr 25, 2019

@jiayuzhang in linux repl module will inherit the flag --experimental-repl-await, and replServer.start() will have top level await, but it seems like it doesn't inherit it in other OSes and it doesn't work. Looks like a bug.

@filips123
Copy link

@filips123 filips123 commented Jul 1, 2019

Is there any specific reason why is await under an experimental flag? Why is it not enabled by default?

@devsnek
Copy link
Member

@devsnek devsnek commented Jul 1, 2019

@filips123 when await is enabled we run input through a separate parser which doesn't necessarily match the way node normally parses code, so there could be incompatibilities. In the long run, a combination of top-level-await and a standardized repl parse goal should fix this.

https://github.com/tc39/proposal-top-level-await
https://github.com/bmeck/js-repl-goal

@rayfoss
Copy link

@rayfoss rayfoss commented Jul 19, 2019

Summary, it works with an experimental flag. use something like the following to load some js and drop you into a repl after the variables are set, kind of like bashrc.

#!/bin/bash
node  --experimental-repl-await -i -e "$(< scripts/repl.js)"`

With enough run commands, it might be possible to replace bash with node lol
https://stackoverflow.com/questions/33850903/use-node-js-as-shell

@nahtnam
Copy link

@nahtnam nahtnam commented Sep 5, 2019

Has anyone figured out how to get the experimental flag to work in a REPL that is spawned with .start()? The following snippet doesn't work.

process.env.NODE_OPTIONS = '--experimental-repl-await';
const local = repl.start({ prompt: '> ' });
@aherlihy
Copy link

@aherlihy aherlihy commented Oct 3, 2019

Having the same issue as @nahtnam with trying to get await mode turned on from repl.start. Any suggestions?

@bapti
Copy link

@bapti bapti commented Dec 1, 2019

I did this and it worked

"scripts": {
    "repl": "node --experimental-repl-await ./repl.js"
}

And in repl.js

const { readFileToArray } = require("./utils");
const repl = require("repl");

repl.start("> ").context.readFileToArray = readFileToArray;

When I run it it works fine and I can await on my function readFileToArray

@tomasgvivo
Copy link

@tomasgvivo tomasgvivo commented Mar 27, 2020

Top level await is now supported in V8 (https://v8.dev/features/top-level-await)
Is it possible to add something like vm.runAsyncInContext to avoid using a different parser for code containing async in repl?

Also, I could really use vm.runAsyncInContext for a jupyter-notebook-like node project.

@devsnek
Copy link
Member

@devsnek devsnek commented Mar 28, 2020

@tomasgvivo TLA is part of modules. Once we land support for it (#30370), you can use it in a vm.SourceTextModule instance.

There is another thing V8 is working on called "repl mode" which might help with await in the repl though: https://docs.google.com/document/d/1bIKMRrLVObsAfmMAQ-tmsWkQZu1cFg87RRrPpiwTL2M/edit

@cyrus-and
Copy link

@cyrus-and cyrus-and commented Apr 26, 2020

When the adding the command line --experimental-repl-await is not an option, one could use this hack to always await promises on a custom REPL:

const defaultEval = myRepl.eval;
myRepl.eval = (cmd, context, filename, callback) => {
    defaultEval(cmd, context, filename, async (err, result) => {
        if (err) {
            // propagate errors from the eval
            callback(err);
        } else {
            // awaits the promise and either return result or error
            try {
                callback(null, await Promise.resolve(result));
            } catch (err) {
                callback(err);
            }
        }
    });
};

It is not a general solution but worked well for my use case.

@YoraiLevi
Copy link

@YoraiLevi YoraiLevi commented Jun 3, 2020

When the adding the command line --experimental-repl-await is not an option, one could use this hack to always await promises on a custom REPL:

const defaultEval = myRepl.eval;
myRepl.eval = (cmd, context, filename, callback) => {
    defaultEval(cmd, context, filename, async (err, result) => {
        if (err) {
            // propagate errors from the eval
            callback(err);
        } else {
            // awaits the promise and either return result or error
            try {
                callback(null, await Promise.resolve(result));
            } catch (err) {
                callback(err);
            }
        }
    });
};

It is not a general solution but worked well for my use case.

// index.js
const repl = require("repl");
const myRepl = repl.start();

const defaultEval = myRepl.eval;
myRepl.eval = (cmd, context, filename, callback) => {
  defaultEval(cmd, context, filename, async (err, result) => {
    if (err) {
      // propagate errors from the eval
      callback(err);
    } else {
      // awaits the promise and either return result or error
      try {
        callback(null, await Promise.resolve(result));
      } catch (err) {
        callback(err);
      }
    }
  });
};

usage node index.js
type:

function delayPromise(delay){
    //async delay function
    return new Promise(resolve => setTimeout(() => resolve(), delay))
}
delayPromise(3000)
undefined
//wait 3 seconds until can proceed typing
pstaender added a commit to pstaender/strapi that referenced this issue Jun 26, 2020
As stated here nodejs/node#13209 (comment) , the feature is available since node ^10. So there should not be any drawbacks for any supported Node version
@andyvanee
Copy link

@andyvanee andyvanee commented Oct 20, 2020

I'm not sure why, but using --experimental-repl-await affects how util.inspect.defaultOptions are interpreted. In normal code you could adjust the output of console.log using the following:

import util from "util"
util.inspect.defaultOptions.getters = true
util.inspect.defaultOptions.depth = 4
util.inspect.defaultOptions.maxStringLength = 80

But when using --experimental-repl-await it needs to be changed to this:

import util from "util"
util.inspect.replDefaults.getters = true
util.inspect.replDefaults.depth = 4
util.inspect.replDefaults.maxStringLength = 80

The docs seem to suggest that the replDefaults should mirror defaultOptions, but this doesn't seem to be the case when using the async version of the REPL.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.