Feature wanted: PTY support for interactive REPLs

[leana8959]

Problem Description

When used with Haskell's Cabal like this: watchexec -e --restart 'hs' 'cabal repl', the backspace fails to work.

Expected

Using backspace would delete the last character.

Actual

Using backspace creates an escaped code \?

Attempts

Using -n or --no-environment didn't change this behaviour.
If I remember correctly, this didn't happen with Scala's sbt in repl mode.

Misc

• Watchexec's version:
  watchexec 1.23.0 (2023-08-30)
  build-date: 2023-08-30
  release: 1.23.0
  features:

• The OS you're using:
  macOS 13.5.2

• A log with -vvv --log-file
  https://gist.github.com/leana8959/6d7afb8d892d5138eac4349bba7c51ae

Thank you for the help, have a nice day :)

[leana8959]

Update: I installed rlwrap and it worked like a charm. It is now working as normal when I use the command watchexec --restart -e 'hs' 'rlwrap cabal repl' instead.
However, I'm leaving this issue open for the moment, since I don't know if this is just bypassing a potential problem of watchexec, which closing the issue would leave it unfixed.

Thank you!

[passcod]

Hmm. I think that's because there's no TTY available for the command. Presumably some repls (I've tried node) use STDIN, and some use TTY, and the latter won't work properly.

I'm actually looking to add pseudo-ttys to watchexec in the future via https://lib.rs/crates/pty-process, though I wasn't coming at it from this angle, so that's more motivation to do so!

[max-sixty]

I added some notes on experiments in #209.

One more thing I just found and makes the current state much better:

Running without -r makes typing into pytest work fine even after restart. It's only with -r that we get into the state described in #209.

(I would otherwise have a preference to run with -r. And it's still subtly different from waiting until the program exits and then restarting, since it captures C-c to exit watchexec, rather than pytest capturing it. But it's still very tolerable)

[unphased]

hmmmm so i am launching a shell script (which implements a great deal of logic for how to handle watchexec events) from watchexec, which in turn launches my program which is a node.js script that serves a webserver. Because I found that it was not possible to convince watchexec on its own to properly terminate my node script the shell script implements a bunch of logic that boils down to sending a sigint to the process group of the node process, this works well for me.

Now what I'm wanting is to be able to pipe stdin given to the watchexec/shellscript down into the node process. Which is actually being launched in the background.

I think that means I'm looking at the same situation here, but I also think rlwrap on its own won't help me. I need a real pty here. Right?

[unphased]

i realized once i switch to the json event stream execution mode instead of the program-capturing-env-var-providing execution mode, then it puts me in control over control flow, I can insert watching related logic into my app proper, and the above complexities will vanish.

[passcod]

You can install watchexec from source from #699 it's essentially done, I just need to finish writing cross platform tests. That has the json-event-stream mode.

[unphased]

I mean i think i already have it since you helped set me up with it the other day

watchexec 1.23.0 (2023-11-15)
build-date: 2023-11-15
release: 1.23.0

[passcod]

Oh, very true. In any case, I plan on releasing 1.24.0 in the next half hour.

[mgunyho]

Hi, thank you for watchexec!

I was running into similar problems when trying to run python -i script.py, where -i drops down to a REPL after the script has finished. I found that the following command works for me:

watchexec -e py -r --no-process-group 'rlwrap --always-readline python3 -i script.py'

This works both with or without -c (but not -c reset, see below). Here are other combinations of flags I tried:

watchexec -e py -r --no-process-group python3 -i script.py

• Input and output work normally on first run.
• After the script is modified and re-run, keystrokes are not shown but they are recorded and pressing enter works as expected. I think this is the same behavior that @max-sixty reported.
• Also, if I exit the REPL with ctrl-D and then stop watchexec with ctrl-C, the keystrokes are invisible in the bash prompt too. (This doesn't happen with xonsh, the shell I normally use.)

watchexec -e py -r --no-process-group 'rlwrap python3 -i script.py'

• Works as expected, except that rlwrap outputs a warning:

  rlwrap: warning: rlwrap appears to do nothing for python3, which asks for
  single keypresses all the time. Don't you need --always-readline
  and possibly --no-children? (cf. the rlwrap manpage)
  

watchexec -e py -r --no-process-group 'rlwrap --always-readline python3 -i script.py'

• Seems to work as expected, and no warning is printed.
• Also seems to work fine with -c

watchexec -e py -r --no-process-group --clear=reset 'rlwrap --always-readline python3 -i script.py'

• Input works as normal, but whenever I press enter, the text I have written is replaced by the output, see the video below.
• Again, after exiting the prompt and watchexec with ctrl-D + ctrl-C, the bash prompt is broken.

watchexec-python-i-clear-reset2.mp4

Running without -r doesn't work for my use case (like it does for pytest), since the point is to just kill the REPL and re-run the script when it is modified.

The input doesn't work at all if I leave out --no-process-group, which from what I understand is the expected behavior. (Also, if I leave it out, rlwrap says rlwrap: error: Unexpected error: Input/output error when I input any keystroke, and then it stops working, it no longer seems to accept keystrokes).

I hope this contains at least some useful information. I'm not sure if Python is using a pty or not. The output of python -h for the -i flag says "forces a prompt even if stdin does not appear to be a terminal".

Watchexec version is

watchexec 1.25.1 (2024-01-14) +pid1
build-date: 2024-01-14
release: 1.25.1
features: default,pid1

rlwrap is version 0.46.1

Python is version 3.11.6.

I'm using foot as the terminal emulator, on Fedora 38.

[passcod]

The input doesn't work at all if I leave out --no-process-group, which from what I understand is the expected behavior.

That's correct

---

Thanks for the detailed report! I've made early exploratory work into adding pty support to watchexec, and it's not as trivial as I originally thought, but will keep trying.

[unphased]

FWIW I have been extremely satisfied with the JSON output only mode, in this mode the control flow is not inverted, which is a lot more natural for most applications. In this way, watchexec is only responsible for its main job of handling filesystem events, instead of providing an execution environment and all of the requisite complexities. We would need to manage the launching of watchexec and properly handle its output stream in this case.

Of course, for general usability having both approaches available is ideal.

[passcod]

I'm also working on a tool called super which does the process supervision bit separate from watchexec, so if you still want the process supervision aspects but without the inversion of control, you (will soon) can.

[unphased]

very neat! yeah dealing with processes is always tricky. lately what i tend to do the most is node and between async/await and streams, it provides some good tools for manipulating processes and their output. I have been finding Typescript pretty ergonomic lately. But this does not feel very unixy.

[passcod]

Watchexec 2.0.0 has --wrap-process=session, which may help. I've also done the background work laying the path for proper PTY support, which first needs to land in https://github.com/watchexec/process-wrap and then can easily flow on to Watchexec.