Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to capture output of programs that use alternate mode #2404

Closed
nh2 opened this issue Jun 12, 2017 · 18 comments
Closed

How to capture output of programs that use alternate mode #2404

nh2 opened this issue Jun 12, 2017 · 18 comments

Comments

@nh2
Copy link
Contributor

@nh2 nh2 commented Jun 12, 2017

In bash you can do

OUTPUT=$(fzf)

then select an entry in the fzf list (which runs in alternate mode), and afterwards you can

echo $OUTPUT

and will obtain what you selected.

In xonsh, this doesn't work:

  • OUTPUT = $(fzf) doesn't show the alternate mode (fzf is running but invisible)
  • OUTPUT = $[fzf] makes it visible but that doesn't capture the final stdout that fzf produces

How to do this?

@nh2
Copy link
Contributor Author

@nh2 nh2 commented Jun 16, 2017

This is actually somewhat a dealbreaker, because in bash you can do git show $(git branch | fzf) and it works out of the box as expected, but in xonsh it doesn't.

@nh2
Copy link
Contributor Author

@nh2 nh2 commented Jun 16, 2017

Is it possible that this is just a bug?

The docs say:

The $() syntax captures and returns the standard output stream of the command as a Python string. This is similar to how $() performs in Bash. For example,

But in bash $() doesn't capture stderr, and it seems that in xonsh it does.

To reproduce:

$ echo $(bash -c '(>&2 echo "error")')

In bash this prints error, in xonsh this prints nothing. All-in-one repro:

$ bash -c "echo $(bash -c '(>&2 echo error)')"
error

$ xonsh -c "echo $(bash -c '(>&2 echo error)')"

Another indication that this might be a bug is that the function that implements $() is called subproc_captured_stdout() which doesn't sound at all like it should capture stderr.

@gforsyth
Copy link
Collaborator

@gforsyth gforsyth commented Jun 17, 2017

Hey @nh2, thanks for reporting! Can you please paste the output of xonfig? fzf works on my machine, although the usage will vary slightly since xonsh is not bash.
so I can do

> $OUTPUT = $(git branch | fzf)
> $OUTPUT
'  master\n'

Not sure about those leading spaces, but the trailing newline is the expected behavior of xonsh grabbing output from something.

For your second comment, instead of doing

git checkout $(git branch | fzf)

you can do

git checkout @$(git branch | fzf)

where the @$ operator will automatically strip the spaces and trailing newline. (This is also how you would accomplish the bash-like vim $(which xonsh) -> vim @$(which xonsh)

@nh2
Copy link
Contributor Author

@nh2 nh2 commented Jun 18, 2017

$ xonfig
+------------------+----------------------+
| xonsh            | 0.5.12               |
| Git SHA          | e9129111             |
| Commit Date      | Jun 18 01:29:22 2017 |
| Python           | 3.5.2                |
| PLY              | 3.9                  |
| have readline    | True                 |
| prompt toolkit   | 1.0.14               |
| shell type       | prompt_toolkit       |
| pygments         | 2.2.0                |
| on posix         | True                 |
| on linux         | True                 |
| distro           | Ubuntu               |
| on darwin        | False                |
| on windows       | False                |
| on cygwin        | False                |
| is superuser     | False                |
| default encoding | utf-8                |
| xonsh encoding   | utf-8                |
| encoding errors  | surrogateescape      |
+------------------+----------------------+
$ $OUTPUT = $(git branch | fzf)
^C

@gforsyth Ah, it actually works for you? For me it just hangs, not showing any output, as in the paste above.

@nhoad
Copy link

@nhoad nhoad commented Jul 11, 2017

FWIW, I'm having some very similar issues.

Capturing the output of fzf actually breaks my terminal such that I can't highlight text, or interact with it via my mouse at all. If I run xonsh with readline instead of prompt_toolkit, then mouse interaction causes tty scruft to start getting printed to the screen. I'm not sure which specific changeset introduced this, but I've figured out that this started happening in 0.5.5. If I run fzf again without capturing it and ^C it, then my terminal returns to normal.

Also, capturing fzf output sometimes works, e.g. I'm selecting master in both invocations below:

$ xonsh --no-rc

Welcome to the xonsh shell (0.5.12)

~ Break out of your shell ~


xonfig tutorial    ->    Launch the tutorial in the browser
xonfig wizard      ->    Run the configuration wizard and claim your shell 
(Note: Run the Wizard or create a ~/.xonshrc file to suppress the welcome screen)

nhoad@nhoad-laptop ~/projects/xonsh master $ xonfig
+------------------+----------------------+
| xonsh            | 0.5.12               |
| Git SHA          | f6fb9ba5             |
| Commit Date      | Apr 18 04:41:11 2017 |
| Python           | 3.6.1                |
| PLY              | 3.9                  |
| have readline    | True                 |
| prompt toolkit   | 1.0.14               |
| shell type       | prompt_toolkit       |
| pygments         | 2.2.0                |
| on posix         | True                 |
| on linux         | True                 |
| distro           | arch                 |
| on darwin        | False                |
| on windows       | False                |
| on cygwin        | False                |
| is superuser     | False                |
| default encoding | utf-8                |
| xonsh encoding   | utf-8                |
| encoding errors  | surrogateescape      |
+------------------+----------------------+
nhoad@nhoad-laptop ~/projects/xonsh master $ $OUTPUT = $(git branch | fzf)
nhoad@nhoad-laptop ~/projects/xonsh master $ 

~/projects/xonsh master $ $OUTPUT = $(git branch | fzf)
~/projects/xonsh master $ $OUTPUT
''
~/projects/xonsh master $ $OUTPUT = $(git branch | fzf)
~/projects/xonsh master $ $OUTPUT
'* master\n'

I've straced fzf and verified that it is indeed writing '* master\n' to stdout in both situations, so I'm not sure what's going on. A race condition of some sort, maybe? I'm not sure how best to debug xonsh, so if there's anything I can do to help figure this out, let me know.

In case it's relevant, I've tried this with both readline and prompt_toolkit, same result. I've gone back a couple of releases and this has been present in all of them for me.

EDIT: If it's any consolation to people who want to be able to use something in the meantime, I've found that neither of my issues occur with https://github.com/garybernhardt/selecta or https://github.com/rschmitt/heatseeker, so I'm just going to migrate to using one of these. I'm still happy to help out with debugging the issues I've found though.

@nhoad
Copy link

@nhoad nhoad commented Jul 14, 2017

I've got two potential fixes for my issues, one of which might also fix your issues, @nh2. Would you be willing to try applying this patch, and see if it works for you?

index eef3d933..8e218a44 100644
--- a/xonsh/built_ins.py
+++ b/xonsh/built_ins.py
@@ -741,7 +741,7 @@ def _update_last_spec(last):
     elif ON_WINDOWS and not callable_alias:
         last.universal_newlines = True
         last.stderr = None  # must truly stream on windows
-    else:
+    elif captured != 'stdout':
         r, w = pty.openpty() if use_tty else os.pipe()
         _safe_pipe_properties(w, use_tty=use_tty)
         last.stderr = safe_open(w, 'w')

This changes things such that the final process in a spec won't have its stderr captured unless the capture mode indicates it should be, which should be closer in behaviour to what more traditional shells do, and matches the documentation for $() (it currently kind of matches the doc, but I won't go into it).

My other patch is a little more involved and is more of a workaround than a fix, so I'm more interested in this one first, as it takes xonsh in the right direction (IMO).

If one of the fantastic Xonsh devs could weigh in on whether or not this tiny patch is acceptable, that'd be good! I'm happy to write up some tests for it and submit a PR. I've found a good work around is to throw tee on the end, e.g. git branch | fzf | tee -, then fzf's stderr won't be captured and things will work fine (but of course, fixing the code would be preferable!).

@nh2
Copy link
Contributor Author

@nh2 nh2 commented Jul 14, 2017

@nathan-hoad I tried your patch, it does seem to bring an improvement but also breaks something else:

Before your patch:

$ @("ls ".strip())
CHANGELOG.rst     codecov.yml            requirements-docs.txt   xonsh.egg-info
CONTRIBUTING.rst  conftest.py            requirements-tests.txt  xonsh.sublime-project

after your patch:

$ @("ls ".strip())
CHANGELOG.rst     codecov.yml            requirements-docs.txt   xonsh.egg-info
                                                                               CONTRIBUTING.rst  conftest.py            requirements-tests.txt  xonsh.sublime-project

@nhoad
Copy link

@nhoad nhoad commented Jul 14, 2017

Huh, interesting! I don't experience that behaviour. Is ls an alias that adds colour options or anything? If you try running xonsh with --no-rc, does it still happen?

What OS + terminal are you using? If I can try to match your setup, I might be able to replicate it.

@nh2
Copy link
Contributor Author

@nh2 nh2 commented Jul 18, 2017

@nathan-hoad

Is ls an alias that adds colour options or anything?

Yes, it seems so, if I do which ls it just prints ls --color=auto -v. It also does that when I run it from xonsh --no-rc.

What OS + terminal are you using? If I can try to match your setup, I might be able to replicate it.

Ubuntu 16.04 and terminator.

But I think it was just that the state of that terminal was broken. Retrying it now and it works. In urxvt it also works.

So I need to revise my answer, your patch seems to have have the desired result for me.

@scopatz
Copy link
Member

@scopatz scopatz commented Jul 22, 2017

Any chance we could get a pull request, please? Thanks for bringing this up!

@anki-code
Copy link
Member

@anki-code anki-code commented Apr 4, 2021

Yeah, this is another issue around threading/unthreading and capturing/uncapturing. I've taken a look into this. Some comments around my tests on Arch Linux for the future researchers who want to understand.

1. About $()

The command:

showcmd $(ls /tmp | fzf)
# ['Temp\n']

This command calls subproc_captured_stdout with captured="stdout":

xonsh/xonsh/built_ins.py

Lines 165 to 169 in 5a792a6

def subproc_captured_stdout(*cmds, envs=None):
"""Runs a subprocess, capturing the output. Returns the stdout
that was produced as a str.
"""
return xonsh.procs.specs.run_subproc(cmds, captured="stdout", envs=envs)

On the first look it works. But has two issues:

  1. The $() output will return the pure output of fzf with trailing new lines and will not work for the elegant substitution use case (hello #3924):

    showcmd $(ls /tmp | fzf)
    # ['Temp\n']
  2. The output has weird behavior for fzf. The output is empty in random cases (the speed of pressing Enter has no effect):

    for i in range(5):
        showcmd $(ls /tmp | fzf)
    # ['Temp\n']
    # ['']
    # ['']
    # ['Temp\n']
    # ['Temp\n']

2. About @$()

The command:

showcmd @$(ls /tmp | fzf)

This command calls subproc_captured_inject with captured="object":

xonsh/xonsh/built_ins.py

Lines 172 to 179 in 5a792a6

def subproc_captured_inject(*cmds, envs=None):
"""Runs a subprocess, capturing the output. Returns a list of
whitespace-separated strings of the stdout that was produced.
The string is split using xonsh's lexer, rather than Python's str.split()
or shlex.split().
"""
o = xonsh.procs.specs.run_subproc(cmds, captured="object", envs=envs)
o.end()

And this is the cause why it works different and it's not working in fact. It just freezing the terminal because it's blocked by fzf that waiting the input from the user and raises the RecursionError in the background:

echo @$(ls /tmp | fzf)
# ^C
#Traceback (most recent call last):
#  File "/opt/miniconda3/lib/python3.8/site-packages/xonsh/base_shell.py", line 374, in default
#    run_compiled_code(code, self.ctx, None, "single")
#  File "/opt/miniconda3/lib/python3.8/site-packages/xonsh/codecache.py", line 67, in run_compiled_code
#    func(code, glb, loc)
#  File "<xonsh-code>", line 1, in <module>
#  File "/opt/miniconda3/lib/python3.8/site-packages/xonsh/built_ins.py", line 179, in subproc_captured_inject
#    o.end()
#  File "/opt/miniconda3/lib/python3.8/site-packages/xonsh/procs/pipelines.py", line 452, in end
#    self._end(tee_output=tee_output)
#  File "/opt/miniconda3/lib/python3.8/site-packages/xonsh/procs/pipelines.py", line 460, in _end
#    for _ in self.tee_stdout():
#  File "/opt/miniconda3/lib/python3.8/site-packages/xonsh/procs/pipelines.py", line 362, in tee_stdout
#    for line in self.iterraw():
#  File "/opt/miniconda3/lib/python3.8/site-packages/xonsh/procs/pipelines.py", line 326, in iterraw
#    proc.wait()
#  File "/opt/miniconda3/lib/python3.8/site-packages/xonsh/procs/posix.py", line 434, in wait
#    rtn = self.proc.wait(timeout=timeout)
#  File "/opt/miniconda3/lib/python3.8/subprocess.py", line 1079, in wait
#    return self._wait(timeout=timeout)
#  File "/opt/miniconda3/lib/python3.8/subprocess.py", line 1804, in _wait
#    (pid, sts) = self._try_wait(0)
#  File "/opt/miniconda3/lib/python3.8/subprocess.py", line 1762, in _try_wait
#    (pid, sts) = os.waitpid(self.pid, wait_flags)
#  File "/opt/miniconda3/lib/python3.8/site-packages/xonsh/procs/posix.py", line 308, in _signal_int
#    signal.pthread_kill(threading.get_ident(), signal.SIGINT)
#  File "/opt/miniconda3/lib/python3.8/site-packages/xonsh/procs/posix.py", line 308, in _signal_int
#    signal.pthread_kill(threading.get_ident(), signal.SIGINT)
#  File "/opt/miniconda3/lib/python3.8/site-packages/xonsh/procs/posix.py", line 308, in _signal_int
#    signal.pthread_kill(threading.get_ident(), signal.SIGINT)
#  [Previous line repeated 487 more times]
#  File "/opt/miniconda3/lib/python3.8/site-packages/xonsh/procs/posix.py", line 304, in _signal_int
#    self.send_signal(signal.CTRL_C_EVENT if xp.ON_WINDOWS else signum)
#  File "/opt/miniconda3/lib/python3.8/site-packages/xonsh/procs/posix.py", line 482, in send_signal
#    rtn = self.proc.send_signal(signal)
#RecursionError: maximum recursion depth exceeded

The manipulations with threading has no effect:

__xonsh__.commands_cache.threadable_predictors['fzf'] = lambda *a, **kw: False
showcmd @$(ls /tmp | fzf)
# no output

3. Workaround

After experiments the most stable workaround is to pipe the fzf output into cat (hello #4214 (comment)):

for i in range(5):
    showcmd $(ls /tmp | fzf | cat)
    
# ['Temp\n']
# ['Temp\n']
# ['Temp\n']
# ['Temp\n']
# ['Temp\n']

And put it to Python substitution to get clean lines/arguments (hello #3924):

showcmd @($(ls /tmp | fzf | cat).splitlines())
# ['Temp']

4. Finally

I believe that this logic could be improved to support alternate mode for fzf and similar tools.

To start improving you can understand how xonsh executes the command:

  1. It converts syntax @$() to built_ins.subproc_captured_inject
  2. It calls specs.run_subproc
  3. It creates specs and put it to CommandPipeline
  4. The CommandPipeline do spec.run
  5. That calls self.cls (i.e. subprocess.Popen) or self._run_binary and return it to CommandPipeline
  6. Then CommandPipeline do logic around reading stdout/stderr and communicate with terminal. And this process has many logic around threaded/unthreaded and captured/uncaptured.

5. Quick test

I've found that after removing this line:

and self.captured != "object"

The command showcmd @$(ls /tmp | fzf) will work as expected but with unstable output returning like in point 1 above.

The truth is out there :) We want to believe :)

@jnoortheen
Copy link
Member

@jnoortheen jnoortheen commented Apr 5, 2021

the patch said above works by applying to the file xonsh/procs/specs.py:770 for me. @anki-code could you try with that?

@anki-code
Copy link
Member

@anki-code anki-code commented Apr 5, 2021

This is not working for me:

bash
cd /tmp
git clone https://github.com/xonsh/xonsh/
cd xonsh && git branch 5a792a6  # current master
vim xonsh/procs/specs.py
awk 'NR==769,NR==770' xonsh/procs/specs.py
#    #else:
#    elif captured != 'stdout':
pip install .
xonsh --no-rc
echo @$(ls /tmp | fzf)
# freezed

@anki-code
Copy link
Member

@anki-code anki-code commented May 2, 2021

Link to #4214

@jnoortheen
Copy link
Member

@jnoortheen jnoortheen commented May 2, 2021

Link to #4214

can we try using async-subprocess to get rid of threading around subprocess and remove the complexities?

@anki-code
Copy link
Member

@anki-code anki-code commented May 2, 2021

Welcome to 2159

@yaxollum
Copy link
Contributor

@yaxollum yaxollum commented Aug 14, 2021

@anki-code The original issue (using OUTPUT = $(fzf) to capture the result shown by fzf) has now been fixed by PR #4336, but it seems like you have brought up some other issues. Should this issue be closed?

@anki-code
Copy link
Member

@anki-code anki-code commented Aug 16, 2021

Yep, it works now. Thanks!

@anki-code anki-code closed this Aug 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants