Simple pipelining

[anki-code]

Hi! Thank you for great project!

🔽 Start reading the thread from here — #3366 (comment)

🥈 Most closest solution here — #3366 (comment)

🥈 Xontrib: xontrib-pipeliner

[con-f-use]

You realise, the Python part of your code is faulty to begin with, so it can't work in xonsh, right?

Here is a correct Python version:

┌─22-23:04 [confus@profusion ~] [1]                                                😱  😱
└$ cat /tmp/test.py                                                                                          
#!/usr/bin/python
import sys
for i, s in enumerate(sys.stdin.readlines()): print("%s %s %s" % (len(s),i,s))

┌─22-23:04 [confus@profusion ~]                                                    😱  😱
└$ echo test | /tmp/test.py                                                                                  
5 0 test

That being said, there is a real problem. The tutorial clearly states, that it should be possible to pipe into python mode:

# directly taken from 
echo "hello" | @(lambda a, s=None: s.read().strip() + " world\n")

But if one runs this in a recent version of xonsh, xonsh seems to wait for input until the cows come home. Even worse, we now have a process that is not stopped by Ctrl+C and idles in one thread at 100% CPU.

[laloch]

# directly taken from 
echo "hello" | @(lambda a, s=None: s.read().strip() + " world\n")

But if one runs this in a recent version of xonsh, xonsh seems to wait for input until the cows come home. Even worse, we now have a process that is not stopped by Ctrl+C and idles in one thread at 100% CPU.

@con-f-use: Works for me just fine on current master/HEAD.

xonfig

+------------------+----------------------+
| xonsh            | 0.9.13               |
| Git SHA          | 5648b8d5             |
| Commit Date      | Oct 17 19:02:06 2019 |
| Python           | 3.7.4                |
| PLY              | 3.11                 |
| have readline    | True                 |
| prompt toolkit   | 2.0.4                |
| shell type       | prompt_toolkit2      |
| pygments         | 2.4.2                |
| on posix         | True                 |
| on linux         | True                 |
| distro           | arch                 |
| on darwin        | False                |
| on windows       | False                |
| on cygwin        | False                |
| on msys2         | False                |
| is superuser     | False                |
| default encoding | utf-8                |
| xonsh encoding   | utf-8                |
| encoding errors  | surrogateescape      |
+------------------+----------------------+

[laloch]

Could you please checkout d782074 and try again?

[laloch]

OK, thanks for testing. Could you please file a bug report against 5648b8d? Thank you!

[anki-code]

@laloch have you any comments about first message of this issue?

[laloch]

@anki-code, as already pointed out by @con-f-use, both your shell code and Python are syntactically wrong. I'm afraid that it's not possible to enumerate lines of standard input on a single line and yield a string as an output at the same time. Let's first create simple helper function to enumerate the lines and apply arbitrary modifications on them:

def filter_stdin(stdin, stdout, filter):
    n = 0
    # call `filter` function for each input line and print the result
    for l in stdin.readlines():
        n += 1
        print(filter(l.rstrip(), n), file=stdout)

Now we can use it in our shell code:

$ ls -1 | @(lambda a,i,o: filter_stdin(i, o, lambda l,n: f"{len(l)} {l} {n}")) | sort -rn | head | @(lambda a,i,o: filter_stdin(i, o, lambda l,n: f"Winner {l}")) > result.txt

[anki-code]

@laloch thank you for example!

My question is about how to reduce the overhead (exclude function and "lambda a,i,o" in your example).

Is there a way to create function @@() and write just

ls -1 | @@(for filename in stdin: print(filename+'!!!')) | head

?

If there is no way I suggest to make it because it looks very elegant and useful. In common case we just want to get stdin, split by lines and do something with lines. Let's just make it simple like it done in @$().

// Could you please hide con-f-use comments? It's confusing :) and about another issue.

[con-f-use]

It's not really another issue, because if piping doesn't work at all, nothing you want can. I'd suggest you edit your opening post to make it clearer, what you are after and that there actually is a way to do it - just not a nice way - because your suggestion is actually a good idea. Also include a "skip ahead" link to a post where the non-confusing conversation starts. Btw. @con-f-use can hide his comments himself, so you don't have to talk over his head 😜

On the subject, I agree that the lambda thing is very clunky and would like to see a better way.

Is there a way to create function @@()?

see https://github.com/xonsh/xonsh/blob/master/xonsh/parsers/base.py#L2988

[laloch]

@anki-code: if you don't want to fiddle with the parser code, you can achieve similar results using callable alias:

def _stdfilter(args, stdin, stdout):
    command_string = "lambda line,index:"
    for arg in args:
        command_string += arg + " "
    fn = eval(command_string)
    index = 0
    for line in stdin.readlines():
        index += 1
        res = fn(line.rstrip(), index)
        if res is not None:
            print(res, file=stdout, flush=True)

aliases['stdfilter'] = _stdfilter

$ ls -1 | stdfilter "str(len(line))"
$ ls -1 | stdfilter "f'{len(line)} {line} {index}' if len(line) > 15 else None"
$ ls -1 | stdfilter "f'{len(line)} {line} {index}'" | sort -rn | head | stdfilter "f'Winner {line}'" > result.txt

[anki-code]

@laloch many thanks! The more simple version works for me like a charm:

def _pl(args, stdin, stdout):
    fn = eval('lambda line:'+args[0])
    for line in stdin.readlines():
        res = fn(line.rstrip(os.linesep))
        print(res, file=stdout, flush=True)
			
aliases['pl'] = _pl
del _pl

$ ls -1 / | pl "line+'!!!'" | head -n2
bin!!!
boot!!!

[laloch]

You are welcome. This is indeed very useful.
Just a side note: The correct way of stripping the line ends would be:

import os
res = fn(line.rstrip(os.linesep))

[anki-code]

The pl alias turned out to be so convenient that I added a separate xxh-plugin-pipe-liner for xxh and using it day by day.

It will be cool if there will be an ability to create macro like ls -1 | pl(line + '!') | head with syntax highlighting inside pl(...).
JFYI @scopatz

[anki-code]

xontrib-pipeliner is here #3558