windows doesn't detect stdin automatically

[BurntSushi]

Namely, running rg pat < file will recursively search the current directory instead of file. A workaround is to do rg pat - < file, which will search file.

This is a consequence of fixing #19. Maybe there is a better way to detect whether stdin is a pipe automatically, but I don't know it. @vadz in particular points out that it is quite hairy:

Oh, sorry, I should have thought about this, this is actually a pretty well-known issue -- but without any good solution, unfortunately. The problem is that Windows doesn't have any concept of PTY, so when a program is running in any kind of terminal emulator, and not the standard console window, its stdin is always connected to a pipe, as far as Windows is concerned. Cygwin applications can distinguish between "real" pipes and normal input from the terminal, but I don't know of any way to do this without linking to cygwin1.dll.

I think you should still be able to detect whether stdin is a TTY when running inside the native console and you should be able to check for this (running inside console, I mean). But everything console-related in Windows is pretty hairy, just look at our own code for detecting whether we can write to a console...

[mungre]

For what it's worth, you can detect this particular case. If you run type file | rg pat then the standard input handle in rg will be a pipe handle, but if you do rg pat < file then the standard input handle will be a file handle. You can distinguish them using the result of

GetFileType(GetStdHandle(STD_INPUT_HANDLE))

One caveat: this is true of cmd.exe. I don't know that other shells implement this the same way.

[silverwind]

Also noticed that cat something | rg does not work as expected on Windows and still searches recursively. I'm on Cygwin and installed through cargo, if that matters. ag does not show this issue, so maybe some of their checks could be reimplemented.

[BurntSushi]

@silverwind Does ag have the issue described in #19?

[silverwind]

Nope, no such issue with ag. I compiled ag from source under the Cygwin. I think many of the Cygwin issues that were reported are because of the fact that people are installing native Windows binaries, which in most cases is very bad. These binaries expect Windows interfaces and behaviour. Much better to build on the system's native toolchain.

I assume by installing through cargo , everything is built from source, right?

Oh, and I did some more testing with rg. I've also tested with https://github.com/rprichard/winpty which is basically a wrapper to cmd which helps native Windows binaries to run smoothly in a Cygwin terminal. My results:

• echo x | rg x has the issue that it searches recursively
• winpty cmd /c 'echo x | rg x' also has the issue
• echo x | rg x in a native cmd has the issue
• winpty cmd and entering echo x | rg x has the issue

edit: updated, my bad. It doesn't work in any of the attempts.

[silverwind]

Oh, and because it likely matters, Rust nightly was installed in the GNU version:

rustc 1.14.0-nightly (ad19c32a5 2016-10-06)

So far, I don't see any other issues. Everything work except this stdin issue.

[BurntSushi]

I don't have time at the moment to dig into this, but I'm skeptical that there's any big difference between compiling ripgrep from source on Windows and distributing a binary for Windows. If using a Windows binary is bad, could you please explain why in detail?

If you install with cargo, then yes, it's compiled from source.

Please also consider that I don't know much about Windows. I'm making it up as I go.

[silverwind]

If using a Windows binary is bad, could you please explain why in detail?

Take my answer with a grain of salts as I'm not really involved in low-level development, but as a long-time Cygwin user, this is my understanding:

Natively build binaries (especially ones build under MSVS) target native Windows APIs, which are pretty much second-class citizens inside the Cygwin environment and prone to have a few bugs, especially when it comes to fundamental platform differences like TTY emulation.

On the other hand the POSIX APIs are much better supported and it's a goal of the project enough of POSIX so that projects written for Linux can be built without any changes.

[elahn]

On Windows 10 native terminal, this works perfectly:

/// Returns true if there is a tty on stdin.
#[cfg(windows)]
pub fn on_stdin() -> bool {
    use kernel32::{GetConsoleMode, GetStdHandle};
    use winapi::winbase::STD_INPUT_HANDLE;

    unsafe {
        let mut out = 0;
        GetConsoleMode(GetStdHandle(STD_INPUT_HANDLE), &mut out) != 0
    }
}

From what I gather, this was removed due to #19. If the Cygwin environment can't be detected and skip this, I think Cygwin users should use a different build with the current behaviour. I prefer the native terminal and pipe is important enough for me to maintain a fork until this is resolved.

Btw, ripgrep is awesome!

[BurntSushi]

Allow me to summarize where I'm at with this incredibly annoying issue:

• In Hangs when searching recursively under Cygwin #19, ripgrep would hang when running simple queries like rg foo in mintty. This is because there's (apparently) no good way to detect whether stdin in a tty or not inside of MSYS2 based terminal emulators like mintty.
• Since Hangs when searching recursively under Cygwin #19 represents, IMO, the worst failure mode possible, it was critical that we fix it. I therefore changed ripgrep to believe it always had a tty on stdin. This means that rg foo will do the right thing, but that piping, e.g., cat bar | rg foo no longer works because ripgrep doesn't detect the pipe on stdin. Instead, it searches the current directory recursively and ignores stdin completely. This is a terrible failure mode, but not quite the worst possible failure mode. Thus, we have this issue.
• Applications distributed with cygwin, like grep, appear to get this right. That is, echo foo | grep foo does the right thing in both mintty and in cmd.exe. So clearly, fixing this bug properly is possible, but the implementation escapes me.
• git faced a similar issue and uses a spectacular hack to work around it. I don't understand it.
• mintty has at least two issues on this topic: mintty/53 and mintty/482. I'm not sure that there's anything actionable there.
• Solutions like "cygwin users just need to compile from source" are insufficient, not only because I haven't actually observed that to work (I'm compiling ripgrep from source on Windows when testing all of this), but it's also terrible UX.

What are the next steps?

• The status quo is terrible because native Windows users suffer , but reverting the fix to Hangs when searching recursively under Cygwin #19 is even worse.
• The key problem is that if GetConsoleMode(stdin) returns false, then we can't trust it because it always returns false in mintty. However, if GetConsoleMode(stdout) or GetConsoleMode(stderr) return true, then we could trust a GetConsoleMode(stdin) call that returned false because we know we're in a console.

That leads to the following logic:

    let on_stdin = unsafe {
        let mut out = 0;
        GetConsoleMode(GetStdHandle(STD_INPUT_HANDLE), &mut out) != 0
    };
    if on_stdin {
        // good enough, on_stdin is only true if there's a console
        true
    } else if on_stdout() || on_stderr() {
        // on_stdin could be a false negative, but if stdout/stderr is a
        // console, then we can trust the negative result.
        false
    } else {
        // We don't know whether we have a console or not, so always pretend
        // that we have a tty on stdin.
        true
    }

I've tried this out and it does now at least let ripgrep operate sanely on native Windows. It doesn't fix the mintty problem and it can still spaz out on you. For example, running this produces unexpected results:

echo foo | rg foo > log 2>&1

In this case, GetConsoleMode(stdin), GetConsoleMode(stdout) and GetConsoleMode(stderr) all return false, and we therefore cannot distinguish between a false negative and a true negative.