Skip to content

Cannot read from STDIN with read_to_string if reading before piping Rust program writes to STDOUT in another Rust program #141714

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kiseitai3 opened this issue May 29, 2025 · 6 comments · May be fixed by #142102
Labels
A-docs Area: Documentation for any part of the project, including the compiler, standard library, and tools T-libs Relevant to the library team, which will review and decide on the PR/issue.

Comments

@kiseitai3
Copy link

Using std::io::read_to_string, stdin().read_to_string, or stdin().read_to_end() blocks on nonexisting buffer and ignores incoming buffer. This happens if I pipe a program compiled with Rust to itself. For example, ./target/debug/rumtk-v2-interface --port 55556 --local | ./target/debug/rumtk-v2-interface --daemon --outbound --port 55555 --local. Reading using the buffer methods work well assuming I am able to piece together the buffer before converting to UTF-8 String. Flushing STDOUT from the piping program does not force the STDIN read to succeed.

I tried this code. Only read_some_stdin works:

pub fn read_stdin_() -> RUMResult<RUMString> {
        let mut stdin_handle = stdin().lock();
        let has_data = match stdin_handle.has_data_left() {
            Ok(flag) => flag,
            Err(e) => return Err(format_compact!("No data in STDIN: {}", e)),
        };
        println!("{}", &has_data);
        if has_data {
            match std::io::read_to_string(stdin_handle) {
                Ok(s) => {
                    println!("{}", &s);
                    Ok(s.into())
                }
                Err(e) => Err(format_compact!("Error reading from STDIN: {}", e)),
            }
        } else {
            Ok(RUMString::default())
        }
    }

    pub fn read_stdin() -> RUMResult<RUMString> {
        let mut buf: Vec<u8> = Vec::new();
        match stdin().read_to_end(&mut buf) {
            Ok(s) => {
                println!("{}", &s);
                Ok(buf.to_rumstring())
            }
            Err(e) => Err(format_compact!("Error reading from STDIN: {}", e)),
        }
    }

    pub fn read_some_stdin(input: &mut StdinLock) -> RUMResult<RUMString> {
        let mut buf: [u8; 1024] = [0; 1024];
        match input.read_exact(&mut buf) {
            Ok(_) => Ok(buf.as_slice().to_rumstring()),
            Err(e) => Err(format_compact!(
                "Error reading 1024 bytes from STDIN: {}",
                e
            )),
        }
    }

I expected to see this happen: Have the stdout buffer pass to the stdin buffer of a Rust program piped to itself via the terminal. I expected to receive the buffer when reading or at least get an error I could handle/ignore if a read is empty.

Instead, this happened: Nothing happens even though program can read from stdin and write to stdout in isolated tests.

This is similar to forum? issue: [How to handle Stdin hanging when there is no input](https://users.rust-lang.org/t/how-to-handle-stdin-hanging-when-there-is-no-input/93512).

I am currently not sure why the issue happened after looking at the std library's STDIN source code, but I am not experienced in the codebase so it could be a me issue.

Image

Thanks for your work on Rust and thank you for your help!

Meta

rustc --version --verbose:

rustc 1.86.0-nightly (4a4309466 2025-02-02)
binary: rustc
commit-hash: 4a43094662727040d8522163f295b19d1aed0e49
commit-date: 2025-02-02
host: x86_64-unknown-linux-gnu
release: 1.86.0-nightly
LLVM version: 19.1.7

Backtrace

<backtrace>

@kiseitai3 kiseitai3 added the C-bug Category: This is a bug. label May 29, 2025
@rustbot rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label May 29, 2025
@RalfJung
Copy link
Member

RalfJung commented May 29, 2025

Does the sending part of the pipe ever close their end of it (e.g. by exiting)? If not, this is entirely expected behavior: read_to_end reads until the end, as the name says -- and as long as the sending program keeps the pipe open, the end has not been reached yet, more data could come in any time.

Can you share the code for the sending part?

Generally, read_to_end and read_to_string should only be used for IO streams that have a clear end, like files. They make no sense on an IO stream that is expected to stay around and keep streaming data.

@kiseitai3
Copy link
Author

Hmm. I agree with your observation.

I had a similar idea last night while writing the bug report. To test that hypothesis, I tried adding a break. Same issue as reported. Tonight, I introduced a panic! on the sender to confirm that nothing gets picked up on the receiving end despite the pipe technically closing.

Image

Hello, the blank line, and the panic printout comes from the receiver. The JSON bit is actually printed from the receiving end. Nothing was sent to the other endpoint I was using in the test, but that might be a separate issue. The null hypothesis seems to be correct here. The pipe closure does allow the receiver to capture the input.

I believe you are right.

I did check the repository and confirmed that trait Read is fully defined for Stdin and StdinLock.
I think that the docs should warn against usage of these methods with stdin if the sender is not expected to exit or close the stream.
I clearly misunderstood what was meant by reading to end. I assumed it meant reading to end of message and overlooked the fact that std{in,out,err} are present for the lifetime of the program and have no concept of message. I need to read using the buffer methods until the first empty read.

Image

Image

Depending on how you think it is best to proceed (and if you do not mind), this could be a great opportunity for me to open a PR and contribute.

Thank you!

@RalfJung
Copy link
Member

Yeah, pointing this out in the docs seems like a good idea. I won't have the time to oversee a doc fix here, but if you just make a PR, a suitable reviewer will be assigned to that. :)

@hkBst
Copy link
Member

hkBst commented Jun 1, 2025

@rustbot label A-docs

@rustbot rustbot added the A-docs Area: Documentation for any part of the project, including the compiler, standard library, and tools label Jun 1, 2025
@kiseitai3
Copy link
Author

Added a note to the two methods making clear that EOF is never reached/ expected to be reached if reading from an stdin pipe that does not get closed.

@kiseitai3
Copy link
Author

Just to add more documentation to the record for future others that may fail to read the documentation, Once data is picked up from stdin, it looks like we never get back to 0 bytes.

In this test, I broke reads into 512 bytes max using ...

pub fn read_stdin() -> RUMResult<RUMString> {
        let mut stdin_lock = stdin().lock();
        let mut stdin_buffer: Vec<u8> = Vec::with_capacity(BUFFER_SIZE);
        let (mut size, mut buf) = read_some_stdin(&mut stdin_lock)?;
        println!("Reading STDIN!");
        while size > 0 {
            for itm in buf.iter() {
                stdin_buffer.push(*itm);
            }
            let result = read_some_stdin(&mut stdin_lock)?;
            size = result.0;
            buf = result.1;
            println!("Read {} bytes", &size);
        }
        print!("{}", &stdin_buffer.to_rumstring());
        Ok(stdin_buffer.to_rumstring())
    }

Image

It seems that the minimum resultant buffer is all zeroes. I am not sure if this is a side effect from the kernel or writing to stdout in Rust.

Image

This Go Reddit confirms our suspicions about reading and writing to std. Someone does recommend forcing a flush of buffers on writer side but my writer function already flushes after every write and the receiving end never triggers.

At any rate, I could not figure out the reason for the returned 61 bytes upon completing the buffer reads from the code review alone. I will see if I return to this in the future to have a more complete account of events, but for now, I can just check if the first byte of a returned buffer is zero. That should be good enough for my purpose.

Hopefully, this report helps others in the future.

@jieyouxu jieyouxu added T-libs Relevant to the library team, which will review and decide on the PR/issue. and removed C-bug Category: This is a bug. needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. labels Jun 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-docs Area: Documentation for any part of the project, including the compiler, standard library, and tools T-libs Relevant to the library team, which will review and decide on the PR/issue.
Projects
None yet
5 participants