Integration testing: Interactive programms

[bewee]

Describe the bug

Spawning a process for an interactive main programm during integration testing and then communicating with it (e.g. via stdin/stdout) leads to falsely uncovered lines.

Same happens when similarly testing routes of a web server program (my original problem). However, there I even experience tarpaulin crashing ("Failed to get test coverage! Error: ESRCH: No such process"). Although I can imagine this having a similar cause, I should maybe open a separate issue?

To Reproduce

src/main.rs

fn main() {
    loop {
        let mut buffer = String::new();
        io::stdin().read_line(&mut buffer).unwrap();
        print!("Received {}", buffer);
    }
}

tests/tests.rs

#[test]
fn test() {
    let mut child = Command::new("./target/debug/main")
        .stdout(Stdio::piped())
        .stdin(Stdio::piped())
        .spawn()
        .unwrap();
    let mut stdin = child.stdin.take().unwrap();
    let mut stdout = child.stdout.take().unwrap();
    let t = thread::spawn(move || {
        let mut reader = BufReader::new(&mut stdout);
        let mut buff = String::new();
        reader.read_line(&mut buff).unwrap();
        assert_eq!(buff, "Received asdf\n");
    });
    let mut writer = BufWriter::new(&mut stdin);
    writer.write_all(b"asdf\n").unwrap();
    writer.flush().unwrap();
    t.join().unwrap();
    child.kill().unwrap();
}

Running cargo tarpaulin --follow-exec

Output:

INFO cargo_tarpaulin::report: Coverage Results:
|| Tested/Total Lines:
|| src/main.rs: 0/4 +0.00%
|| tests/tests.rs: 15/15 +0.00%
|| 
78.95% coverage, 15/19 lines covered, +0% change in coverage

Distro: Ubuntu 20.04
Rustc version: 1.57.0-nightly (b69fe5726 2021-09-10)

Expected behavior

100% coverage

[bewee]

Ok, turns out that the child.kill() caused all those problems. Similar to #252 (comment), I added an exit route to my project (hidden behind a feature flag) which I now use instead.

Similarly, in the simplified example above, one could exit whenever some specific string is read on stdin.

I don't like this hack very much, but for now it works.

[xd009642]

Ah glad you found it, I figured seeing the initial email in my inbox it would be the kill. I have some idea on how to solve this, generally by catching the kill signal from the test forwarding and marking that the binary is expected to exit not-of-its-own-volition. I'll update you when I've had a chance to work on this and if I make any progress

[xd009642]

I did fix some kill based thing in 2022 iirc hopefully this is now solved 🤞 if not feel free to reopen