TCP_NODELAY setsockopt uses BYTE on Windows when it should use BOOL

[chrisnc]

This bug was found and mitigated by the Wine project here:
https://source.winehq.org/git/wine.git/commit/d6ea38f32dfd3edbe107a255c37e9f7f3da06ae7

The types for each setsockopt are documented here:
https://docs.microsoft.com/en-us/windows/win32/api/winsock/nf-winsock-setsockopt

As the Wine commit explains, Windows itself is more liberal in what values of optlen it will accept for boolean arguments, which is why this has not affected Windows proper, but it would still be better to pass the expected type and length, so that the Rust bindings will also work on more strict implementations of winsock, such as Wine's prior to this fix. I confirmed that this is the case using Wine 7.0, which is the current stable branch available from https://wiki.winehq.org/Debian. Wine 5.0.3, which is the version that Debian/Ubuntu distribute themselves, is not affected by this problem.

I tried this code:

use std::io::{self, Write};
use std::net::TcpStream;

fn main() -> io::Result<()> {
    let mut stream = TcpStream::connect("127.0.0.1:8080")?;
    stream.set_nodelay(true)?;
    stream.write(b"hello")?;
    Ok(())
}

I set up a listening peer with nc -l -p 8080,
built with cargo build --target x86_64-pc-windows-gnu,
and ran with wine target/x86_64-pc-windows-gnu/debug/tcp-nodelay-text.exe.

I expected to see this happen: all the socket operations succeed, and nc prints "hello" and exits.

Instead, this happened: set_nodelay returns an Err:

Error: Os { code: 10022, kind: InvalidInput, message: "OS Error 10022 (FormatMessageW() returned error 317)" }

If I run with strace -f wine target/x86_64-pc-windows-gnu/debug/tcp-nodelay-text.exe, which shows the Linux system calls that wine makes on behalf of the program, I can see:

<pid> setsockopt(9, SOL_TCP, TCP_NODELAY, "\1", 1)         = -1 EINVAL (Invalid argument)

I tested this using rust 1.58.1, but I confirmed that the code in question is still doing the same thing on the main branch of rust.

Here is a commit which should fix the issue:
chrisnc@33c1de8

I was able to test it myself and confirmed that the error goes away:

$ rustc +stage2 --target x86_64-pc-windows-gnu src/main.rs # the rustc+std that I built with this change
$ wine main.exe # works
$ rustc --target x86_64-pc-windows-gnu src/main.rs # rustc 1.58.1
$ wine main.exe # fails
Error: Os { code: 10022, kind: InvalidInput, message: "OS Error 10022 (FormatMessageW() returned error 317)" }

That this error isn't formatted correctly is probably another bug, but I haven't looked into it.

[ChrisDenton]

Thanks for reporting this! Would you mind creating a pull request with your changes? The rustc dev guide has information on the rustc PR workflow.

[chrisnc]

Sure thing: #94094