kevent signals data available on a socket but ioctl(...,FIONREAD,...) returns 0

[stephentoub]

There appears to be a race condition in the implementation of kqueues on macOS, such that kevent can wake up for a read on a socket (meaning at least 1 byte of data is available) but an ioctl(..., FIONREAD, ...) call immediately after that completion can still return 0. This can manifest as code like:

await socket.ReceiveAsync(Array.Empty<byte>(), 0, 0, SocketFlags.None);
int count = socket.Available;

getting a count of 0 instead of something larger.

[karelz]

@stephentoub how often it happens? Mac is not production environment, could we move it to Future?
cc @geoffkizer

[stephentoub]

how often it happens?

Very frequently, at least on 10.11; I've not tried on 10.12.

[karelz]

OK, assigning to @geoffkizer for further look.

[geoffkizer]

Is this actually a bug? It seems like users just need to be aware that this happens and deal with it.

[stephentoub]

Is this actually a bug?

Here's the situation that caused this to be opened. You've got a high-scale server receiving, let's say, on 50,000 sockets. You don't want to have to create 50,000 buffers, so instead you issue a 0-length ReceiveAsync on each, so that you can be notified when there is data available to be read, and only then get a buffer to use. To know the size buffer to use, you check DataAvailable, but on macOS it sometimes returns 0 even though you were just notified that there's data available.

Certainly seems like a bug, but if nothing else, it's a noticeable difference in behavior between Windows/Linux and macOS.

I'm not exactly sure how you'd deal with it in a situation like that above. Spin until DataAvailable became positive? Spin in an asynchronous loop issuing ReadAsync/DataAvailable calls until it became positive? Something else?

[geoffkizer]

Spin in an asynchronous loop issuing ReadAsync/DataAvailable calls until it became positive?

Yes, this seems like the right approach. Most likely you've got an async loop that's consuming incoming data. If you get DataAvailable = 0, just continue to the next iteration of the loop.

I do agree this is annoying, but I'm not sure how we would fix it without introducing potential perf problems. Thoughts?

[stephentoub]

If you get DataAvailable = 0, just continue to the next iteration of the loop.

How do you differentiate "ReadAsync completed because there's data available" and "ReadAsync completed because the server is done sending data"?

[geoffkizer]

That's a very good question. Seems like we need to do something here, then.

[geoffkizer]

Can you reproduce this easily?

[stephentoub]

I've not tried since Nov. At the time I was able to repro it almost on demand, just running a little repro in a loop.

[stephentoub]

Can you reproduce this easily?

Yes. I made the test run 1000 times and submitted it to CI:
stephentoub/corefx@dfba21c
https://ci.dot.net/job/dotnet_corefx/job/master/job/osx_debug_prtest/4001/
It failed basically every iteration.

[geoffkizer]

Do we know if freebsd/MacOS actually supports this? I'm just speculating, but it seems possible that freebsd just happily succeeds any zero-byte read, even when no data is available.

[stephentoub]

That may very well be the problem. In which case that's the underlying issue for which it'd be great to find a solution :)

[geoffkizer]

It would be easy to determine if that is the underlying issue -- the assert about IsCompleted would fire. Unfortunately the previous test run results no longer seem to be available.

[geoffkizer]

Per @Priya91, this is in fact failing on the IsCompleted assert.

Need to think about how to fix this.

[geoffkizer]

I think the right way to fix this is to special-case zero-length receive and check DataAvailable instead. If DataAvailable is 0, we need to enqueue the op as usual, and then check DataAvailable again.

There are some cases where we might be able to know in advance that data is available; e.g. we've received an epoll notification and haven't done a receive in the mean time. It's not clear to me if trying to do this optimization is worth it.