EventLoop::deregister(TcpStream) falls to 'No such file or directory (os error 2)'

[workanator]

Hello,
after a TCP socket is shutdown by remote host call to EventLoop::deregister(TcpStream) fails with error No such file or directory (os error 2). MIO v0.4.x lacks of that behavior so I was able to deregister active or shutdown sockets without getting the error.
In my project sockets can be closed by my application or by remote hosts and I need to deregister them properly. Is there a way I can test if socket is active before call to EventLoop::deregister?
I run on Mac OS X El Captain, Rust 1.6, branch master.

[workanator]

Maybe that will be helpful. I tracked down the issue to kqueue.rs Selector:deregister and compared versions of v0.4.x and v0.5.x.

// MIO v0.4.x
    pub fn deregister(&mut self, fd: RawFd) -> io::Result<()> {
        try!(self.ev_push(fd, 0, EventFilter::EVFILT_READ, EV_DELETE));
        try!(self.ev_push(fd, 0, EventFilter::EVFILT_WRITE, EV_DELETE));

        Ok(())
    }

// MIO v0.5.x
    pub fn deregister(&mut self, fd: RawFd) -> io::Result<()> {
        self.ev_push(fd, 0, EventFilter::EVFILT_READ, EV_DELETE);
        self.ev_push(fd, 0, EventFilter::EVFILT_WRITE, EV_DELETE);

        self.flush_changes()
    }

So in v0.4.x you just push events to queue and do not flush it while in v0.5.x you push events and flush immediately. Also in v0.4.x you flush events only if the queue is full.

Well I could just ignore that error but the bad thing is that after the error happen all read/write oprations stop woking. I still can connect to my application but cannot get any response because because read/write events do not work. So it seems that is the bug.

[dwrensha]

Is it possible that you are somehow calling deregister() twice on the same TcpStream? How are you deciding when to call deregister()?

[carllerche]

What @dwrensha said. That change was to fix a bug on OS X, it's possible that the fix exposed a problem with your code?

[workanator]

I did trace of my application and I'm sure I call deregister() only once.

The error I mentioned that read/write operations stop working is my bug. I registered TCP listener with PollOpt::oneshot(). Thank you for your replies they made me focus here and squish it. But the error No such file or directory (os error 2) on deregister() is still here. For testing I replaced the body of Selector::deregister() with that from v0.4.x and the error gone.

Btw the error happens even if my application initiates socket shutdown. So no matter if the socket is active or shutdown by remote host deregistering it falls into the error.

Well that is not a big deal if I can simply ignore it without any affect of the functionality but please let me know if I can provide you more info. Thanks.

[dwrensha]

Are you registering the TcpStream with PollOpt::oneshot() also? According to man kevent:

     EV_ONESHOT     Causes the event to return only the first occurrence of the filter being
                    triggered.  After the user retrieves the event from the kqueue, it is
                    deleted.

I just verified that if I register() a stream as PollOpt::oneshot(), receive an event on it, and attempt to deregister() it, then I get a "No such file or directory" error.

[workanator]

I register() TCP Listener with EventSet::all() and PollOpt::edge(). When the listener accepts a connection (TCP Stream) I register the connection with EventSet::readable() or EventSet::writable() and PollOpt::edge() | PollOpt::oneshot() and then on data read/written I reregister the connection with PollOpt::edge() | PollOpt::oneshot() again until time has come to close the connection.

[dwrensha]

Looks like the problem is that kqueue's EV_ONESHOT effectively builds-in a deregister() after an event occurs, while epoll's EPOLLONESHOT merely disables the file descriptor. I'm not sure what the best way would be to reconcile these behaviors.

Here's a simple example program that illustrates the issue:

extern crate mio;

use mio::unix::{pipe, PipeReader, PipeWriter};
use std::io::Write;

struct Handler {
    reader: PipeReader,
}

impl mio::Handler for Handler {
    type Timeout = ();
    type Message = ();
    fn ready(&mut self, event_loop: &mut mio::EventLoop<Self>, token: mio::Token, _: mio::EventSet) {
        if token == mio::Token(1) {
            println!("Readable!. Deregistering...");
            event_loop.deregister(&self.reader).expect("deregistration failed");
        }
    }
}

pub fn main() {
    let mut event_loop: mio::EventLoop<Handler> = mio::EventLoop::new().unwrap();
    let (mut reader, mut writer) = pipe().unwrap();

    event_loop.register(&reader, mio::Token(1), ::mio::EventSet::readable(),
                        mio::PollOpt::edge() | mio::PollOpt::oneshot()).unwrap();

    let mut handler = Handler {
        reader: reader,
    };

    writer.write(b"hello");
    event_loop.run_once(&mut handler, None);
    println!("done");
}

On Linux this outputs

Readable!. Deregistering...
done

On OSX it outputs

Readable!. Deregistering...
thread '<main>' panicked at 'deregistration failed: Error { repr: Os { code: 2, message: "No such file or directory" } }', ../src/libcore/result.rs:709
Process didn't exit successfully: `target/debug/server2` (exit code: 101)

[dwrensha]

I think that in mio v0.4.x, kevent() actually still does report the "No such file or directory" error, but mio silently swallows it, due to the problem addressed by 6c04408. For a bit more discussion, see #270.

[workanator]

Yes, I believe so. In v0.4.x the error is simply ignored in ev_push and I think the franework should handle and report all errors. But from my point of view, as the user of the framework, I think such behavior should be hidden by the framework and deregistering should not raise the error. So what I can suggest is...

Is KEvent allow to test if the descriptor is registered?

• YES - test before deregister.
• NO - track somehow if the descriptor was registered for oneshot and do not deregister if it was and the event it was registered for happened.

[workanator]

man kevent says

EV_DELETE
    Removes the event from the kqueue. Events which are attached  
    to file descriptors are automatically deleted on the last close of the descriptor. 

and man epoll says

Q6  Will closing a file descriptor cause it to be removed from all
           epoll sets automatically?

A6  Yes, but be aware of the following point.  A file descriptor is a
           reference to an open file description

The most important part is Events which are attached to file descriptors are automatically deleted on the last close of the descriptor and epoll acts the same on Fd close. So probably I do not have to deregister() my TCP streams and just drop them removing from the Slab.

[workanator]

Simply closing TCP streams works (who could know? :) ). Thank you for the help. Feel free to close the issue if you think nothing should be done here.

[carllerche]

Thanks for figuring this out. I'm going to keep this issue open for now. I think that what was discovered here needs to be included in the Mio documentation somehow.