Log rotation bug with polling watcher

[cheesestraws]

I am in an environment with quite rapid/intense logging, and I am using tail to watch these log files. I have been hitting a repeatable issue where after log rotation, tail simply failed to pick up the new log files, and I believe I've tracked down why.

In ChangeEvents() in watch/polling.go, changes.NotifyDeleted() is used if the file either goes away or is a different file from the open file. In both cases, changes.NotifyDeleted() is called and then the goroutine exits---so only one deletion event will ever be sent from any given polling goroutine for any given deletion/rotation.

Unfortunately, these deletion events can get lost: looking in NotifyDeleted() in watch/filechanges.go, it only sends the deletion event down the channel if the channel is empty at the time. This means if there's still a pending modification event, then the deletion event will be dropped on the floor, and because the goroutine that sent the deletion event has returned, there will not be another deletion event sent, and the file will not be reopened.

I instrumented sendOnlyIfEmpty to log whether delete requests were in fact being dropped on the floor (by checking whether ch == fc.Deleted) and they were.

Locally, I've hacked around this by making NotifyDeleted() just send down the channel whether or not there's space and block if it has to, and this seems to have worked for my use case so far.