Watcher race with directory rename on Linux

[davidmorgan]

Another one :) @mraleph

If you watch a very-recently-renamed directory on Linux, it can do two surprising things:

• report events with the path from before the rename
• close itself because it thinks it just got deleted

import 'dart:io';

void main() async {
  final directory = Directory.systemTemp.createTempSync();
  directory.watch().listen(
      (e) => print('for ${directory.path}: $e'),
      onDone: () => print('done ${directory.path}'));
  File('${directory.path}/f').createSync();
  final newName = '${directory.path}-renamed';
  directory.rename(newName);
  Directory(newName).watch().listen(
      (e) => print('for $newName: $e'),
      onDone: () => print('done ${newName}'));

  File('$newName/g').createSync();

  await Future.delayed(Duration(seconds: 1));
}

prints

for /tmp/FBUFLC: FileSystemCreateEvent('/tmp/FBUFLC/f', isDirectory=false)
for /tmp/FBUFLC-renamed: FileSystemCreateEvent('/tmp/FBUFLC/f', isDirectory=false)
for /tmp/FBUFLC: FileSystemDeleteEvent('/tmp/FBUFLC')
for /tmp/FBUFLC-renamed: FileSystemDeleteEvent('/tmp/FBUFLC')
done /tmp/FBUFLC
done /tmp/FBUFLC-renamed

• the creation of the file "f" is reported by both watchers, notice Directory('/tmp/FBUFLC-renamed').watch reporting an event from a different (named) directory.
• the delete in the old location is also reported by both watchers, and causes both watchers to close, even though /tmp/FBUFLC-renamed continues to exist and it would be nice to receive events for it :)

If you know about this then you can work around it, so I guess at least it's worth documenting.

But I wonder if the VM can do any better, e.g. perhaps it could filter out events with the wrong prefix, perhaps it could continue watching or restart watching past a delete event for the wrong prefix?

There is a related but different issue if the watched directory is a subdirectory of a renamed directory, I will open a different issue for that next.

[mraleph]

Linux inotify is inode based rather than path based. So when you watch the path you actually watch the underlying inode rather than the path itself. Rename does not change the inode - that explains why you get old events as well.

This looks like a race between draining old events and adding a new watch via inotify_add_watch: we have a bunch of events accumulated for the original watch, but we have not drained or processed them. We add a watch for the new path, but inotify_add_watch just returns the same watch id because underlying inode is the same and there is still a watch. We drain events and distribute them to both watchers.

There is no timestamps available in the inotify_event structure that would allow us to know when event was actually generated. The move generates IN_MOVE_SELF event - but it does not actually include any information about where directory moved to. So I don't think there is any reliable way to partition the stream of events.

[biggs0125]

@brianquinlan Pointed out that this same behavior can be seen on recent versions of macOS. The OS will buffer all events for a few milliseconds before notifying.

[davidmorgan]

@biggs0125 @brianquinlan Thanks, indeed we do see old events on MacOS, but it doesn't lead to this exact problem because there is recursive directory watching on MacOS, so there is no need to worry about opening/closing watchers for subdirectories.

[mraleph]

All watchers buffer - it would be prohibitively expensive to notify user-space about file system events directly without buffering.

The problem with rename is very specific to Linux though because of how inotify works