10.0 dev reload with inotify #31855

Icallhimtest · 2019-03-14T17:38:05Z

Add the alternative of inotify instead of watchdog to watch the addons path the server was started with.

Reason: watchdog spawns 2 threads per path to watch. When there are a lot of addons paths, this can become too costly. With inotify we watch all the repositories in a single thread.

https://github.com/dsoprea/PyInotify

installation:
pip install inotify

This PR is a bit more dirty than what's suggested in https://github.com/odoo-dev/odoo/tree/10.0-dev-reload-dve but a lot more efficient.

Python2 uses IOError instead. /!\ DO NOT FORWARD PORT AFTER 11.* /!\

Before this commit every worker would start his own FSWatcher.

KangOl · 2019-03-14T17:47:40Z

odoo/service/server.py

-    if 'reload' in config['dev_mode']:
-        if watchdog:
+    if 'reload' in config['dev_mode'] and not odoo.evented:
+        if inotify and not odoo.evented:


check on odoo.evented is useless (already checked the line before)

whoops, indeed

KangOl · 2019-03-14T17:49:53Z

odoo/service/server.py

+            if not self.watcher:
+                self.watcher = InotifyTree(path, mask=INOTIFY_LISTEN_EVENTS, block_duration_s=1.0)
+            else:
+                self.watcher._InotifyTree__load_tree(path)


Ho yes, it's dirty

Yeah, But this avoids having to iterate through different InotifyTree and have all the events on a single fd, which, on my 500 addons-paths test prevents a constant 2% CPU usage to iterate on every path every second.

KangOl · 2019-03-14T17:53:15Z

odoo/service/server.py

+        while self.started:
+            for event in self.watcher.event_gen(timeout_s=0, yield_nones=False):
+                (_, type_names, path, filename) = event
+                if filename.endswith('.py') and not filename.startswith('.~') and 'IN_ISDIR' not in type_names:


Can you extract this block inside a common function with FSWatcher using a common ancestor?

KangOl · 2019-03-14T17:54:35Z

odoo/service/server.py

+            if os.name == 'posix':
+                module = 'inotify'
+            else:
+                module = 'Watchdog'


KangOl · 2019-03-14T18:00:34Z

odoo/service/server.py

            watcher = FSWatcher()
            watcher.start()
        else:
-            _logger.warning("'watchdog' module not installed. Code autoreload feature is disabled")
+            if os.name == 'posix':


there is not inotify on osx. watchdog is still needed

Icallhimtest · 2019-03-15T11:47:30Z

@KangOl @antonylesuisse
The FSWatcherInotify runs fine for the expected scenarios, but a few questions remain:

Inotify behaves a bit differently when moving folders containing files than watchdog. Watchdog will give the notifications for each file in the folder, while inotify only gives the folder move event (I believe because the files get created before inotify gets a chance to add the folder to its watchlist, not sure how watchdog handles this). The question becomes:

a) Do we automatically reload upon directory moves?
b) Do we os.walk the directory in search of python files and reload only if it contains one & it compiles?
c) Do we ignore them? <-- current implementation

Current watchdog implementation ignores when we delete or move a file away from the folder, even a python file.

Is this intended behaviour? Current inotify implementation copies this. It doesn't listen to IN_DELETE, IN_DELETE_SELF, IN_MOVE_SELF, IN_MOVED_FROM events. ( http://man7.org/linux/man-pages/man7/inotify.7.html )

Is it ok to use the __load_tree method from the InotifyTree object? Its signature hasn't changed since its addition 4 years ago. https://github.com/dsoprea/PyInotify/blame/master/inotify/adapters.py#L342

KangOl · 2019-03-15T12:48:38Z

On 15 Mar 2019, at 12:47, Denis Vermylen wrote: @KangOl @antonylesuisse The `FSWatcherInotify` runs fine for the expected scenarios, but a few questions remain: 1) Inotify behaves a bit differently when moving folders containing files than watchdog. Watchdog will give the notifications for each file in the folder, while inotify only gives the folder move event (I believe because the files get created before inotify gets a chance to add the folder to its watchlist, not sure how watchdog handles this). The question becomes: a) Do we automatically reload upon directory moves? b) Do we os.walk the directory in search of python files and reload only if it contains one & it compiles? c) Do we ignore them? <-- current implementation It's better to have the same behaviour with both packages. I would go with b) then. 2) Current watchdog implementation ignores when we delete or move a file away from the folder, even a python file. Is this intended behaviour? Current inotify implementation copies this. It doesn't listen to `IN_DELETE, IN_DELETE_SELF, IN_MOVE_SELF, IN_MOVED_FROM` events. ( http://man7.org/linux/man-pages/man7/inotify.7.html ) You choose. You can keep current status or implement it. 3) Is it ok to use the `__load_tree` method from the `InotifyTree` object? Its signature hasn't changed since its addition 4 years ago. https://github.com/dsoprea/PyInotify/blame/master/inotify/adapters.py#L342 There is actually InotifyTrees (with a s) that take a list of paths.

Icallhimtest · 2019-03-15T13:14:09Z

@KangOl whoops, good thing you saw the InotifyTrees class, I completely missed it. It'll be nice & clean this way.

For point 1, looking at watchdog, they generate all the events manually when a directory is created:

https://github.com/gorakhargosh/watchdog/blob/master/src/watchdog/observers/inotify.py#L156
https://github.com/gorakhargosh/watchdog/blob/master/src/watchdog/events.py#L612

I'm hesitating between submitting a PR to inotify adding this feature somewhere here https://github.com/dsoprea/PyInotify/blame/master/inotify/adapters.py#L287
or simply doing it in Odoo.

KangOl · 2019-03-15T13:26:45Z

From the reactivity of maintainer of pyinotify, it's better to handle it inside odoo, even if a PR is made upstream.

Add the alternative of inotify instead of watchdog to watch the addons paths the server was started with. Reason: watchdog spawns 2 threads per path to watch. When there are a lot of addons paths, this can become too costly. With inotify we watch all the repositories in a single thread. https://github.com/dsoprea/PyInotify installation: pip install inotify

Avoid potential tracebacks from the FSWatcher's thread being killed. Drawback: Server shut down can have an extra small delay (only applies when the --dev=reload option is given)

Icallhimtest · 2019-03-15T14:21:46Z

@KangOl alrighty,

1b) applied & tested on my end
2) kept same as watchdog
3) done

Should be good to go.

antonylesuisse · 2019-03-18T10:42:20Z

Make sure it also works for editor that write in place.

Icallhimtest · 2019-03-18T18:45:21Z

@antonylesuisse Yeah, it does.

It listens to IN_WRITE or IN_MOVED_TO events. (+ IN_CREATE)

Both swapping or writing in place are covered by this.

Icallhimtest · 2019-03-20T16:05:16Z

@KangOl Want any more changes or are we good to go here?

KangOl

@robodoo rebase-ff r+

robodoo · 2019-03-20T16:28:23Z

Merge method set to rebase and fast-forward

Avoid potential tracebacks from the FSWatcher's thread being killed. Drawback: Server shut down can have an extra small delay (only applies when the --dev=reload option is given) closes #31855 Signed-off-by: Christophe Simonis <chs@odoo.com>

robodoo · 2019-03-20T17:11:51Z

Merged, thanks!

Avoid potential tracebacks from the FSWatcher's thread being killed. Drawback: Server shut down can have an extra small delay (only applies when the --dev=reload option is given) closes odoo#31855 Signed-off-by: Christophe Simonis <chs@odoo.com>

Icallhimtest added 2 commits March 14, 2019 11:06

[FIX] server.py: FileNotFoundError doesn't exist in P2

c5381c2

Python2 uses IOError instead. /!\ DO NOT FORWARD PORT AFTER 11.* /!\

[FIX] server.py: don't start multiple FSWatcher in multiworker mode

5e1a53f

Before this commit every worker would start his own FSWatcher.

Icallhimtest changed the title ~~10.0 dev reload dve~~ 10.0 dev reload with inotify Mar 14, 2019

robodoo added the seen 🙂 label Mar 14, 2019

Icallhimtest requested a review from KangOl March 14, 2019 17:38

C3POdoo added the RD research & development, internal work label Mar 14, 2019

robodoo added the CI 🤖 Robodoo has seen passing statuses label Mar 14, 2019

KangOl requested changes Mar 14, 2019

View reviewed changes

Icallhimtest force-pushed the 10.0-dev-reload-dirty-dve branch from 18b4645 to 4fc2b60 Compare March 15, 2019 11:46

robodoo removed the CI 🤖 Robodoo has seen passing statuses label Mar 15, 2019

robodoo added the CI 🤖 Robodoo has seen passing statuses label Mar 15, 2019

Icallhimtest force-pushed the 10.0-dev-reload-dirty-dve branch from 4fc2b60 to 419538d Compare March 15, 2019 13:16

robodoo removed the CI 🤖 Robodoo has seen passing statuses label Mar 15, 2019

Icallhimtest force-pushed the 10.0-dev-reload-dirty-dve branch 2 times, most recently from 24940af to ecbb498 Compare March 15, 2019 13:47

robodoo added the CI 🤖 Robodoo has seen passing statuses label Mar 15, 2019

Icallhimtest added 2 commits March 15, 2019 15:18

[FIX] server.py: stop the FSWatcher when stopping the server

0085192

Avoid potential tracebacks from the FSWatcher's thread being killed. Drawback: Server shut down can have an extra small delay (only applies when the --dev=reload option is given)

Icallhimtest force-pushed the 10.0-dev-reload-dirty-dve branch from ecbb498 to 0085192 Compare March 15, 2019 14:19

robodoo removed the CI 🤖 Robodoo has seen passing statuses label Mar 15, 2019

KangOl approved these changes Mar 20, 2019

View reviewed changes

robodoo added the r+ 👌 label Mar 20, 2019

robodoo added CI 🤖 Robodoo has seen passing statuses merging 👷 and removed merging 👷 labels Mar 20, 2019

robodoo closed this Mar 20, 2019

odony deleted the 10.0-dev-reload-dirty-dve branch March 25, 2019 12:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

10.0 dev reload with inotify #31855

10.0 dev reload with inotify #31855

Icallhimtest commented Mar 14, 2019

KangOl Mar 14, 2019

Icallhimtest Mar 14, 2019

KangOl Mar 14, 2019

Icallhimtest Mar 14, 2019

KangOl Mar 14, 2019

Icallhimtest Mar 14, 2019

KangOl Mar 14, 2019

KangOl Mar 14, 2019

Icallhimtest commented Mar 15, 2019

KangOl commented Mar 15, 2019 via email

Icallhimtest commented Mar 15, 2019

KangOl commented Mar 15, 2019

Icallhimtest commented Mar 15, 2019

antonylesuisse commented Mar 18, 2019

Icallhimtest commented Mar 18, 2019

Icallhimtest commented Mar 20, 2019

KangOl left a comment

robodoo commented Mar 20, 2019

robodoo commented Mar 20, 2019

10.0 dev reload with inotify #31855

10.0 dev reload with inotify #31855

Conversation

Icallhimtest commented Mar 14, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Icallhimtest commented Mar 15, 2019

KangOl commented Mar 15, 2019 via email

Icallhimtest commented Mar 15, 2019

KangOl commented Mar 15, 2019

Icallhimtest commented Mar 15, 2019

antonylesuisse commented Mar 18, 2019

Icallhimtest commented Mar 18, 2019

Icallhimtest commented Mar 20, 2019

KangOl left a comment

Choose a reason for hiding this comment

robodoo commented Mar 20, 2019

robodoo commented Mar 20, 2019