New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support using inotify with --reload #1368

Closed
mark-adams opened this Issue Oct 14, 2016 · 8 comments

Comments

Projects
None yet
4 participants
@mark-adams
Contributor

mark-adams commented Oct 14, 2016

--reload is a very useful option when debugging and working on an application. However, in some situations (such as running gunicorn inside of a VM or a Docker container on Mac with host mounted directories), the current file polling method can cause high CPU load due to the overhead of dealing with a virtualized filesystem.

Many of these filesystems now have support for the kernel's inotify API which allows applications to monitor for filesytem events instead of polling. This is much less resource intensive and can improve CPU usage of the Reloader in many situations. It would be nice to either have gunicorn support inotify by default (on supporting platforms) or via an extra argument.

mark-adams added a commit to mark-adams/gunicorn that referenced this issue Oct 14, 2016

Fix #1368 by adding 'use-inotify' configuration option
Currently, '--reload' uses FS polling to find out when files have
changed. For some time, the Linux kernel has had a feature called
inotify that allows applications to monitor for FS events without
polling.

This commit adds a new 'use-inotify' configuration option that will
cause gunicorn to use the new 'InotifyReloader' instead of the default
'Reloader' when 'reload' is enabled.

Using inotify can result in lower CPU consumption by gunicorn especially
when working with virtualized filesystems or environments with a large
number of watched files / directories.

mark-adams added a commit to mark-adams/gunicorn that referenced this issue Oct 14, 2016

Fix #1368 by adding 'use-inotify' configuration option
Currently, '--reload' uses FS polling to find out when files have
changed. For some time, the Linux kernel has had a feature called
inotify that allows applications to monitor for FS events without
polling.

This commit adds a new 'use-inotify' configuration option that will
cause gunicorn to use the new 'InotifyReloader' instead of the default
'Reloader' when 'reload' is enabled.

Using inotify can result in lower CPU consumption by gunicorn especially
when working with virtualized filesystems or environments with a large
number of watched files / directories.

mark-adams added a commit to mark-adams/gunicorn that referenced this issue Oct 14, 2016

Fix #1368 by adding 'use-inotify' configuration option
Currently, '--reload' uses FS polling to find out when files have
changed. For some time, the Linux kernel has had a feature called
inotify that allows applications to monitor for FS events without
polling.

This commit adds a new 'use-inotify' configuration option that will
cause gunicorn to use the new 'InotifyReloader' instead of the default
'Reloader' when 'reload' is enabled.

Using inotify can result in lower CPU consumption by gunicorn especially
when working with virtualized filesystems or environments with a large
number of watched files / directories.

mark-adams added a commit to mark-adams/gunicorn that referenced this issue Oct 14, 2016

Fix #1368 by adding InotifyReloader and 'use-inotify' configuration o…
…ption

Currently, '--reload' uses FS polling to find out when files have
changed. For some time, the Linux kernel has had a feature called
inotify that allows applications to monitor for FS events without
polling.

This commit adds a new 'use-inotify' configuration option that will
cause gunicorn to use the new 'InotifyReloader' instead of the default
'Reloader' when 'reload' is enabled.

Using inotify can result in lower CPU consumption by gunicorn especially
when working with virtualized filesystems or environments with a large
number of watched files / directories.
@benoitc

This comment has been minimized.

Show comment
Hide comment
@benoitc

benoitc Oct 16, 2016

Owner

good idea :)

Can you send us a PR? Looking at your patch I would prefer to set it by augmenting the current setting: --reload=inotify . So we don't introduce another setting.

Owner

benoitc commented Oct 16, 2016

good idea :)

Can you send us a PR? Looking at your patch I would prefer to set it by augmenting the current setting: --reload=inotify . So we don't introduce another setting.

@benoitc benoitc added the Improvement label Oct 16, 2016

mark-adams added a commit to mark-adams/gunicorn that referenced this issue Oct 18, 2016

Fix #1368 by adding InotifyReloader and 'use-inotify' configuration o…
…ption

Currently, '--reload' uses FS polling to find out when files have
changed. For some time, the Linux kernel has had a feature called
inotify that allows applications to monitor for FS events without
polling.

This commit adds a new 'use-inotify' configuration option that will
cause gunicorn to use the new 'InotifyReloader' instead of the default
'Reloader' when 'reload' is enabled.

Using inotify can result in lower CPU consumption by gunicorn especially
when working with virtualized filesystems or environments with a large
number of watched files / directories.

@benoitc benoitc closed this in 64b26ef Oct 25, 2016

@elliotbay22

This comment has been minimized.

Show comment
Hide comment
@elliotbay22

elliotbay22 Nov 4, 2016

Hey, I just happened upon this as something I'd like for my development flow too. When should I expect this will be available on pip?

elliotbay22 commented Nov 4, 2016

Hey, I just happened upon this as something I'd like for my development flow too. When should I expect this will be available on pip?

@mark-adams

This comment has been minimized.

Show comment
Hide comment
@mark-adams

mark-adams Dec 8, 2016

Contributor

Just wanted to follow-up @benoitc, do you have an estimate on when the next gunicorn release will happen that will include these changes?

Contributor

mark-adams commented Dec 8, 2016

Just wanted to follow-up @benoitc, do you have an estimate on when the next gunicorn release will happen that will include these changes?

@benoitc

This comment has been minimized.

Show comment
Hide comment
@benoitc

benoitc Dec 10, 2016

Owner

@mark-adams there is 1-2 patch to add to this release but there is no reason it can't be done next week.

Owner

benoitc commented Dec 10, 2016

@mark-adams there is 1-2 patch to add to this release but there is no reason it can't be done next week.

@elliotbay22

This comment has been minimized.

Show comment
Hide comment
@elliotbay22

elliotbay22 Jan 13, 2017

Ping! I keep finding that this would be useful to me, but I'd rather not cut myself off from pip to patch the implementation myself.

elliotbay22 commented Jan 13, 2017

Ping! I keep finding that this would be useful to me, but I'd rather not cut myself off from pip to patch the implementation myself.

@mark-adams

This comment has been minimized.

Show comment
Hide comment
@mark-adams

mark-adams Jan 19, 2017

Contributor

@benoitc I hope I'm not being a bother, but I wanted to follow-up on a release containing this version. Is there any way that I can get my hands dirty and help move us towards a release?

Contributor

mark-adams commented Jan 19, 2017

@benoitc I hope I'm not being a bother, but I wanted to follow-up on a release containing this version. Is there any way that I can get my hands dirty and help move us towards a release?

@berkerpeksag

This comment has been minimized.

Show comment
Hide comment
@berkerpeksag

berkerpeksag Jan 19, 2017

Collaborator

@mark-adams I think #1422 is the only blocker for a new release at the moment. You could help us to review #1430 :)

Collaborator

berkerpeksag commented Jan 19, 2017

@mark-adams I think #1422 is the only blocker for a new release at the moment. You could help us to review #1430 :)

@mark-adams

This comment has been minimized.

Show comment
Hide comment
@mark-adams

mark-adams Jan 19, 2017

Contributor

Just left a comment. Thanks @berkerpeksag

Contributor

mark-adams commented Jan 19, 2017

Just left a comment. Thanks @berkerpeksag

fofanov pushed a commit to fofanov/gunicorn that referenced this issue Mar 16, 2018

Fix #1368 by adding InotifyReloader and 'use-inotify' configuration o…
…ption

Currently, '--reload' uses FS polling to find out when files have
changed. For some time, the Linux kernel has had a feature called
inotify that allows applications to monitor for FS events without
polling.

This commit adds a new 'use-inotify' configuration option that will
cause gunicorn to use the new 'InotifyReloader' instead of the default
'Reloader' when 'reload' is enabled.

Using inotify can result in lower CPU consumption by gunicorn especially
when working with virtualized filesystems or environments with a large
number of watched files / directories.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment