Reloader option to exclude watch paths

[rotemdan]

Seems like the server gets reloaded when gen_py caching files (created by the win32com module) are written or modified (in a temporary directory):

 * Detected change in 'C:\\Users\\X\\AppData\\Local\\Temp\\gen_py\\3.6\\C866CA3A-32F7-11D2-9602-00C04F8EE628x0x5x4\\__pycache__\\SpVoice.cpython-36.pyc', reloading
 * Detected change in 'C:\\Users\\X\\AppData\\Local\\Temp\\gen_py\\3.6\\C866CA3A-32F7-11D2-9602-00C04F8EE628x0x5x4\\__pycache__\\_ISpeechVoiceEvents.cpython-36.pyc', reloading

This seems to happen quite often. Any way for this to be prevented?

(Using Python 3.6, Flask 1.0.2, Werkzeug 0.14.1 on Win10 64bit. FLASK_ENV=debug, FLASK_DEBUG=1)

Edit: this also causes occasional cache corruptions (causing runtime errors and requiring wiping the gen_py directory to resolve) since it abruptly cuts out the process while it is writing (usually several files) to disk.

[angadkalra]

set this environment variable: PYTHONDONTWRITEBYTECODE=1

[rotemdan]

Thanks for the advice. I wiped the temporary gen_py directory and I tried it. Seems like .pyc files aren't written, however there are still .py files being written there. Causing the server to reload:

 * Detected change in 'C:\\Users\\X\\AppData\\Local\\Temp\\gen_py\\3.6\\C866CA3A-32F7-11D2-9602-00C04F8EE628x0x5x4\\ISpeechObjectTokens.py', reloading
 * Detected change in 'C:\\Users\\X\\AppData\\Local\\Temp\\gen_py\\3.6\\C866CA3A-32F7-11D2-9602-00C04F8EE628x0x5x4\\SpObjectToken.py', reloading
 * Detected change in 'C:\\Users\\X\\AppData\\Local\\Temp\\gen_py\\3.6\\C866CA3A-32F7-11D2-9602-00C04F8EE628x0x5x4\\ISpeechObjectToken.py', reloading
 * Detected change in 'C:\\Users\\X\\AppData\\Local\\Temp\\gen_py\\3.6\\C866CA3A-32F7-11D2-9602-00C04F8EE628x0x5x4\\SpMemoryStream.py', reloading
 * Detected change in 'C:\\Users\\X\\AppData\\Local\\Temp\\gen_py\\3.6\\C866CA3A-32F7-11D2-9602-00C04F8EE628x0x5x4\\ISpeechMemoryStream.py', reloading
 * Detected change in 'C:\\Users\\X\\AppData\\Local\\Temp\\gen_py\\3.6\\C866CA3A-32F7-11D2-9602-00C04F8EE628x0x5x4\\SpAudioFormat.py', reloading
 * Detected change in 'C:\\Users\\X\\AppData\\Local\\Temp\\gen_py\\3.6\\C866CA3A-32F7-11D2-9602-00C04F8EE628x0x5x4\\ISpeechAudioFormat.py', reloading

[angadkalra]

Does your app successfully reload after seeing those lines above?

[rotemdan]

@angadkalra The app reloads (although there is no practical need for it to). Though since the win32com module is writing multiple files to disk and is abruptly terminated by the reloader, it occasionally results in a cache corruption (some files successfully written, some don't or only partially). Next time it loads it may sometimes crash the server with a runtime error of the sort Can't load COM module ... since the cache has been corrupted and only wiping out the gen_py dir would seem to resolve it.

The goal is not to have the server reload when files (including .py) are written to cache or temporary directories. It seems reasonable there's already a way to do it but I'm not sure how to achieve it.

[davidism]

I realized this will break the reloader if the virtualenv itself is in the temp folder. None of the tools I commonly use do that by default (pew temp envs, tox, nox), and I don't think Pipenv or Poetry do, but there are a lot of other tools and workflows out there.

Instead, let's allow passing a list of exclude_paths like the extra_files. I'm not sure of the exact semantics for this. I know there's a difference between the stat and watchdog reloaders about watching paths. I'm not sure what order extra and exclude should be considered in.

[sphakka]

Please consider using wildcards for exclude_paths: I see reload occurring on changing editor's (Emacs) backup files (.#file.py), which is utterly annoying as those are modified automatically without saving the original file.

So, exclude_paths = [ '.#*.py', ... ] would be great, and should possibly be the default with other tricky patterns.

[danihodovic]

#1861

The reloader fails to detect changes if there is a file owned by root and permissions set to 0700 in the project directory. This is common if using docker-compose for development with locally stored database files.

It seems like you can pass ignore_patterns to watchdog. Maybe that's helpful. https://pythonhosted.org/watchdog/api.html#watchdog.events.PatternMatchingEventHandler.ignore_patterns