Remove the imported app module from sys.modules on SyntaxError

[segevfiner]

When a SyntaxError occurrs in a package's __init__.py file, Python seems to leave an empty module in sys.modules. This will result in any request after the first to give you a NoAppException instead of the SyntaxError again.

I'm not 100% sure this doesn't have any side effects... It's an hack... So please review and tell me what you think about this. (Maybe more conditions for triggering this?)

Fixes #2423

Sample

In hello/__init__.py:

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'

set FLASK_APP=hello
set FLASK_DEBUG=1
flask run

Than edit __init__.py and introduce a syntax error.

[davidism]

Is there an easy way to write a test for this? Otherwise, seems fine to me.

[mitsuhiko]

This does not just apply for packages, it affects a boatload of situations and this only catches on case. I'm not sure what the best approach here is.

[mitsuhiko]

One way would be to patch __import__ when the reloader is active and catch down ImportErrors and clean up recursively.

[segevfiner]

Hmm. I see... This won't catch sub-packages that fail to import on SyntaxError, only the main app package. As such, they will be left empty, and on the next request the app will get them as an empty module. Flask won't fail to find the app, but the app will fail to find stuff in that package...

Seriously that Python behavior is so off...

[davidism]

Maybe modify the Werkzeug reloader to record the loaded modules before starting the reloader, then reset them before triggering a reload.

[davidism]

I can't reproduce this on Python 2.7 or 3.6, Flask 0.12 and master. I tried starting with valid syntax then editing in an error, and vice versa.

[davidism]

The test you added passes without your patch.

[segevfiner]

Weird. It didn't pass without the patch for me... I wonder what is different... I tested on Python 2.7.14 32-bit, running on Windows 10 x64 (Latest ATM), inside a virtualenv that contains only Flask & Werkzeug from Git and their dependencies. Another idea is that maybe that weird behavior isn't in core Python and is triggered by something else... At that point the bug will be in that code and not in Flask. (Some import machinery hook...? That fails to delete the module from sys.modules on SyntaxError) I will try to reproduce and figure this out later when I have the time. (I will try to also post more exact environment details) בתאריך 9 בינו' 2018 17:21,‏ "David Lord" <notifications@github.com> כתב:

…

The test you added passes without your patch. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2588 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AXlg__1wOmGnaXv9lG2ahDrBUJ0hs5Aaks5tI4QNgaJpZM4RW0lH> .

[davidism]

I've had developers try to reproduce this on Windows, Mac, and Linux with and without the Watchdog reloader. It works for all of them.

[davidism]

The normal Python importer is careful about cleaning up after failed imports. It's likely that something else in your stack is affecting this. I'm going to close this for now, but if you can figure out what's going on I can reopen it later.

[mitsuhiko]

@davidism there are long running bugs in the interpreter where cleanup does not correctly happen. It's a limitation of the python importer.

[segevfiner]

I went with addressing the underlying issue in CPython python/cpython#5142.

[davidism]

I'm not doubting that the error happens for you, but I'm curious why no one I asked was able to reproduce this in a variety of environments. I think getting cpython to fix it is the right course of action.

[segevfiner]

@davidism Yeah I got curious about that too. Which is why I went ahead and debugged it. The bug is obviously in 2.7 but not in 3.6. I literally went in with a debugger and saw the exact flow of code that causes this. It's hard to tell why they weren't able to reproduce it though.