'subprocess' module monkey-patch warning in boot.py

[changemewtf]

WARNING boot.py Could not add the subprocess module to the sandbox: 'module' object has no attribute 'Popen'

This happens at the end of the setup_project() method in boot.py, in the try/except clause labeled via comment as "Restore the real subprocess module."

        elif not on_production_server:
        try:
            # Restore the real subprocess module.
            from google.appengine.api.mail_stub import subprocess
            sys.modules['subprocess'] = subprocess
            # Re-inject the buffer() builtin into the subprocess module.
            from google.appengine.tools import dev_appserver
            subprocess.buffer = dev_appserver.buffer
        except Exception, e:
            logging.warn("Could not add the subprocess module to the "
                         "sandbox: %s" % e)

I'm not sure why this code chooses to import subprocess from the google.appengine.api.mail_stub module, because the "real" import of subprocess at the top of mail_stub.py appears to be overriden at some point with the monkey-patched subprocess module, which of course does not have a Popen attribute.

The caught error is thrown during the import of mail_stub.py itself:

Traceback (most recent call last):
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 196, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 266, in _LoadHandler
    __import__(cumulative_path)
  File "/Users/mcantor/src/_____/src/_____/djangoappengine/main/__init__.py", line 28, in <module>
    setup_env()
  File "/Users/mcantor/src/_____/src/_____/djangoappengine/boot.py", line 75, in setup_env
    setup_project()
  File "/Users/mcantor/src/_____/src/_____/djangoappengine/boot.py", line 175, in setup_project
    from google.appengine.api.mail_stub import subprocess
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/mail_stub.py", line 48, in <module>
    class MailServiceStub(apiproxy_stub.APIProxyStub):
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/mail_stub.py", line 214, in MailServiceStub
    popen=subprocess.Popen,
AttributeError: 'module' object has no attribute 'Popen'

[aburgel]

I'm not sure where this error is coming from. The djangoappengine code is not modifying the subprocess module, its just adding it back to the list of installed modules (dev_appserver removes it in order to match the locked-down production environment). So if you subprocess is broken in some way, its unlikely that djangoappengine is causing it.

What versions of python, appengine SDK and djangoappengine are you using?

[changemewtf]

Python 2.7.2, GAE 1.8.1, and I've tried this with djangoappengine 1.3 and 1.4.

None of the obvious stuff seems to be the problem (I have no files or folders called "subprocess" or "subprocess.py", etc).

I haven't figured out where in google_appengine itself subprocess is being changed, but it is being changed. If I put print "Subprocess:", subprocess.__file__ at the top of mail_stub.py, then my server output looks like this:

Subprocess: /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.pyc
INFO     2013-07-16 16:19:20,447 sdk_update_checker.py:244] Checking for updates to the SDK.
INFO     2013-07-16 16:19:20,593 sdk_update_checker.py:272] The SDK is up to date.
INFO     2013-07-16 16:19:20,619 api_server.py:138] Starting API server at: http://localhost:51625
INFO     2013-07-16 16:19:20,623 dispatcher.py:164] Starting server "default" running at: http://0.0.0.0:8080
INFO     2013-07-16 16:19:20,634 dispatcher.py:164] Starting server "reporter" running at: http://0.0.0.0:8081
INFO     2013-07-16 16:19:20,636 admin_server.py:117] Starting admin server at: http://localhost:8000
Subprocess: /Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/dist/subprocess.pyc
WARNING  2013-07-16 16:19:22,264 boot.py:183] Could not add the subprocess module to the sandbox: 'module' object has no attribute 'Popen'

As you can see, when mail_stub.py is first loaded, its subprocess is the Python module. However, by the time the sandbox starts up and tries to import subprocess from it later, it has been monkeypatched to the google_appengine's version.

It seems worth noting that this file--$APPENGINE/google/appengine/dist/subprocess.py--is empty except for comments.

I've grepped for dist.subprocess in google_appengine itself, but there were no results, so it's likely that some relative path sorcery is being used. I haven't sifted through all of the results of grepping for just subprocess yet.

I may be going down the wrong rabbit hole here, because it seems like this problem would have happened to a lot of other users by now if it was an issue with djangoappengine itself rather than my individual setup. I've tried re-installing GAE and djangoappengine in case some kind of far-fetched file garbling happened, so I'm not sure what approach to take next.

It's possible though vanishingly unlikely that my application code is causing this problem. I've grepped for subprocess throughout the entire codebase, and the only hits are in djangoappengine. So, unless there is some way for my configuration to screw up GAE's sandbox routine such that its fake subprocess ends up clobbering the real one at the wrong time, there is something else going on.

[aburgel]

It looks like you're using the new dev_appserver, which is not officially supported by djangoappengine yet. There's a pull request for it, but I haven't merged it in yet. Try switching back to the old dev_appserver and let me know if that still has problems. Or alternatively try this pull request: #46

[changemewtf]

@aburgel - Please forgive my naïvety on GAE basics, I am still getting up to speed on our app. I'm running $GAE/dev_appserver.py; do you mean I should be running $GAE/old_dev_appserver.py?

Also, I didn't know about djangoappengine not supporting the new dev_appserver. I didn't see any references to that in the docs or anything. If #46 isn't going to be released very soon, do you think it's worth considering to add a check during startup that causes djangoappengine to simply explode when you try to run it with the new dev_appserver? It would completely prevent issues like this, and could be easily removed once support has been added.

[aburgel]

You should actually be running ./manage.py runserver that will use the correct version of the dev_appserver. In the past, it was possible to run it the way you are, but then google released a new version of dev_appserver which seems to have broken a lot of things, as you discovered.

I will look into adding a check to prevent others from running into the same issue that you did.

[changemewtf]

Oh, of course! That's in the first two paragraphs of the docs. I'm sorry; I should have noticed that much sooner. I was a bit flummoxed because the GAE docs specifically admonish users to run dev_appserver.py instead of native tools, but it makes sense that djangoappengine would coerce django into running it correctly.

Things appear to be working smoothly now except for this error on startup:

WARNING  2013-07-16 12:21:09,935 dev_appserver.py:3579] The datastore file stub is deprecated, and
will stop being the default in a future release.
Append the --use_sqlite flag to use the new SQLite stub.

You can port your existing data using the --port_sqlite_data flag or
purge your previous test data with --clear_datastore.

This happens after running python manage.py syncdb to load some text fixtures, then running python manage.py runserver. (If I don't run syncdb first, then the app does not create .gaedata/datastore itself and complains of not being able to find the datastore file.)

Am I correct in thinking that this warning is (currently) benign, and can be circumvented by configuring my app to use sqlite by default?

Thanks very much for your help.

[aburgel]

That warning can be fixed if you update your settings.py, you need to tell it to use sqlite, you can do something like this:

DATABASES['native']['DEV_APPSERVER_OPTIONS'] = { 'high_replication': True, 'use_sqlite': True }

[changemewtf]

Excellent, that works like a charm. Thanks again!

[veenahr]

aburgel, Tried your solution but it did not work. This is the error I am seeing.

DATABASES['native']['DEV_APPSERVER_OPTIONS'] = { 'high_replication': True, 'use_sqlite': True }

KeyError: 'native'
INFO 2014-04-27 22:28:19,651 module.py:627] default: "GET /_ah/warmup HTTP/1.1" 500 -
WARNING 2014-04-27 22:29:08,593 boot.py:184] Could not add the subprocess module to the sandbox: 'module' object has no attribute 'Popen'
INFO 2014-04-27 22:29:09,260 init.py:44] Validating models...

Any help or direction on solving this would be helpful.

[aburgel]

It looks like you do not have a database with the name 'native'. Is yours named 'default'?

[veenahr]

This is what I currently have in my settings.py file.

DATABASES['native'] = DATABASES['default']
DATABASES['default'] = {'ENGINE': 'dbindexer', 'TARGET': 'native'}
AUTOLOAD_SITECONF = 'indexes'

Any help would be greatly appreciated.

[aburgel]

The KeyError you're seeing is because you are accessing DATABASES['native'] without it being in the dictionary yet. settings.py is just a regular python file, so you need to follow regular python rules.

It can also be debugged like a regular python file, so you can add in print and logging statements to see whats going on. I would suggest doing that to figure out what's going wrong.

[veenahr]

Thanks. I will do that. Newbie in this right now.

Veena.

--Veena.

Veena Hampapur
PTENlab, Inc.

3150 Porter Drive
Palo Alto, CA 94304
Tel: +1-510-304-3799
E-mail: veena@p veena@ptenlab.comtenlab.com
Website: www. http://dynamicbio.com/ptenlab.com

On Thu, May 1, 2014 at 12:37 PM, Alex Burgel notifications@github.comwrote:

The KeyError you're seeing is because you are accessing
DATABASES['native'] without it being in the dictionary yet. settings.py
is just a regular python file, so you need to follow regular python rules.

It can also be debugged like a regular python file, so you can add in
print and logging statements to see whats going on. I would suggest doing
that to figure out what's going wrong.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/53#issuecomment-41947812
.