manage.py error in SDK 1.9.17, old_dev_appserver removed

[pv-g]

With the new SDK 1.9.17, manage.py now raises an error : (on branch master)

Exception: Could not get appid. Is your app.yaml file missing? Error was: cannot import name old_dev_appserver

With branch devappserver2, it says :

File "[...]/djangoappengine/db/stubs.py", line 80, in setup_local_stubs
   from google.appengine.tools import dev_appserver_main
ImportError: cannot import name dev_appserver_main

[jacobg]

Me too.

[jacobg]

I commented out most of setup_local_stubs() function, and somehow it seems to work fine. Admittedly, I don't really understand all the details.

[npgauth]

I'm having the same problem. Is it possible to downgrade our development server? Will this cause any issues when we deploy to the cloud?

[pv-g]

I've downgraded the SDK for my current developments.
You can find the links here : https://cloud.google.com/appengine/downloads (Previous SDKs).
There currently is a forbidden error on the 1915 (1.9.15) Mac dmg, but the 1914 works fine.

I don't think you'd get issues deploying on GAE, I haven't tried, but you can safely deploy a new version, while your default (prod) version won't be affected.

[jacobg]

Like I said, I'm using the latest version of the SDK with the djangoappengine devappserver2 branch, and it works fine as long as I comment out some lines of code in db/stubs.py as follows:

def setup_local_stubs(self, connection):
    if self.active_stubs == 'local':
        return
    #from .base import get_datastore_paths
    #from google.appengine.tools import dev_appserver_main
    #args = dev_appserver_main.DEFAULT_ARGS.copy()
    #args.update(get_datastore_paths(connection.settings_dict))
    #args.update(connection.settings_dict.get('DEV_APPSERVER_OPTIONS', {}))
    log_level = logging.getLogger().getEffectiveLevel()
    logging.getLogger().setLevel(logging.WARNING)

    #try:
    #    from google.appengine.tools import dev_appserver
    #except ImportError:
    #    from google.appengine.tools import old_dev_appserver as dev_appserver
    #dev_appserver.SetupStubs(appid, **args)
    logging.getLogger().setLevel(log_level)
    self.active_stubs = 'local'

[npgauth]

Great thank you both. For some reason I wasn't able to get devappserver2 working - and it wasn't a stubs.py error that was causing the problem. Unfortunately it is a bit of a hectic week for me to try to figure out the problem, but I'll try to take a look again next week.. Regardless, rolling back to a prior version worked like a charm. Thanks again!

[aburgel]

I updated the devappserver2 branch to fix this issue. Please give it a try and let me know if it works. You can find it in this pull request: #77.

I'll merge it once I get a couple of confirmations that its working. Its becoming more difficult to support older versions of the SDK. So I may remove support for the old dev_appserver, now that its out of the SDK.

If you still use an old SDK for some reason, let me know.

[npgauth]

@aburgel - I just tested the devappserver2 branch and the startup errors were gone and my app seemed to function correctly.

Unfortunately, manage.py createsuperuser seems to no longer actually write to the local database (I did not try manage.py remote). I rolled back to the master branch and app engine 1.9.14 and the command then executes and writes to the database properly.. Has anything changed with the database writes in this branch?

Thanks for all your work!

[pv-g]

@aburgel : almost same thing here.
I could't test a lot of things, but my app seems to work correctly now.

The small things I noticed :

• a couple of warnings disappeared (GAE SDK backends depreciation, etc., this is a good thing)
• manage.py runserver now warns : sdk_update_checker.py:273] This SDK release is newer than the advertised release.
• manage.py syncdb doesn't print the GQL debug infos anymore

Thanks for all too :)
I'll try ton do more tests, using GAE modules, datastore and blobstore.

[aburgel]

the new devappserver changes how the database is managed. previously, you could startup django, open the database and make some changes, and then when django launched the appserver it would reuse the same database connection.

now the new devappserver manages its own database connection, so you need to ensure that django is talking to the same db file, which is a bit complicated because devappserver will sometimes store the database in a temp directory.

i'm still trying to wrap my head around it, so i don't know what the best way to approach the problem is yet.

[jacobg]

When I run manage.py tests, I get this exception:

NeedIndexError: This query requires a composite index that is not defined.
You must update the index.yaml file in your application root.

The suggested index for this query is:

- kind: django_content_type

• properties:*
• • name: app_label*
• • name: name*

I did not get this error before. Is something in the latest update not
finding index.yaml, or ... ?

ᐧ

On Tue, Dec 16, 2014 at 10:42 AM, Alex Burgel notifications@github.com
wrote:

the new devappserver changes how the database is managed. previously, you
could startup django, open the database and make some changes, and then
when django launched the appserver it would reuse the same database
connection.

now the new devappserver manages its own database connection, so you need
to ensure that django is talking to the same db file, which is a bit
complicated because devappserver will sometimes store the database in a
temp directory.

i'm still trying to wrap my head around it, so i don't know what the best
way to approach the problem is yet.

—
Reply to this email directly or view it on GitHub
#95 (comment)
.

[jacobg]

So in stubs.activate_test_stubs, if I add root_path=PROJECT_DIR to
init_datastore_v3_stubs like it used to have, then the error goes away and
tests run fine:

self.testbed.init_datastore_v3_stub(datastore_file=datastore_path,
root_path=PROJECT_DIR, **datastore_opts)

Should root_path be added back?
ᐧ

On Wed, Dec 17, 2014 at 9:18 AM, Jacob Gur jacob@fareclock.com wrote:

When I run manage.py tests, I get this exception:

NeedIndexError: This query requires a composite index that is not
defined. You must update the index.yaml file in your application root.

The suggested index for this query is:

- kind: django_content_type

• properties:*
• • name: app_label*
• • name: name*

I did not get this error before. Is something in the latest update not
finding index.yaml, or ... ?

On Tue, Dec 16, 2014 at 10:42 AM, Alex Burgel notifications@github.com
wrote:

the new devappserver changes how the database is managed. previously, you
could startup django, open the database and make some changes, and then
when django launched the appserver it would reuse the same database
connection.

now the new devappserver manages its own database connection, so you need
to ensure that django is talking to the same db file, which is a bit
complicated because devappserver will sometimes store the database in a
temp directory.

i'm still trying to wrap my head around it, so i don't know what the best
way to approach the problem is yet.

—
Reply to this email directly or view it on GitHub
#95 (comment)
.

[pv-g]

I have another issue when running manage.py syncdb.
In db/stubs.py as well, it runs :

• setup_stubs
• activate_stubs
• activate_test_stubs
  It runs setup_local_stubs and not setup_local_stubs, is it normal?

It then prints the result :

Creating tables ...
Installing custom SQL ...
Installing indexes ...
Installed 1 object(s) from 1 fixture(s)

But nothing is written in the local database.
No .gaedata/datastore file is even written.

[pv-g]

@aburgel : I'm trying a GAE multiple-modules architecture (with multiple Django), maybe the simpler would be to add a datastore_path param in the settings, something like :

DATABASES = {
    'default': {
        'ENGINE': 'djangoappengine.db',
        'DEV_APPSERVER_OPTIONS': {
            'use_sqlite': True,
            'datastore_path': '[path]'
        },
    }
}

I tried to manually set DATA_ROOT in boot.py, and it works good!

[pv-g]

(sorry, closed accidentally)

[pv-g]

Another thing, I don't know if I did a mistake but when my app's entry point is :

# main.py
from djangoappengine.main import application

I get this error :

ERROR    2014-12-17 17:08:38,097 wsgi.py:263] 
Traceback (most recent call last):
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
    obj = __import__(path[0])
  File "[...]/AppEngine/module1/main.py", line 14, in <module>
    from djangoappengine.main import application
  File "[...]/AppEngine/module1/lib/djangoappengine/main/__init__.py", line 30, in <module>
    from django.conf import settings
ImportError: No module named django.conf
INFO     2014-12-17 17:08:38,111 module.py:718] module1: "GET / HTTP/1.1" 500 -

I fix it this way :

# main.py
from django import conf # tmp debug
from djangoappengine.main import application

but I don't really understand why.

[aburgel]

@jacobg root_path doesn't seem to be used by testbed any more, though maybe its further down the call chain. i will investigate this weekend.

[aburgel]

@Philoozushi does this mean you can't run your app in production? because i think djangoappengine.main is the entry point is used in production

[jacobg]

@aburgel root_path is one of the named parameters of datastore_file_stub.DatastoreFileStub, which is called by init_datastore_v3_stub. Adding it fixed the NeedIndexError I was getting. You don't get that error when running a unit test? Try require_indexes to True.

[aburgel]

@jacobg I've been using sqlite which uses datastore_file. I will add root_path back in.

[jacobg]

I'm also using datastore_path, but it's set to outside my project directory. root_path is for index.yaml. Are you saying that if datastore_path is where index.yaml is, then GAE derives that as well?

[jacobg]

Yeah, I don't think the local stubs has been updated to work with the latest SDK version which removed the old devappserver. @aburgel Can you help us here please? Thanks!

[jacobg]

Regarding my last two comments, see the following new issue I created to address it. It includes patch fix:
#97

[interpegasus]

Hi everyone, After switching to devappserver2 the app(s) works fine again, but it is slower. It looks like models are validated on each request (Even many times per request).
Before starting the server was very fast. And now it takes around 2-4 secs.
Also according to logs Google development console, the app received more than 100000 requests in 24h period. Likely those are internal API request to the datastore from the devappserver2?
Is there a way to optimize or solve this issue? Thanks

• nonrel-1.6 branch + devappserver2 branch + latest Google SDK

[pv-g]

Hi @interpegasus,
I run the same config as you, and I don't have such a problem.
It may be related with something more specific to your code.
Or do you have more details?

[pv-g]

Sorry @aburgel, I missed your last mention.
My main.py is still the same, with that little fix.
It was running well locally, and on AppEngine production.

But I just tried to remove it, and now it works well too, locally, and on production!
I'm unable to tell you why the error disappeared...

[interpegasus]

@Philoozushi Thanks for confirming the same configuration, works fine for you.
The issue has been fixed on my app(s). And these were the steps I took:

• Installing latest version of all python libraries, Google App Launcher Python SDK, django nonrel and dependencies.
• Improving the app's cache system.
  The app is fast again, Datastore Read Operations are optimal, and models are validated once instead of many times.

Thanks again!

[dho825]

Hey guys, after updated appengine sdk to 1.9.18(or 17), and applying all the fixes and patches here, I can't seem to get syncdb to create a superuser and access the admin section

Regarding @jacobg's #97, I realized that the datastore_path is None instead of pointing to the usual .gaedata folder in the PROJECT_DIR. Not sure if @Philoozushi meant this as well, but I imported DATA_ROOT from base.py and applied an os.path.join(DATA_ROOT, 'datastore'). (This is similar in base.py where DATABASE_PATHS['datastore_path'] is defined as such)

I'm now wondering if there are issues with syncdb or any other datastore related devappserver operations since this problem only occurs on a fresh project. Using a pre-existing .gaedata/datastore with the initial admin and auth models built did not throw any errors. Haven't played around enough to identify any issues, but having the datastore_path=None forces the devappserver to create a temp db (for me at /vars/folders/6_/<gibberish>/appengine.appname-dev.username/)

I recognize this as low priority if this only impacts the dev environment, not production...but I hope to get some feedback on this hack-y patch. Are all the imports in db/stubs.py for dev_appserver_*.py all related to the old_dev_appserver?

[dho825]

@GxelA from the logs it seems you have not migrated over to using devappserver2.py since the new SDK, old_dev_appserver.py has been removed. I would recommend pulling from the devappserver2 branch and following the patches above to see if that fixes things