New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow ndb to run from outside of App Engine #272

Open
pcostell opened this Issue Nov 10, 2016 · 26 comments

Comments

Projects
None yet
@pcostell
Member

pcostell commented Nov 10, 2016

No description provided.

@kevinjdolan

This comment has been minimized.

kevinjdolan commented Nov 10, 2016

IS THIS HAPPENING FOR REAL?

@erugeri

This comment has been minimized.

erugeri commented Nov 16, 2016

Would love to see it for objectify / ndb to be able to migrate to flex.
Thanks!

@gcbirzan

This comment has been minimized.

gcbirzan commented Nov 16, 2016

The ndb interface to the datastore is vastly superior to the datastore one, at least for Python, hopefully this gets fixed.

@jredl-va

This comment has been minimized.

jredl-va commented Dec 5, 2016

With the deprecation of the compat-runtime for appengine-flex this functionality would be a game changer ultimately enabling us to move some of our workloads off of appengine standard into kubernetes without rewriting large chunks of our applications.

@Erfa

This comment has been minimized.

Contributor

Erfa commented Dec 19, 2016

This seems like one of the most important things to fix in order to make it easier to migrate from standard App Engine to flexible environments.

@jonathan-s

This comment has been minimized.

jonathan-s commented Jan 22, 2017

Would love to see this!

@aatreya

This comment has been minimized.

aatreya commented Feb 2, 2017

@pcostell Is this still under development? Do you have a rough timeframe on an alpha/beta release?

@ekampf

This comment has been minimized.

ekampf commented Feb 9, 2017

Any news on this?
Google is really dropping the ball on GAE users here...

@artooro

This comment has been minimized.

artooro commented Mar 10, 2017

Just want to comment, in the hopes there will be an EAP for this. I'd love be on it.

@vanpelt

This comment has been minimized.

vanpelt commented Mar 12, 2017

+1000

@Bogdanp

This comment has been minimized.

Bogdanp commented Mar 28, 2017

Excuse the shameless self-promotion but those of you on Python 3.6+ might find this project useful: https://github.com/Bogdanp/anom-py

@bsmithyman

This comment has been minimized.

bsmithyman commented Apr 7, 2017

Would it be possible for someone from Google (e.g., @pcostell) to update this issue?

With the new flexible environment expansion and GA status, we are left trying to decide whether to pursue a Python 3 refactor of our backend or continue using NDB. At present, we can't do both.

This is pushing us to consider switching databases, purely from a business risk point of view. Likely, that would be to Cloud Datastore directly, but it is not the only one in the running. A bit of extra information here would be incredibly helpful. We certainly want to continue to use NDB, but it's starting to look pretty risky. From the outside, it seems like we may need to choose between NDB and Python 3 (which is a pretty tough sell).

Thanks!

@artooro

This comment has been minimized.

artooro commented Apr 7, 2017

@bsmithyman I'm not from Google, but have been told this week that they are working on it. Even so it would be nice to get a technical update from a google engineer here just to regain some confidence on the app engine platform.

@bsmithyman

This comment has been minimized.

bsmithyman commented Apr 10, 2017

Thanks @artooro; appreciated.

@mikelambert

This comment has been minimized.

mikelambert commented May 30, 2017

So from what I understand, the GAE Managed VM python-compat bridge is being turned down end-of-October, in favor of GAE Flexible VMs use of python images.

Is "Make NDB work outside of GAE Standard" a blocker for "turn down the python-compat bridge" for the GAE/FlexVM/NDB teams?

Or will I need to plan for the possibility that my GAE NDB code will not run on python GAE Flexible? And if I need to rewrite...what is the recommended DB solution (gcloud.datastore, DB, and NDB) for common-code that needs to run in both GAE Standard and GAE Flexible? Currently, they all seem to have drawbacks:
gcloud.datastore in GAE Standard (super slow versus native bridge)
DB/NDB in GAE Flexible (library doesn't exist, and isn't supported)

@cpavon

This comment has been minimized.

cpavon commented Jun 26, 2017

@mikelambert Same issue here.... python-compat is gonna be decommissioned in October so we have to face some important decisions.

We also need to keep code running in both environments: Standard and Flexible. So if we don't have a common way to access to datastore we will have a problem.

Someone has face the same problem? alternatives?

@kevinjdolan

This comment has been minimized.

kevinjdolan commented Jun 26, 2017

It's very clear that Google is not pushing the standard AppEngine environment anymore -- to the point of deprecation, in my opinion. I decided to just pull the bandaid and move to Container Engine.

@MrMaxRoss

This comment has been minimized.

MrMaxRoss commented Jun 26, 2017

Hi Kevin,

I'm the eng director for App Engine. I saw your comment about the Standard environment and felt compelled to respond.

I hope Container Engine is working well for you, but there's actually a massive investment in the Standard environment underway, and you're going to start seeing the results of that investment very soon in the form of updated Standard runtimes. I'm not making any announcements here, but if you look at the blog post where we brought the Flex environment to GA you'll see we've been running an Alpha for Java 8 on Standard, and a recent blog post on Python for App Engine also points at big changes in the pipeline.

Thanks,
Max

@workmanw

This comment has been minimized.

workmanw commented Jun 26, 2017

@MrMaxRoss I really hope you announce something soon, hinting at changes can only take us so far when we're trying to make real business decisions. Right now we use NDB on Flex via legacy (vm:true). However that is being sunset in October. So we're actively trying to figure out what parts of our application we need to rewrite.

@bsmithyman

This comment has been minimized.

bsmithyman commented Jun 26, 2017

I would echo @workmanw, inasmuch as we would love to be using both NDB and Python 3, and the uncertainty makes it hard to bet on this environment. A Python 3 Standard Environment with NDB (as hinted in previous comms) would be very welcome. Thanks for updating as you're able, @MrMaxRoss.

@MrMaxRoss

This comment has been minimized.

MrMaxRoss commented Jun 27, 2017

We are still planning to sunset vm:true in October, but you can upgrade to env:flex and request an exception to continue using the compat runtime past October. There are some devs testing NDB on env:flex using the compat runtime right now (one bug so far). I will report back when we are confident that it works.

I know this is different from a fully supported NDB on Python on GAE Flex, but I want you to know that we expect there to be an option for NDB beyond October.

@brianmhunt

This comment has been minimized.

brianmhunt commented Sep 20, 2017

Hi @MrMaxRoss — I think a lot of folks (including myself) would very much enjoy an update, insofar as you can, on where this issue stands. Thank you, sincerely!

@wsh

This comment has been minimized.

Member

wsh commented Sep 26, 2017

@brianmhunt hi there! We can't provide specific guidance about dates just yet, but we have engineers working on this right now. (I'm one of them.) I'll be sure to keep you posted on this thread.

@wsh wsh self-assigned this Sep 26, 2017

@lcasartelli

This comment has been minimized.

lcasartelli commented Feb 9, 2018

Hi @wsh! Do you have any news for us? Thanks!

@robhaswell

This comment has been minimized.

robhaswell commented May 30, 2018

Hey @wsh how are you getting on?

@jer-tx

This comment has been minimized.

jer-tx commented Jul 31, 2018

Any updates on this? We're looking to move from Standard to Flexible and not having an ORM to talk to datastore is a massive barrier.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment