Client Datastore API with App Engine Devserver #1328

mitchhentges · 2016-10-21T00:40:29Z

I've investigated into #216, but I can't quite get my Client Datastore connection to work with the App Engine Devserver.

I've tried:

# No environment variables, DatastoreOptions.defaultInstance().service()
com.google.datastore.v1.client.DatastoreException: Request had invalid authentication credentials.

# DATASTORE_EMULATOR_HOST=localhost:8080
# DatastoreOptions.defaultInstance().service()
WARNING: No file found for: /v1/projects/navistream2dev:runQuery
SEVERE: com.google.cloud.datastore.DatastoreException: Non-protobuf error: <html><head><title>Error 404</title></head>
<body><h2>Error 404</h2></body>
</html>. HTTP status code was 404.

# export DATASTORE_PROJECT_ID=navistream2dev   
# export DATASTORE_EMULATOR_HOST=localhost:8080
# DatastoreOptions.defaultInstance().service()
WARNING: No file found for: /v1/projects/navistream2dev:runQuery
SEVERE: com.google.cloud.datastore.DatastoreException: Non-protobuf error: <html><head><title>Error 404</title></head>
<body><h2>Error 404</h2></body>
</html>. HTTP status code was 404.

# export DATASTORE_PROJECT_ID=navistream2dev
# DatastoreOptions.defaultInstance().service()
WARNING: failed _ah_AbandonedTransactionDetector: java.lang.RuntimeException: Could not get Cloud Datastore options from environment.
Oct 20, 2016 4:45:00 PM com.google.apphosting.utils.jetty.JettyLogger warn
WARNING: Failed startup of context com.google.appengine.tools.development.DevAppEngineWebAppContext@21ffbe3b{/,/home/mitch/dev/navistream/server/navistream/build/exploded-app}
java.lang.RuntimeException: Could not get Cloud Datastore options from environment.
Caused by: java.io.IOException: Application Default Credentials failed to create the Google App Engine service account credentials class com.google.appengine.repackaged.com.google.api.client.googleapis.extensions.appengine.auth.oauth2.AppIdentityCredential$AppEngineCredentialWrapper. Check that the component 'google-api-client-appengine' is deployed.
Caused by: java.lang.ClassNotFoundException: com.google.appengine.repackaged.com.google.api.client.googleapis.extensions.appengine.auth.oauth2.AppIdentityCredential$AppEngineCredentialWrapper

// Interestingly, this is trying to get `AppIdentityCredential` from "com.google.appengine.repackaged.com.google.api...", rather than from "com.google.api..."

Am I missing an environment variable for configuration to make this work nicely?

The text was updated successfully, but these errors were encountered:

mziccard · 2016-10-21T06:37:10Z

What are you trying to achieve here?

Issue #216 says that you cannot access Devserver's Datastore as it does not expose either REST or gRPC APIs:

currently local dev app services such as Datastore are not accessible via the cloud interface (apiary or gRpc).

So you will not be able to access Devserver's Datastore by changing DatastoreOptions or setting environment variables.

What is your final goal?

mitchhentges · 2016-10-21T07:03:56Z

I was really hoping that that part of the first comment would be inaccurate, considering that I'm having unauthenticated errors, and there was a comment about ApplicationDefaultAuthCredentials.

But yeah, what I want to accomplish, is being able to run my application locally. Unfortunately, it's still tied to some appengine APIs (memcache). So, I need to run the application within the appengine devserver, or the memcache-related component will fail. At the same time, I need to talk to a datastore emulator to store my data. If I don't need to run an external datastore emulator, it would make my life 200% easier 😉

mziccard · 2016-10-21T07:49:28Z

@mitchhentges I believe that comment was about using the actual Datastore service, not the DevServer's emulator, from code running in the DevServer.

I know this is suboptimal but is using the external Datastore emulator such a pain for you? You should be able to start it with:

gcloud beta emulators datastore start

And once you export the emulator host with export DATASTORE_EMULATOR_HOST=localhost:<yourport> you should also be able to access it from the DevServer.

mitchhentges · 2016-10-21T07:54:48Z

The reason that starting the external datastore emulator is so annoying is because the port always changes.

With the old, appengine datastore API:

From within IDE, ./gradlew appEngineRun
Application is live

Once we have to deal with an external datastore:

Start datastore emulator
Copy port
Edit IDE run preferences, set environment variable for datastore emulator host/port
./gradlew appengineRun
Application is live

mziccard · 2016-10-21T08:02:52Z

You can set the port when you start the emulator:

gcloud beta emulators datastore start --host-port=localhost:<yourpreferredport>

See gcloud beta emulators datastore start --help for more options.

This is still suboptimal but gets rid of points 2 and 3 :)

mitchhentges · 2016-10-21T08:25:23Z

Ah, that helps, and it's confirmed that the client datastore api cannot communicate with the appengine devserver datastore. I have clear direction now, thanks!

josenicomaia · 2022-03-20T20:09:34Z

@mitchhentges, you can also create an adapter between then. When using Spring Data I can do this:

@SpringBootApplication
public class Application {

    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }

    @Bean
    public DatastoreService datastoreService() {
        return DatastoreServiceFactory.getDatastoreService();
    }

    @Value("${spring.cloud.gcp.datastore.project-id}")
    public String projectId;

    @Bean
    public Datastore datastore(DatastoreService datastoreService) {
        return new DatastoreAdapter(datastoreService, projectId);
    }
}

public class DatastoreAdapter implements Datastore {
    private final DatastoreService datastoreService;
    private final String projectId;

    public DatastoreAdapter(DatastoreService datastoreService, String projectId) {
        this.datastoreService = datastoreService;
        this.projectId = projectId;
    }

    @Override
    public Transaction newTransaction(TransactionOptions transactionOptions) {
        return null;
    }

    @Override
    public Transaction newTransaction() {
        return null;
    }

    @Override
    public <T> T runInTransaction(TransactionCallable<T> transactionCallable) {
        return null;
    }

    @Override
    public <T> T runInTransaction(TransactionCallable<T> transactionCallable, TransactionOptions transactionOptions) {
        return null;
    }

    @Override
    public Batch newBatch() {
        return null;
    }

    @Override
    public Key allocateId(IncompleteKey incompleteKey) {
        return null;
    }

    @Override
    public List<Key> allocateId(IncompleteKey... incompleteKeys) {
        return null;
    }

    @Override
    public List<Key> reserveIds(Key... keys) {
        return null;
    }

    @Override
    public Entity add(FullEntity<?> fullEntity) {
        return null;
    }

    @Override
    public List<Entity> add(FullEntity<?>... fullEntities) {
        return null;
    }

    @Override
    public void update(Entity... entities) {

    }

    @Override
    public Entity put(FullEntity<?> fullEntity) {
        return null;
    }

    @Override
    public List<Entity> put(FullEntity<?>... fullEntities) {
        List<com.google.appengine.api.datastore.Entity> gaeEntityList = Arrays.stream(fullEntities)
                .map((FullEntity<?> fullEntity) -> {
                    Key incompleteKey = (Key) fullEntity.getKey();
                    com.google.appengine.api.datastore.Key key = convertGCToGAEKey(incompleteKey);
                    com.google.appengine.api.datastore.Entity entity = new com.google.appengine.api.datastore.Entity(key);
                    copyGCToGAEEntityProperties(fullEntity, entity);

                    return entity;
                })
                .collect(Collectors.toList());

        datastoreService.put(gaeEntityList);

        return Arrays.stream(fullEntities)
                .map(fullEntity -> {
                    Key key = (Key) fullEntity.getKey();
                    return Entity.newBuilder(key, fullEntity).build();
                })
                .collect(Collectors.toList());
    }

    @Override
    public void delete(Key... keys) {

    }

    @Override
    public KeyFactory newKeyFactory() {
        return new KeyFactory(projectId, "default");
    }

    @Override
    public Entity get(Key key, ReadOption... readOptions) {
        return null;
    }

    @Override
    public Iterator<Entity> get(Iterable<Key> iterable, ReadOption... readOptions) {
        return null;
    }

    @Override
    public List<Entity> fetch(Iterable<Key> iterable, ReadOption... readOptions) {
        return null;
    }

    @Override
    public <T> QueryResults<T> run(Query<T> query, ReadOption... readOptions) {
        return null;
    }

    @Override
    public DatastoreOptions getOptions() {
        return null;
    }

    @Override
    public Entity get(Key key) {
        try {
            com.google.appengine.api.datastore.Entity entity = datastoreService.get(convertGCToGAEKey(key));
            Entity.Builder builder = Entity.newBuilder(key);
            copyGAEToGCEntityProperties(builder, entity);

            return builder.build();
        } catch (EntityNotFoundException e) {
            throw new RuntimeException(e);
        }
    }

    @Override
    public Iterator<Entity> get(Key... keys) {
        return null;
    }

    @Override
    public List<Entity> fetch(Key... keys) {
        List<com.google.appengine.api.datastore.Key> keyList = Arrays.stream(keys)
                .map(this::convertGCToGAEKey)
                .collect(Collectors.toList());

        Collection<com.google.appengine.api.datastore.Entity> values = datastoreService.get(keyList)
                .values();

        return values
                .stream()
                .map(entity -> {
                    Key key = Key.newBuilder(projectId, entity.getKind(), entity.getKey().getName())
                            .build();

                    Entity.Builder builder = Entity.newBuilder(key);
                    copyGAEToGCEntityProperties(builder, entity);

                    return builder.build();
                }).collect(Collectors.toList());
    }

    @Override
    public <T> QueryResults<T> run(Query<T> query) {
        return null;
    }

    private com.google.appengine.api.datastore.Key convertGCToGAEKey(Key key) {
        return com.google.appengine.api.datastore.KeyFactory.createKey(key.getKind(), key.getName());
    }

    private Key convertGAEToGCKey(Key k) {
        return Key.newBuilder(projectId, k.getKind(), k.getName()).build();
    }

    private void copyGAEToGCEntityProperties(Entity.Builder GCEntityBuilder, com.google.appengine.api.datastore.Entity GAEEntity) {
        GAEEntity.getProperties().forEach((k, v) -> {
            if (v instanceof String) {
                GCEntityBuilder.set(k, (String) v);
            } else if (v instanceof Long) {
                GCEntityBuilder.set(k, (Long) v);
            } else if (v instanceof Double) {
                GCEntityBuilder.set(k, (Double) v);
            } else if (v instanceof Boolean) {
                GCEntityBuilder.set(k, (Boolean) v);
            } else {
                throw new UnsupportedOperationException("Not implemented yet");
            }
        });
    }

    private void copyGCToGAEEntityProperties(FullEntity<?> fullEntity, com.google.appengine.api.datastore.Entity entity) {
        fullEntity.getProperties().forEach((k, v) -> entity.setProperty(k, v.get()));
    }
}

But I don't think this is a good way haha. I guess that just accessing the Google Cloud Datastore API is way easier.

… to v1.5.4 (#1328) [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [com.google.cloud:google-cloud-resourcemanager](https://togithub.com/googleapis/java-resourcemanager) | `1.5.3` -> `1.5.4` | [![age](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-resourcemanager/1.5.4/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-resourcemanager/1.5.4/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-resourcemanager/1.5.4/compatibility-slim/1.5.3)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-resourcemanager/1.5.4/confidence-slim/1.5.3)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes <details> <summary>googleapis/java-resourcemanager</summary> ### [`v1.5.4`](https://togithub.com/googleapis/java-resourcemanager/blob/HEAD/CHANGELOG.md#154-httpsgithubcomgoogleapisjava-resourcemanagercomparev153v154-2022-08-31) [Compare Source](https://togithub.com/googleapis/java-resourcemanager/compare/v1.5.3...v1.5.4) ##### Dependencies - update dependency com.google.apis:google-api-services-cloudresourcemanager to v1-rev20220828-2.0.0 ([#829](https://togithub.com/googleapis/java-resourcemanager/issues/829)) ([4821d7a](https://togithub.com/googleapis/java-resourcemanager/commit/4821d7a6ebeffa5b8ee5ff4bfc3be72309800d00)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, click this checkbox. --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/java-asset).

🤖 I have created a release *beep* *boop* --- ## [3.6.0](googleapis/java-asset@v3.5.0...v3.6.0) (2022-09-15) ### Features * Add client library support for AssetService v1 BatchGetEffectiveIamPolicies API ([3919a1d](googleapis/java-asset@3919a1d)) * Add client library support for AssetService v1 BatchGetEffectiveIamPolicies API ([#1300](googleapis/java-asset#1300)) ([3919a1d](googleapis/java-asset@3919a1d)) * Release of query system ([3919a1d](googleapis/java-asset@3919a1d)) ### Dependencies * Update dependency com.google.api.grpc:proto-google-cloud-orgpolicy-v1 to v2.3.2 ([#1302](googleapis/java-asset#1302)) ([d01d900](googleapis/java-asset@d01d900)) * Update dependency com.google.api.grpc:proto-google-cloud-orgpolicy-v1 to v2.3.3 ([#1332](googleapis/java-asset#1332)) ([c1511c2](googleapis/java-asset@c1511c2)) * Update dependency com.google.api.grpc:proto-google-cloud-os-config-v1 to v2.5.2 ([#1309](googleapis/java-asset#1309)) ([cf96ee9](googleapis/java-asset@cf96ee9)) * Update dependency com.google.api.grpc:proto-google-cloud-pubsub-v1 to v1.102.11 ([#1297](googleapis/java-asset#1297)) ([d56eedd](googleapis/java-asset@d56eedd)) * Update dependency com.google.api.grpc:proto-google-cloud-pubsub-v1 to v1.102.12 ([#1316](googleapis/java-asset#1316)) ([a3713fd](googleapis/java-asset@a3713fd)) * Update dependency com.google.api.grpc:proto-google-cloud-pubsub-v1 to v1.102.13 ([#1321](googleapis/java-asset#1321)) ([883b7b8](googleapis/java-asset@883b7b8)) * Update dependency com.google.api.grpc:proto-google-cloud-pubsub-v1 to v1.102.14 ([#1334](googleapis/java-asset#1334)) ([56cb4b4](googleapis/java-asset@56cb4b4)) * Update dependency com.google.api.grpc:proto-google-identity-accesscontextmanager-v1 to v1.4.1 ([#1307](googleapis/java-asset#1307)) ([b90baf7](googleapis/java-asset@b90baf7)) * Update dependency com.google.cloud:google-cloud-bigquery to v2.14.4 ([#1301](googleapis/java-asset#1301)) ([64f7ea5](googleapis/java-asset@64f7ea5)) * Update dependency com.google.cloud:google-cloud-bigquery to v2.14.6 ([#1315](googleapis/java-asset#1315)) ([fa179e2](googleapis/java-asset@fa179e2)) * Update dependency com.google.cloud:google-cloud-bigquery to v2.14.7 ([#1320](googleapis/java-asset#1320)) ([06d1a16](googleapis/java-asset@06d1a16)) * Update dependency com.google.cloud:google-cloud-bigquery to v2.15.0 ([#1326](googleapis/java-asset#1326)) ([df36595](googleapis/java-asset@df36595)) * Update dependency com.google.cloud:google-cloud-bigquery to v2.16.0 ([#1336](googleapis/java-asset#1336)) ([870779d](googleapis/java-asset@870779d)) * Update dependency com.google.cloud:google-cloud-core to v2.8.10 ([#1323](googleapis/java-asset#1323)) ([09e03b8](googleapis/java-asset@09e03b8)) * Update dependency com.google.cloud:google-cloud-core to v2.8.11 ([#1331](googleapis/java-asset#1331)) ([a112cec](googleapis/java-asset@a112cec)) * Update dependency com.google.cloud:google-cloud-core to v2.8.9 ([#1314](googleapis/java-asset#1314)) ([8edc2b8](googleapis/java-asset@8edc2b8)) * Update dependency com.google.cloud:google-cloud-pubsub to v1.120.11 ([#1298](googleapis/java-asset#1298)) ([172b34b](googleapis/java-asset@172b34b)) * Update dependency com.google.cloud:google-cloud-pubsub to v1.120.12 ([#1317](googleapis/java-asset#1317)) ([1ea636f](googleapis/java-asset@1ea636f)) * Update dependency com.google.cloud:google-cloud-pubsub to v1.120.13 ([#1322](googleapis/java-asset#1322)) ([b7522b9](googleapis/java-asset@b7522b9)) * Update dependency com.google.cloud:google-cloud-pubsub to v1.120.14 ([#1335](googleapis/java-asset#1335)) ([e9142b4](googleapis/java-asset@e9142b4)) * Update dependency com.google.cloud:google-cloud-resourcemanager to v1.5.3 ([#1318](googleapis/java-asset#1318)) ([aa3a1bb](googleapis/java-asset@aa3a1bb)) * Update dependency com.google.cloud:google-cloud-resourcemanager to v1.5.4 ([#1328](googleapis/java-asset#1328)) ([fca0ce5](googleapis/java-asset@fca0ce5)) * Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.0.2 ([#1330](googleapis/java-asset#1330)) ([ccb704c](googleapis/java-asset@ccb704c)) * Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.0.3 ([#1340](googleapis/java-asset#1340)) ([0d87a9d](googleapis/java-asset@0d87a9d)) * Update dependency com.google.cloud:google-cloud-storage to v2.11.3 ([#1299](googleapis/java-asset#1299)) ([d59e6c6](googleapis/java-asset@d59e6c6)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

mitchhentges closed this as completed Oct 21, 2016

konrad-garus mentioned this issue Mar 21, 2018

Using Objectify in local dev server objectify/objectify#361

Closed

JustinBeckwith assigned mitchhentges Feb 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Client Datastore API with App Engine Devserver #1328

Client Datastore API with App Engine Devserver #1328

mitchhentges commented Oct 21, 2016

mziccard commented Oct 21, 2016

mitchhentges commented Oct 21, 2016 •

edited

mziccard commented Oct 21, 2016

mitchhentges commented Oct 21, 2016 •

edited

mziccard commented Oct 21, 2016

mitchhentges commented Oct 21, 2016

josenicomaia commented Mar 20, 2022 •

edited

Client Datastore API with App Engine Devserver #1328

Client Datastore API with App Engine Devserver #1328

Comments

mitchhentges commented Oct 21, 2016

mziccard commented Oct 21, 2016

mitchhentges commented Oct 21, 2016 • edited

mziccard commented Oct 21, 2016

mitchhentges commented Oct 21, 2016 • edited

mziccard commented Oct 21, 2016

mitchhentges commented Oct 21, 2016

josenicomaia commented Mar 20, 2022 • edited

mitchhentges commented Oct 21, 2016 •

edited

mitchhentges commented Oct 21, 2016 •

edited

josenicomaia commented Mar 20, 2022 •

edited