Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Datastore - Failed to load from the backing store - java.lang.ClassNotFoundException: com.google.cloud.datastore.emulator.impl.LocalDatastoreFileStub #4661

Closed
andrewcampey opened this issue Mar 8, 2019 · 5 comments
Closed

Datastore - Failed to load from the backing store - java.lang.ClassNotFoundException: com.google.cloud.datastore.emulator.impl.LocalDatastoreFileStub #4661

andrewcampey opened this issue Mar 8, 2019 · 5 comments
Assignees
@ajaaym
Labels
api: datastore Issues related to the Datastore API. type: question Request for information or clarification. Not an issue.

Comments

@andrewcampey
Copy link

andrewcampey commented Mar 8, 2019 •
edited

We are hitting a brick wall with trying to use our local datastore.
We can export successfully and the import creates a local_db.bin that LOOKS correct however when we try and connect to it we hit the error mentioned below.

If we stop and re-run our local environment it will blow this file away and start again.

Environment details

  1. We are trying to run local datastore using the emulator
  2. Windows 10
  3. Java version: java version "1.8.0_201"
  4. google api version: 1.25.0

Steps to reproduce

  1. Export datastore from the cloud
  2. Import to local datastore (Creates a local_db.bin file)
  3. Run App engine locally. Error is thrown and no data is visible.

Code example

POWERSHELL SCRIPTS
Export

$GOOGLE_CLOUD_BUCKET = 'BUCKETNAME'
gcloud datastore export --namespaces="(default)" gs://$GOOGLE_CLOUD_BUCKET

Import

# constants
$GOOGLE_CLOUD_BUCKET = 'BUCKETNAME'
$GOOGLE_CLOUD_EXPORT_FOLDER = '2019-03-07T05:54:26_51880'
$GOOGLE_CLOUD_EXPORT_METADATA = '2019-03-07T05:54:26_51880.overall_export_metadata'
$LOCAL_DATASTORE_HOSTNAME = 'localhost'
$LOCAL_DATASTORE_PORT = '8081'
$LOCAL_TEMP_DIRECTORY = 'C:/TEMP/datastore_import'
$LOCAL_TEMP_METADATA_FILE = 'overall_export_metadata'
$PROJECT_ID = 'PROJECT_ID'

# create base bucket uri
$bucketUri = 'gs://' + $GOOGLE_CLOUD_BUCKET + '/' + $GOOGLE_CLOUD_EXPORT_FOLDER

# download meta data file
$metaDataUri = $bucketUri + '/' + $GOOGLE_CLOUD_EXPORT_METADATA
$metaDataFileDestination = $LOCAL_TEMP_DIRECTORY + '/' + $LOCAL_TEMP_METADATA_FILE

gsutil cp $metaDataUri $metaDataFileDestination

# check to see if we need to create the 'default namespace folder'
$newFolderPath = $LOCAL_TEMP_DIRECTORY + '/default_namespace'

if (!(Test-Path -Path $newFolderPath)) {
  new-item -Path $LOCAL_TEMP_DIRECTORY -Name 'default_namespace' -ItemType directory
}

# download default namespace folder
$defaultNamespaceFolderUri = $bucketUri + '/default_namespace'

gsutil -m cp -r $defaultNamespaceFolderUri $LOCAL_TEMP_DIRECTORY

# create local data store uri for use in curl command
$localDatastoreUri = 'http://' + $LOCAL_DATASTORE_HOSTNAME + ':' + $LOCAL_DATASTORE_PORT + '/v1/projects/' + $PROJECT_ID + ':import'

# create import uri for use in curl command
$importFilesPath = $LOCAL_TEMP_DIRECTORY + '/' + $LOCAL_TEMP_METADATA_FILE

# set tls to 1.2
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

# execute request to update data
$body = '{"input_url":"' + $importFilesPath + '"}'
Invoke-WebRequest -Method Post -ContentType 'application/json' -Uri $localDatastoreUri -Body $body

Stack trace

Mar 07, 2019 11:42:28 PM com.google.appengine.api.datastore.dev.LocalDatastoreService load
INFO: Failed to load from the backing store, C:\Users\User\AppData\Roaming\gcloud\emulators\datastore\WEB-INF\appengine-generated\local_db.bin
java.lang.ClassNotFoundException: com.google.cloud.datastore.emulator.impl.LocalDatastoreFileStub$FileProfile
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at com.google.appengine.tools.development.DevAppServerClassLoader.loadClass(DevAppServerClassLoader.java:108)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:686)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1868)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at java.util.HashMap.readObject(HashMap.java:1412)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1170)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2178)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at com.google.appengine.api.datastore.dev.LocalDatastoreService.load(LocalDatastoreService.java:1883)
	at com.google.appengine.api.datastore.dev.LocalDatastoreService.startInternal(LocalDatastoreService.java:637)
	at com.google.appengine.api.datastore.dev.LocalDatastoreService.access$000(LocalDatastoreService.java:145)
	at com.google.appengine.api.datastore.dev.LocalDatastoreService$2.run(LocalDatastoreService.java:627)
	at java.security.AccessController.doPrivileged(Native Method)
	at com.google.appengine.api.datastore.dev.LocalDatastoreService.start(LocalDatastoreService.java:623)
	at com.google.appengine.tools.development.ApiProxyLocalImpl.startServices(ApiProxyLocalImpl.java:799)
	at com.google.appengine.tools.development.ApiProxyLocalImpl.access$700(ApiProxyLocalImpl.java:53)
	at com.google.appengine.tools.development.ApiProxyLocalImpl$2.run(ApiProxyLocalImpl.java:779)
	at com.google.appengine.tools.development.ApiProxyLocalImpl$2.run(ApiProxyLocalImpl.java:776)
	at java.security.AccessController.doPrivileged(Native Method)
	at com.google.appengine.tools.development.ApiProxyLocalImpl.getService(ApiProxyLocalImpl.java:775)
	at com.google.apphosting.utils.servlet.DatastoreViewerServlet.init(DatastoreViewerServlet.java:124)
	at javax.servlet.GenericServlet.init(GenericServlet.java:244)
	at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:672)
	at org.eclipse.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:521)
	at org.eclipse.jetty.servlet.ServletHolder.prepare(ServletHolder.java:805)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:537)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
	at com.google.appengine.tools.development.jetty9.DevAppEngineWebAppContext.doScope(DevAppEngineWebAppContext.java:94)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at com.google.appengine.tools.development.jetty9.JettyContainerService$ApiProxyHandler.handle(JettyContainerService.java:595)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:502)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
	at java.lang.Thread.run(Thread.java:748)
@yoshi-automation yoshi-automation added the triage me I really want to be triaged. label Mar 8, 2019
@sduskis sduskis added type: question Request for information or clarification. Not an issue. and removed triage me I really want to be triaged. labels Mar 8, 2019
@kolea2 kolea2 added the api: datastore Issues related to the Datastore API. label Mar 8, 2019
@andrewcampey
Copy link
Author

Hi @ajaaym,
Just wondering if:-

  1. You've confirmed that this is an issue
  2. Any advice on what we should do?

@chalcrow
Copy link

chalcrow commented Apr 2, 2019 •
edited

@ajaaym - I'm a colleague of @andrewcampey. Full details of the process we're currently following and the location that we're using for the local_db.bin file is documented in my StackOverflow answer here - https://stackoverflow.com/questions/55037520/app-engine-localdatastoreservice-load-failed-to-load-from-the-backing-store/55465569#55465569

Also, we currently use Firebase for hosting our primary web application. I'm not sure if our setup is utilising 'Cloud Firestore in Datastore mode' though (as described here - https://cloud.google.com/datastore/docs/activate#google_cloud_datastore_for_an_existing_app_engine_application)

@ajaaym
Copy link
Contributor

ajaaym commented Apr 2, 2019

@chalcrow I am not able to find the required class LocalDatastoreFileStub$FileProfile any where. Still working on finding out a process to use datastore locally.

@chalcrow
Copy link

chalcrow commented Apr 2, 2019 •
edited

Thanks @ajaaym - I updated to the following dependencies in our pom.xml - same error occurs:

        <dependency>
            <groupId>com.google.appengine</groupId>
            <artifactId>appengine-api-1.0-sdk</artifactId>
            <version>1.9.73</version>
        </dependency>

        <dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>google-cloud-datastore</artifactId>
            <version>1.67.0</version>
        </dependency>

Here's our full pom.xml Note that it contains the following dependency (with test scope only) - however if I remove this, clean and rebuild we still have the same error:

    <dependency>
        <groupId>com.google.appengine</groupId>
        <artifactId>appengine-api-stubs</artifactId>
        <version>1.9.71</version>
        <scope>test</scope>
    </dependency>

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>
    <packaging>war</packaging>
    <version>1.0-SNAPSHOT</version>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>com.google.cloud</groupId>
                <artifactId>google-cloud-bom</artifactId>
                <version>0.83.0-alpha</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <groupId>group_id</groupId>
    <artifactId>endpoints</artifactId>

    <properties>
        <maven.compiler.target>1.8</maven.compiler.target>
        <maven.compiler.source>1.8</maven.compiler.source>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <endpoints.framework.version>2.0.13</endpoints.framework.version>
        <endpoints.management.version>1.0.4</endpoints.management.version>
        <endpoints.project.id>project_id</endpoints.project.id>
        <endpoints.serviceName>service_name</endpoints.serviceName>
    </properties>

    <dependencies>
        <!-- Compile/runtime dependencies -->
    	<dependency>
		  <groupId>group_id</groupId>
		  <artifactId>artifact</artifactId>
		  <version>0.0.1-SNAPSHOT</version>
		</dependency>

        <dependency>
            <groupId>com.auth0</groupId>
            <artifactId>java-jwt</artifactId>
            <version>3.4.0</version>
        </dependency>

        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-core</artifactId>
            <version>2.9.6</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-annotations</artifactId>
            <version>2.9.6</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.9.8</version>
        </dependency>

        <dependency>
            <groupId>com.google.apis</groupId>
            <artifactId>google-api-services-bigquery</artifactId>
            <version>v2-rev410-1.25.0</version>
        </dependency>
        <dependency>
            <groupId>com.google.apis</groupId>
            <artifactId>google-api-services-cloudiot</artifactId>
            <version>v1-rev53-1.25.0</version>
        </dependency>

        <dependency>
            <groupId>com.google.appengine</groupId>
            <artifactId>appengine-api-1.0-sdk</artifactId>
            <version>1.9.73</version>
        </dependency>

        <dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>google-cloud-bigquery</artifactId>
        </dependency>
        <dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>google-cloud-datastore</artifactId>
            <version>1.67.0</version>
        </dependency>
    	<dependency>
		  <groupId>com.google.cloud</groupId>
		  <artifactId>google-cloud-pubsub</artifactId>
		</dependency>

        <dependency>
            <groupId>com.google.endpoints</groupId>
            <artifactId>endpoints-framework</artifactId>
            <version>2.0.14</version>
        </dependency>
        <dependency>
		    <groupId>com.google.endpoints</groupId>
		    <artifactId>endpoints-framework-auth</artifactId>
		    <version>1.0.7</version>
		</dependency>
        <dependency>
            <groupId>com.google.endpoints</groupId>
            <artifactId>endpoints-framework-guice</artifactId>
            <version>2.0.13</version>
        </dependency>
		<dependency>
            <groupId>com.google.endpoints</groupId>
            <artifactId>endpoints-management-control-appengine</artifactId>
            <version>1.0.7</version>
        </dependency>

        <dependency>
            <groupId>com.google.firebase</groupId>
            <artifactId>firebase-admin</artifactId>
            <version>6.2.0</version>
        </dependency>

        <!-- Test Dependencies -->
        <dependency>
            <groupId>com.google.appengine</groupId>
            <artifactId>appengine-testing</artifactId>
            <version>1.9.71</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>com.google.appengine</groupId>
            <artifactId>appengine-api-stubs</artifactId>
            <version>1.9.71</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>com.google.appengine</groupId>
            <artifactId>appengine-tools-sdk</artifactId>
            <version>1.9.71</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>com.google.truth</groupId>
            <artifactId>truth</artifactId>
            <version>0.33</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.mockito</groupId>
            <artifactId>mockito-all</artifactId>
            <version>1.10.19</version>
            <scope>test</scope>
        </dependency>

    </dependencies>

    <build>
        <!-- for hot reload of the web application-->
        <outputDirectory>${project.build.directory}/${project.build.finalName}/WEB-INF/classes</outputDirectory>
        <plugins>           
            <plugin>
                <groupId>com.google.cloud.tools</groupId>
                <artifactId>appengine-maven-plugin</artifactId>
                <version>1.3.2</version>
                <configuration>                    
                </configuration>
            </plugin>
            <plugin>
                <groupId>com.google.cloud.tools</groupId>
                <artifactId>endpoints-framework-maven-plugin</artifactId>
                <version>2.0.0</version>
                <configuration>
                    <!-- plugin configuration -->                    
                    <hostname>endpoints-dot-gator-acoem-cloud-eu.appspot.com</hostname>                     
                    <basePath>/api</basePath>
                    <serviceClasses>
                        <serviceClass>...</serviceClass>
                        ...
                    </serviceClasses>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>versions-maven-plugin</artifactId>
                <version>2.1</version>
                <executions>
                    <execution>
                        <phase>compile</phase>
                        <goals>
                            <goal>display-dependency-updates</goal>
                            <goal>display-plugin-updates</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
        <pluginManagement>
        	<plugins>
        		<!--This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself.-->
        		<plugin>
        			<groupId>org.eclipse.m2e</groupId>
        			<artifactId>lifecycle-mapping</artifactId>
        			<version>1.0.0</version>
        			<configuration>
        				<lifecycleMappingMetadata>
        					<pluginExecutions>
        						<pluginExecution>
        							<pluginExecutionFilter>
        								<groupId>
        									org.codehaus.mojo
        								</groupId>
        								<artifactId>
        									versions-maven-plugin
        								</artifactId>
        								<versionRange>
        									[2.1,)
        								</versionRange>
        								<goals>
        									<goal>
        										display-plugin-updates
        									</goal>
        									<goal>
        										display-dependency-updates
        									</goal>
        								</goals>
        							</pluginExecutionFilter>
        							<action>
                                        <ignore/>
        							</action>
        						</pluginExecution>
        					</pluginExecutions>
        				</lifecycleMappingMetadata>
        			</configuration>
        		</plugin>
        	</plugins>
        </pluginManagement>
    </build>
</project>

@chalcrow chalcrow mentioned this issue Apr 2, 2019
Closed
@crwilcox
Copy link

crwilcox commented Jun 7, 2019

@chalcrow @andrewcampey , I am sorry for your issue. Though this doesn't seem to involve the Java Client Library. This issue seems to be related to datastore and the emulator.

Could you please file your issue on the Cloud Datastore issue tracker: https://issuetracker.google.com/issues/new?component=187197&template=1162794

@sduskis sduskis closed this as completed Jun 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ajaaym ajaaym

Labels
api: datastore Issues related to the Datastore API. type: question Request for information or clarification. Not an issue.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@sduskis @crwilcox @andrewcampey @chalcrow @ajaaym @yoshi-automation @kolea2