PerUserTable.deleteAllWaves needs to delete in chunks

[GoogleCodeExporter]

Enhancement

If Walkaround is installed in AppEngine, data is stored on Google's High 
Replication Storage. However, some code changes make this data invalid. (I had 
this once). Clearing the data store would solve this problem.

AppEngine does not provide a means to clear the data store. (It does allow you 
to delete 20 records at a time, but this takes a long time as I have 1500 waves 
I might want to import). 

Would it be possible to add a servlet (?) to clear the entire data store? (Like 
/import, but then /cleardatastore with a simple button: clear the data store). 
I think this doesn't require much effort. (Some sites suggest a simple 
statement looking something like: "db.delete(db.Query())".

Thanks,
Guillaume Hanique

Original issue reported on code.google.com by guillaum...@gmail.com on 17 Jan 2012 at 10:45

[GoogleCodeExporter]

If you want to delete all your data:
In App Engine's Admin Console, go to Application Settings and enable the 
Datastore Admin built-in.  A Datastore Admin link should appear in the 
left-side navigation panel.  Select all entity kinds and click "Delete 
Entities".  (There should be no reason to delete AccountRecord2 and Secret, but 
it's not too harmful to delete them either.)

To delete attachment data, you'll have to go to the Blob Viewer and delete them 
manually.  Hopefully you don't have too many.

Don't use the system while the deletion is in progress.

But I am curious: What code changes made your data invalid?  What errors are 
you seeing?

Original comment by oh...@google.com on 17 Jan 2012 at 10:13

[GoogleCodeExporter]

Hi,

Thanks for the suggestion. Very interesting. The Datastore admin doesn't
work in Chrome! It does work in Firefox, however.

I'm not sure exactly when or how data got corrupt. I had already downloaded
the list of waves I could import, then updated Walkaround. When I returned
to /import I got 500-errors. I checked my quota, but had not run out. (I
myself did not make any code changes in the strict sense of the word).

As I already deleted everything manually, I can't reproduce the error.
Sorry about that.

Original comment by guillaum...@gmail.com on 19 Jan 2012 at 6:22

[GoogleCodeExporter]

It's possible that I made an incompatible change to the data format, but I 
didn't mean to, so I'd be curious to understand what happened here.

If you have the time, please see if you can find the log messages; in the Admin 
Console, can you go to Logs, click "Options", enter Filter: /import, select 
Logs with minimum severity: Error, then "Search", skip through all the 
/taskqueue/import entries (I don't know how to exclude them).  Expand a failed 
/import and look for the error messages and stack trace.

Original comment by oh...@google.com on 19 Jan 2012 at 7:56

[GoogleCodeExporter]

OK, thanks.

There are several kinds of errors in your logs:
(1) a bad response from the Robot API that the code works around by retrying;
(2) a timeout trying to delete the wave import information;
(3) some weird NoClassDefFound exception that I don't know what to do with
(4) a DatastoreNeedIndexException.

(1) and (3) happened on /taskqueue/import, (2) and (4) on /import.

(1) should not be a user-visible problem.  (2) is a TODO in PerUserTable 
http://code.google.com/p/walkaround/source/browse/src/com/google/walkaround/wave
/server/googleimport/PerUserTable.java#306 .

(4) is, I think, what you saw.  I updated the instructions on 
http://code.google.com/p/walkaround/wiki/RunningTheCode to include a step to 
wait until all indexes are serving.

From what I can tell, the problem would have resolved itself if you had waited 
a few minutes (perhaps tens of minutes) for the indexes to get ready, then 
tried to load the page again.  I don't see anything that points to a data 
corruption or incompatible change.  After how much waiting and retrying did you 
decide to delete your data?

Sorry about the incomplete instructions, I hope they are better now.


What remains to do here is fixing (2).

I'm ignoring (3) which doesn't make sense to me and is hopefully a fluke:
com.google.walkaround.wave.server.servlet.ServerExceptionFilter sendError: 
NoClassDefFoundError; sending 500: Internal server error
java.lang.NoClassDefFoundError: javax.net.ssl.SSLException is a restricted 
class. Please see the Google App Engine developer's guide for more details.
    at com.google.apphosting.runtime.security.shared.stub.javax.net.ssl.SSLException.<clinit>(SSLException.java)
    at com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationException(URLFetchServiceImpl.java:119)
    at com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchServiceImpl.java:42)
    at com.google.walkaround.wave.server.auth.OAuthedFetchService.fetch1(OAuthedFetchService.java:95)
    at com.google.walkaround.wave.server.auth.OAuthedFetchService.fetch(OAuthedFetchService.java:112)
    at com.google.walkaround.wave.server.googleimport.RobotApi.callRobotApi(RobotApi.java:189)
    at com.google.walkaround.wave.server.googleimport.RobotApi.getWaveView(RobotApi.java:254)
    at com.google.walkaround.wave.server.googleimport.FindRemoteWavesProcessor.expandPrivateReplies(FindRemoteWavesProcessor.java:245)
    at com.google.walkaround.wave.server.googleimport.FindRemoteWavesProcessor.findWaves(FindRemoteWavesProcessor.java:267)
    at com.google.walkaround.wave.server.googleimport.TaskDispatcher.processTask(TaskDispatcher.java:142)
    at com.google.walkaround.wave.server.googleimport.ImportTaskHandler.handleTask(ImportTaskHandler.java:106)
    at com.google.walkaround.wave.server.googleimport.ImportTaskHandler.access$000(ImportTaskHandler.java:48)
    at com.google.walkaround.wave.server.googleimport.ImportTaskHandler$2.run(ImportTaskHandler.java:137)
    at com.google.walkaround.wave.server.auth.ServletAuthHelper.serve(ServletAuthHelper.java:184)
    at com.google.walkaround.wave.server.googleimport.ImportTaskHandler.doPost(ImportTaskHandler.java:134)
    at com.google.walkaround.util.server.servlet.HandlerServlet.doPost(HandlerServlet.java:91)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)

Original comment by oh...@google.com on 19 Jan 2012 at 10:26

• Changed title: PerUserTable.deleteAllWaves needs to delete in chunks
• Changed state: Accepted

[GoogleCodeExporter]

I think I tried a lot of stuff for about 40 minutes, after which I finally
gave up and started deleting the store. (This took several days, deleting
100 records at a time). Then I retried to refresh the wave listed, which
worked fine.

Original comment by guillaum...@gmail.com on 20 Jan 2012 at 10:18

[GoogleCodeExporter]

I think it's possible that building the new indices took more than 40 minutes, 
depending on how much data there was in your datastore.  I apologize that you 
had to go through all this trouble, and I hope that the new instructions on how 
to update indices before deploying a new version will prevent this in the 
future.

Assuming that you saw errors only on /import, not when viewing or editing 
waves, I don't think there's any data corruption problem here -- even if I 
inadvertently changed the import data format incompatibly, that's not a real 
problem, since importing is still experimental.

Original comment by oh...@google.com on 20 Jan 2012 at 7:41

[GoogleCodeExporter]

No problem! I appreciate your effort and like to help.

Original comment by guillaum...@gmail.com on 20 Jan 2012 at 7:55

[GoogleCodeExporter]

This was import-related and is no longer relevant.

Original comment by oh...@google.com on 29 Jun 2012 at 12:33

• Changed state: WontFix