A bulk uploader could potentially upload data for non existent survey

[iperdomo]

This scenario leads to a java.lang.NullPointerException and makes the task failed that needs to be rescheduled.

2013-05-23 21:51:58.964
org.waterforpeople.mapping.app.web.TaskServlet ingestFile: Received Task Queue calls for surveyInstanceKey: 183003
W 2013-05-23 21:51:58.964
com.gallatinsystems.framework.dao.BaseDAO getByKey: No com.gallatinsystems.survey.domain.Survey found with id: 1309064
I 2013-05-23 21:51:59.060
com.gallatinsystems.survey.dao.SurveyUtils notifyReportService: Sending notification (invalidate) for surveys: [1309064]
I 2013-05-23 21:51:59.165
com.gallatinsystems.survey.dao.SurveyUtils notifyReportService: Response from server: OK
E 2013-05-23 21:51:59.165
com.gallatinsystems.framework.rest.AbstractRestApiServlet executeRequest: Could not execute rest request
java.lang.NullPointerException
    at org.waterforpeople.mapping.app.web.TaskServlet.ingestFile(TaskServlet.java:502)
    at org.waterforpeople.mapping.app.web.TaskServlet.handleRequest(TaskServlet.java:411)
    at com.gallatinsystems.framework.rest.AbstractRestApiServlet.executeRequest(AbstractRestApiServlet.java:90)
    at com.gallatinsystems.framework.rest.AbstractRestApiServlet.doPost(AbstractRestApiServlet.java:65)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

[mtwestra]

What happened is that a survey was created, used for training, and then deleted from the dashboard, including all data. After that, data from the phone that was used to collect the data was resubmitted using bulk upload. This leads to data being submitted for which there is no corresponding survey.

[mtwestra]

code reviewed & correct

[caetie]

on the UI it appears that I can upload data zips for a survey that doesn't exist on sandbox, but in the web console I also see "syntax error" after each POST

@iperdomo can you clarify this?

[iperdomo]

@caetie yes, the only way to analyze if the survey definition exists is by uploading and processing the file.
The logs should show some related error like the one mentioned.

com.gallatinsystems.framework.dao.BaseDAO getByKey: No com.gallatinsystems.survey.domain.Survey found with id: 1309064

Previously this lead to a NullPointerException (NPE) and kept the task running forever. What we did is just to avoid this error.

[iperdomo]

@caetie if you found another (but related) error, we should log an issue.

Checking the logs in flowdev1, i see the following:

org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: Error while expanding /tmp/akvo/flow/uploads/847-wfp362793273978zip/wfp36279327
java.util.zip.ZipException: archive is not a ZIP archive]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:224)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
Caused by: Error while expanding /tmp/akvo/flow/uploads/847-wfp362793273978zip/wfp362793273978.zip
java.util.zip.ZipException: archive is not a ZIP archive
        at org.apache.ant.compress.taskdefs.Unzip.expandFile(Unzip.java:88)
        at org.apache.tools.ant.taskdefs.Expand.execute(Expand.java:132)
        at akvo.flow_services.uploader$unzip_file.invoke(uploader.clj:60)
        at akvo.flow_services.uploader$bulk_upload.invoke(uploader.clj:85)
        at akvo.flow_services.scheduler.BulkUploadJob.execute(scheduler.clj:54)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
        ... 1 more
Caused by: java.util.zip.ZipException: archive is not a ZIP archive
        at org.apache.commons.compress.archivers.zip.ZipFile.positionAtCentralDirectory32(ZipFile.java:717)
        at org.apache.commons.compress.archivers.zip.ZipFile.positionAtCentralDirectory(ZipFile.java:672)
        at org.apache.commons.compress.archivers.zip.ZipFile.populateFromCentralDirectory(ZipFile.java:406)
        at org.apache.commons.compress.archivers.zip.ZipFile.<init>(ZipFile.java:206)
        at org.apache.ant.compress.taskdefs.Unzip.expandFile(Unzip.java:62)
        ... 6 more

This means that you tried to upload a single zip file? The bulk uploader expects a collection of files, combined in a single zip file, e.g. the whole surveyal folder.

[caetie]

I dragged in multiple survey zips to the Bulk Upload window I knew had no matching survey on sandbox. They were all .zips. This is how we designed this feature, to handle one or more .zip or .jpg files.