Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Make the upload servlet handle slightly alrger documents.

Our complete history of poll votes is about 300k, but the FileUploadServlet starts using files in /tmp if we are bigger than 10k.
Since AppEngine doesn't support local file systems, we make the inmemory filesize bigger.
Also don't turn it into a list in memory, so we can theoretically deal with it in 50-line chunks.
  • Loading branch information...
commit 68de48d29553b057914684c4c9b27a43c73c963b 1 parent c7a9e21
@bruntonspall bruntonspall authored
Showing with 7 additions and 2 deletions.
  1. +7 −2 src/main/scala/com/gu/polls/servlets/UploadServlet.scala
View
9 src/main/scala/com/gu/polls/servlets/UploadServlet.scala
@@ -10,10 +10,15 @@ import com.google.appengine.api.taskqueue.QueueFactory
import com.google.appengine.api.taskqueue.TaskOptions.Builder._
import cc.spray.json._
import org.apache.commons.fileupload.FileItemFactory
+import org.apache.commons.fileupload.disk.DiskFileItemFactory
case class PollLine(pollId: Long, questionId: Long, questionTotal: Long, answerId: Long, answerTotal: Long)
class UploadServlet extends ScalatraServlet with FileUploadSupport {
+ // Our import is a couple of hundred k, support large in-memory files
+ // We know that this import is done rarely, so this should be fine.
+ override lazy val fileItemFactory = new DiskFileItemFactory(512 * 1024, null)
+
import DefaultJsonProtocol._
implicit val pollLineFormat = jsonFormat5(PollLine.apply)
@@ -23,8 +28,8 @@ class UploadServlet extends ScalatraServlet with FileUploadSupport {
val polls = Source.fromInputStream(fileParams("thefile").getInputStream)
.getLines()
.map { _.split(",") }
- .map { line => PollLine(line(0).toLong, line(1).toLong, line(2).toLong, line(3).toLong, line(4).toLong).toJson }.toList
- polls.grouped(10).foreach { pollList =>
+ .map { line => PollLine(line(0).toLong, line(1).toLong, line(2).toLong, line(3).toLong, line(4).toLong).toJson }
+ polls.grouped(50).foreach { pollList =>
QueueFactory.getDefaultQueue.add(
withUrl("/data/queue")
.param("polls", pollList.toJson.compactPrint)
Please sign in to comment.
Something went wrong with that request. Please try again.