Job submission followed by quick get job details fails with 404

[noorul]

From gitter channel.

Björn Lohrmann @bjoernlohrmann 03:06

my apologies for complaining all the time I am currently doing a smoke-test on the jobserver and I have run into an issue. I am getting the following behavior on a freshly started jobserver (vm-per-context is false):

 $ curl --data-binary @job-server-tests/target/scala-2.10/job-server-tests_2.10-0.7.0-SNAPSHOT.jar localhost:8090/jars/test
 OK
 $ curl -d "" 'localhost:8090/contexts/test-context'
 OK
 $ curl -d "input.string = a b c a b see" 'localhost:8090/jobs appName=test&classPath=spark.jobserver.WordCountExample&context=test-context'
 {
   "duration": "Job not done yet",
   "classPath": "spark.jobserver.WordCountExample",
   "startTime": "2016-06-22T17:29:48.850-04:00",
   "context": "test-context",
   "status": "STARTED",
   "jobId": "725d1dd0-a49f-46d5-b68b-f2f7a77b0617"
 }
 $ curl localhost:8090/jobs
 (this just hangs until I Crtl-C out of it)
 $ curl localhost:8090/jobs/725d1dd0-a49f-46d5-b68b-f2f7a77b0617
 {
   "status": "ERROR",
   "result": "No such job ID 725d1dd0-a49f-46d5-b68b-f2f7a77b0617"
 }

Jobserver log contains the following suspicious looking line:

[2016-06-22 17:29:48,914] INFO r$RemoteDeadLetterActorRef [] [akka://JobServer/deadLetters] - Message [spark.jobserver.JobInfoActor$JobConfigStored$] from Actor[akka://JobServer/user/job-info#1380245094] to Actor[akka://JobServer/deadLetters] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

@noorul is also facing similar issues with jvm-per-context set to tru

[bjoernlohrmann]

note: even if I wait a while (tested with 30secs) before doing

$ curl localhost:8090/jobs/725d1dd0-a49f-46d5-b68b-f2f7a77b0617

I still get a 404. My config is as follows:

spark {
  jobserver {
    port = 8090
    jar-store-rootdir = /tmp/spark-jobserver/jars
    jobdao = spark.jobserver.io.JobFileDAO
    filedao {
      rootdir = /tmp/spark-jobserver/filedao/data
    }
    datadao {
      rootdir = /tmp/spark-jobserver/upload
    }
    context-per-jvm = false
  }

  master = "local[4]"
  context-settings {
    num-cpu-cores = 2
    memory-per-node = 1G
    passthrough {
    }

[velvia]

Have you tried clearing the /tmp/spark-jobserver/filedao/data directory? In the past this has worked for people. It may be corrupted.

On Jun 23, 2016, at 1:01 AM, Björn Lohrmann notifications@github.com wrote:

note: even if I wait a while (tested with 30secs) before doing

$ curl localhost:8090/jobs/725d1dd0-a49f-46d5-b68b-f2f7a77b0617

it does not work. My config is as follows:
`
spark {
jobserver {
port = 8090
jar-store-rootdir = /tmp/spark-jobserver/jars
jobdao = spark.jobserver.io.JobFileDAO
filedao {
rootdir = /tmp/spark-jobserver/filedao/data
}
datadao {
rootdir = /tmp/spark-jobserver/upload
}
context-per-jvm = false
}

master = "local[4]" # Run Spark locally with 4 worker threads
context-settings {
num-cpu-cores = 2
memory-per-node = 1G
passthrough {
}
`

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub #516 (comment), or mute the thread https://github.com/notifications/unsubscribe/ABA32zd7wvPkjvmdV3kfSBah05s3WKjXks5qOj1XgaJpZM4I8cWP.

[bjoernlohrmann]

Yes, I deleted /tmp/spark-jobserver before starting the jobserver.

[noorul]

@bjoernlohrmann Which version of SJS are you using?

[bjoernlohrmann]

I was using the current master at the time.

[noorul]

@bjoernlohrmann A fix 485cb6a related to this went in. Can you see whether this is reproducible now?

[bjoernlohrmann]

I tested a build created from todays master d5190aa which worked (jobserver in local mode, context-per-jvm=false).

[noorul]

closing this.