Skip to content
This repository has been archived by the owner on Jan 8, 2024. It is now read-only.

Incomplete deployment of Periodic Nomad Jobspec jobs #2982

Closed
acaloiaro opened this issue Feb 4, 2022 · 4 comments · Fixed by #3963
Closed

Incomplete deployment of Periodic Nomad Jobspec jobs #2982

acaloiaro opened this issue Feb 4, 2022 · 4 comments · Fixed by #3963
Assignees
@paladin-devops
Labels
enhancement New feature or request plugin/nomad-jobspec

Comments

@acaloiaro
Copy link
Contributor

acaloiaro commented Feb 4, 2022 •
edited

Describe the bug
When deploying periodic Nomad Jobspec jobs with Waypoint, the status of the deployment indicates an error when the Nomad job is successfully scheduled.

The deploy command fails with

! No evaluation with id "" found

Steps to Reproduce

Full repro repo: https://github.com/acaloiaro/example-waypoint-nomad-jobspec-failure

waypoint up with the following files in the local directory

waypoint.hcl

project = "example-periodic-failure"

app "main" {
  build {
    use "docker" {}
  }

  deploy {
    use "nomad-jobspec" {
      jobspec = templatefile("${path.app}/example-periodic-failure.nomad.hcl")
    }
  }
}

Dockerfile

FROM alpine

CMD = [ "echo", "dummy dockerfile" ]

example-periodic-failure.nomad.hcl

job "example-periodic-failure" {

  datacenters = ["dc1"]
  type = "batch"

  periodic {
    cron             = "* * * * * *"
  }

  task "main" {
    driver = "exec"

    config {
      command = "echo"
      args    = ["hello world"]
    }

    resources {
      memory = 10
    }
  }
}

Expected behavior
I expect waypoint to indicate the status of the Nomad job, which is success.

Waypoint Platform Versions
Additional version and platform information to help triage the issue if
applicable:

  • Waypoint CLI Version: CLI: v0.7.1 (062857d)
    Server: v0.7.1
  • Waypoint Server Platform and Version: nomad
  • Waypoint Plugin: none

Additional context

Confirm the job runs in nomad after being deployed with waypoint:

$ nomad alloc logs $(nomad job periodic force example-periodic-failure | grep Allocation | awk '{print $3}' | sed s/\"//g && sleep 5)       
$ hello world

Deploy with verbose logging

waypoint deploy -plain -vv                                                                                                                1 ↵
2022-02-03T16:19:51.052-0800 [INFO]  waypoint: waypoint version: full_string="v0.7.1 (062857d16)" version=v0.7.1 prerelease="" metadata="" revision=062857d16
2022-02-03T16:19:51.052-0800 [DEBUG] waypoint: home configuration directory: path=/home/adriano/.config/waypoint
2022-02-03T16:19:51.052-0800 [INFO]  waypoint.server: attempting to source credentials and connect
2022-02-03T16:19:51.052-0800 [DEBUG] waypoint.serverclient: connection information: address=<REDACTED>:9701 tls=true tls_skip_verify=true send_auth=true has_token=true
2022-02-03T16:19:51.913-0800 [DEBUG] waypoint.server: connection established with sourced credentials
2022-02-03T16:19:52.163-0800 [INFO]  waypoint: server version info: version=v0.7.1 api_min=1 api_current=1 entrypoint_min=1 entrypoint_current=1
2022-02-03T16:19:52.163-0800 [INFO]  waypoint: negotiated api version: version=1
2022-02-03T16:19:52.163-0800 [DEBUG] waypoint: will operate on app: name=main

» Deploying main...
2022-02-03T16:19:52.670-0800 [DEBUG] waypoint.setupLocalJobSystem: determining if a local or remote runner should be used for this and future jobs
2022-02-03T16:19:52.904-0800 [DEBUG] waypoint: Remote operations are disabled for this project - operation cannot occur remotely
2022-02-03T16:19:52.904-0800 [DEBUG] waypoint: starting runner to process local jobs
2022-02-03T16:19:52.904-0800 [DEBUG] waypoint.runner: Created runner: id=01FV11M1Y8F2PND65B201B8NF7
2022-02-03T16:19:52.904-0800 [DEBUG] waypoint.runner: registering runner
2022-02-03T16:19:52.904-0800 [DEBUG] waypoint.runner: runner registered, waiting for first config processing
2022-02-03T16:19:53.119-0800 [INFO]  waypoint.runner.config_recv: new configuration received
2022-02-03T16:19:53.119-0800 [DEBUG] waypoint.runner.watch_config.watchloop: new config variables received, scheduling refresh
2022-02-03T16:19:53.620-0800 [DEBUG] waypoint.runner.watch_config.watchloop: new configuration computed
2022-02-03T16:19:53.620-0800 [INFO]  waypoint.runner: runner registered with server and ready
2022-02-03T16:19:53.620-0800 [DEBUG] waypoint.setupLocalJobSystem: result: isLocal=true
2022-02-03T16:19:53.620-0800 [DEBUG] waypoint: queueing job: operation=*gen.Job_Deploy
2022-02-03T16:19:53.620-0800 [DEBUG] waypoint.runner: opening job stream
2022-02-03T16:19:53.621-0800 [INFO]  waypoint.runner: waiting for job assignment
2022-02-03T16:19:54.100-0800 [DEBUG] waypoint: opening job stream: job_id=01FV11M3ZAN76JFGTV27ESDX7D
2022-02-03T16:19:54.100-0800 [INFO]  waypoint.runner: job assignment received: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy
2022-02-03T16:19:54.101-0800 [INFO]  waypoint.runner: starting job execution: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy
2022-02-03T16:19:54.101-0800 [DEBUG] waypoint.runner: job data downloaded (or local): job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy pwd=/home/adriano/git/example-periodic-failure ref=(*gen.Job_DataSource_Ref)(nil)
2022-02-03T16:19:54.309-0800 [DEBUG] waypoint.runner: plugin search path: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy path=["/home/adriano/git/example-periodic-failure", "/home/adriano/git/example-periodic-failure/.waypoint/plugins", "/home/adriano/.config/waypoint/plugins", "/home/adriano/.config/.waypoint/plugins"]
2022-02-03T16:19:54.309-0800 [DEBUG] waypoint.runner: searching for plugin: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy plugin_name=docker
2022-02-03T16:19:54.309-0800 [DEBUG] waypoint.runner: plugin found as builtin: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy plugin_name=docker
2022-02-03T16:19:54.309-0800 [INFO]  waypoint.runner: register: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy plugin_name=docker type=Mapper nil=false
2022-02-03T16:19:54.309-0800 [INFO]  waypoint.runner: register: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy plugin_name=docker type=Builder nil=false
2022-02-03T16:19:54.310-0800 [DEBUG] waypoint.runner: searching for plugin: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy plugin_name=nomad-jobspec
2022-02-03T16:19:54.310-0800 [DEBUG] waypoint.runner: plugin found as builtin: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy plugin_name=nomad-jobspec
2022-02-03T16:19:54.310-0800 [INFO]  waypoint.runner: register: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy plugin_name=nomad-jobspec type=Platform nil=false
2022-02-03T16:19:54.310-0800 [INFO]  waypoint.runner: register: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy plugin_name=nomad-jobspec type=Mapper nil=false
2022-02-03T16:19:54.312-0800 [DEBUG] waypoint.runner.app.main: loading mapper plugin: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy name=docker
2022-02-03T16:19:54.313-0800 [INFO]  waypoint.runner.app.main.mapper: launching plugin: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy type=Mapper path=/usr/bin/waypoint args=["/usr/bin/waypoint", "plugin", "docker"]
2022-02-03T16:19:54.313-0800 [DEBUG] waypoint.runner.app.main.mapper: starting plugin: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy path=/usr/bin/waypoint args=["/usr/bin/waypoint", "plugin", "docker"]
2022-02-03T16:19:54.313-0800 [DEBUG] waypoint.runner.app.main.mapper: plugin started: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy path=/usr/bin/waypoint pid=2130765
2022-02-03T16:19:54.313-0800 [DEBUG] waypoint.runner.app.main.mapper: waiting for RPC address: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy path=/usr/bin/waypoint
2022-02-03T16:19:54.355-0800 [DEBUG] waypoint.runner.app.main.mapper.waypoint: plugin address: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy @module=plugin address=/tmp/plugin3896974968 network=unix timestamp=2022-02-03T16:19:54.355-0800
2022-02-03T16:19:54.355-0800 [DEBUG] waypoint.runner.app.main.mapper: using plugin: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy version=1
2022-02-03T16:19:54.356-0800 [DEBUG] waypoint.runner.app.main.mapper: plugin successfully launched and connected: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy
2022-02-03T16:19:54.356-0800 [INFO]  waypoint.runner.app.main.mapper: initialized component: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy type=Mapper
2022-02-03T16:19:54.356-0800 [DEBUG] waypoint.runner.app.main: no mappers advertised by plugin, closing: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy name=docker
2022-02-03T16:19:54.357-0800 [DEBUG] waypoint.runner.app.main.mapper.stdio: received EOF, stopping recv loop: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy err="rpc error: code = Unavailable desc = error reading from server: EOF"
2022-02-03T16:19:54.359-0800 [DEBUG] waypoint.runner.app.main.mapper: plugin process exited: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy path=/usr/bin/waypoint pid=2130765
2022-02-03T16:19:54.359-0800 [DEBUG] waypoint.runner.app.main.mapper: plugin exited: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy
2022-02-03T16:19:54.359-0800 [DEBUG] waypoint.runner.app.main: loading mapper plugin: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy name=nomad-jobspec
2022-02-03T16:19:54.359-0800 [INFO]  waypoint.runner.app.main.mapper: launching plugin: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy type=Mapper path=/usr/bin/waypoint args=["/usr/bin/waypoint", "plugin", "nomad-jobspec"]
2022-02-03T16:19:54.359-0800 [DEBUG] waypoint.runner.app.main.mapper: starting plugin: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy path=/usr/bin/waypoint args=["/usr/bin/waypoint", "plugin", "nomad-jobspec"]
2022-02-03T16:19:54.359-0800 [DEBUG] waypoint.runner.app.main.mapper: plugin started: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy path=/usr/bin/waypoint pid=2130778
2022-02-03T16:19:54.359-0800 [DEBUG] waypoint.runner.app.main.mapper: waiting for RPC address: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy path=/usr/bin/waypoint
2022-02-03T16:19:54.390-0800 [DEBUG] waypoint.runner.app.main.mapper.waypoint: plugin address: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy @module=plugin address=/tmp/plugin3365249815 network=unix timestamp=2022-02-03T16:19:54.390-0800
2022-02-03T16:19:54.390-0800 [DEBUG] waypoint.runner.app.main.mapper: using plugin: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy version=1
2022-02-03T16:19:54.391-0800 [DEBUG] waypoint.runner.app.main.mapper: plugin successfully launched and connected: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy
2022-02-03T16:19:54.391-0800 [INFO]  waypoint.runner.app.main.mapper: initialized component: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy type=Mapper
2022-02-03T16:19:54.391-0800 [DEBUG] waypoint.runner.app.main: no mappers advertised by plugin, closing: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy name=nomad-jobspec
2022-02-03T16:19:54.391-0800 [DEBUG] waypoint.runner.app.main.mapper.stdio: received EOF, stopping recv loop: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy err="rpc error: code = Unavailable desc = error reading from server: EOF"
2022-02-03T16:19:54.393-0800 [DEBUG] waypoint.runner.app.main.mapper: plugin process exited: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy path=/usr/bin/waypoint pid=2130778
2022-02-03T16:19:54.393-0800 [DEBUG] waypoint.runner.app.main.mapper: plugin exited: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy
2022-02-03T16:19:54.393-0800 [INFO]  waypoint.runner: project initialized: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy workspace=default
2022-02-03T16:19:54.393-0800 [INFO]  waypoint.runner: executing operation: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy
2022-02-03T16:19:54.393-0800 [WARN]  waypoint.runner.app.main: failed to prepare template variables, will not be available: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy err="Error decoding template data for *gen.PushedArtifact: unexpected end of JSON input"
  Performing operation locally
2022-02-03T16:19:54.509-0800 [WARN]  waypoint: unknown stream event: job_id=01FV11M3ZAN76JFGTV27ESDX7D event="&{job:{id:"01FV11M3ZAN76JFGTV27ESDX7D"  application:{application:"main"  project:"example-periodic-failure"}  workspace:{workspace:"default"}  target_runner:{id:{id:"01FV11M1Y8F2PND65B201B8NF7"}}  data_source:{local:{}}  deploy:{artifact:{application:{application:"main"  project:"example-periodic-failure"}  workspace:{workspace:"default"}  sequence:3  id:"01FV11D0W6HTCMFHZGTD3820AG"  status:{state:SUCCESS  start_time:{seconds:1643933761  nanos:105711834}  complete_time:{seconds:1643933762  nanos:65342624}}  component:{type:BUILDER  name:"docker"}  artifact:{artifact:{[type.googleapis.com/docker.Image]:{image:"waypoint.local/main"  tag:"latest"  docker:{}}}}  build_id:"01FV11CY3VBAQQ0J3P0CAVWSX3"  labels:{key:"waypoint/workspace"  value:"default"}  job_id:"01FV11CXB0D4WHQ8YG9TPQZ4SK"  preload:{}}}  state:WAITING  assigned_runner:{id:"01FV11M1Y8F2PND65B201B8NF7"}  queue_time:{seconds:1643933994  nanos:986308762}  assign_time:{seconds:1643933994  nanos:988701935}  expire_time:{seconds:1643934024  nanos:986288566}}}"
2022-02-03T16:19:54.510-0800 [WARN]  waypoint: unknown stream event: job_id=01FV11M3ZAN76JFGTV27ESDX7D event="&{job:{id:"01FV11M3ZAN76JFGTV27ESDX7D"  application:{application:"main"  project:"example-periodic-failure"}  workspace:{workspace:"default"}  target_runner:{id:{id:"01FV11M1Y8F2PND65B201B8NF7"}}  data_source:{local:{}}  deploy:{artifact:{application:{application:"main"  project:"example-periodic-failure"}  workspace:{workspace:"default"}  sequence:3  id:"01FV11D0W6HTCMFHZGTD3820AG"  status:{state:SUCCESS  start_time:{seconds:1643933761  nanos:105711834}  complete_time:{seconds:1643933762  nanos:65342624}}  component:{type:BUILDER  name:"docker"}  artifact:{artifact:{[type.googleapis.com/docker.Image]:{image:"waypoint.local/main"  tag:"latest"  docker:{}}}}  build_id:"01FV11CY3VBAQQ0J3P0CAVWSX3"  labels:{key:"waypoint/workspace"  value:"default"}  job_id:"01FV11CXB0D4WHQ8YG9TPQZ4SK"  preload:{}}}  state:RUNNING  assigned_runner:{id:"01FV11M1Y8F2PND65B201B8NF7"}  queue_time:{seconds:1643933994  nanos:986308762}  assign_time:{seconds:1643933994  nanos:988701935}  ack_time:{seconds:1643933995  nanos:438218706}  expire_time:{seconds:1643934024  nanos:986288566}}}"
2022-02-03T16:19:54.511-0800 [WARN]  waypoint: unknown stream event: job_id=01FV11M3ZAN76JFGTV27ESDX7D event="&{job:{id:"01FV11M3ZAN76JFGTV27ESDX7D"  application:{application:"main"  project:"example-periodic-failure"}  workspace:{workspace:"default"}  target_runner:{id:{id:"01FV11M1Y8F2PND65B201B8NF7"}}  data_source:{local:{}}  deploy:{artifact:{application:{application:"main"  project:"example-periodic-failure"}  workspace:{workspace:"default"}  sequence:3  id:"01FV11D0W6HTCMFHZGTD3820AG"  status:{state:SUCCESS  start_time:{seconds:1643933761  nanos:105711834}  complete_time:{seconds:1643933762  nanos:65342624}}  component:{type:BUILDER  name:"docker"}  artifact:{artifact:{[type.googleapis.com/docker.Image]:{image:"waypoint.local/main"  tag:"latest"  docker:{}}}}  build_id:"01FV11CY3VBAQQ0J3P0CAVWSX3"  labels:{key:"waypoint/workspace"  value:"default"}  job_id:"01FV11CXB0D4WHQ8YG9TPQZ4SK"  preload:{}}}  state:RUNNING  assigned_runner:{id:"01FV11M1Y8F2PND65B201B8NF7"}  queue_time:{seconds:1643933994  nanos:986308762}  assign_time:{seconds:1643933994  nanos:988701935}  ack_time:{seconds:1643933995  nanos:438218706}  config:{source:FILE}  expire_time:{seconds:1643934024  nanos:986288566}}}"
2022-02-03T16:19:54.717-0800 [INFO]  waypoint.runner.app.main.platform: launching plugin: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy type=Platform path=/usr/bin/waypoint args=["/usr/bin/waypoint", "plugin", "nomad-jobspec"]
2022-02-03T16:19:54.718-0800 [DEBUG] waypoint.runner.app.main.platform: starting plugin: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy path=/usr/bin/waypoint args=["/usr/bin/waypoint", "plugin", "nomad-jobspec"]
2022-02-03T16:19:54.718-0800 [DEBUG] waypoint.runner.app.main.platform: plugin started: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy path=/usr/bin/waypoint pid=2130790
2022-02-03T16:19:54.718-0800 [DEBUG] waypoint.runner.app.main.platform: waiting for RPC address: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy path=/usr/bin/waypoint
2022-02-03T16:19:54.787-0800 [DEBUG] waypoint.runner.app.main.platform.waypoint: plugin address: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy address=/tmp/plugin554223416 network=unix @module=plugin timestamp=2022-02-03T16:19:54.787-0800
2022-02-03T16:19:54.787-0800 [DEBUG] waypoint.runner.app.main.platform: using plugin: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy version=1
2022-02-03T16:19:54.788-0800 [INFO]  waypoint.runner.app.main.platform: platform plugin capable of destroy: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy
2022-02-03T16:19:54.788-0800 [INFO]  waypoint.runner.app.main.platform: platform plugin capable of generation ID creation: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy
2022-02-03T16:19:54.788-0800 [DEBUG] waypoint.runner.app.main.platform: plugin successfully launched and connected: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy
2022-02-03T16:19:54.788-0800 [INFO]  waypoint.runner.app.main.platform: initialized component: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy type=Platform
2022-02-03T16:19:55.794-0800 [DEBUG] waypoint.runner.app.main.deploy: creating metadata on server: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy
2022-02-03T16:19:56.475-0800 [DEBUG] waypoint.runner.app.main.deploy: running local operation: id=01FV11M63VCDS68BDY87QDR04V job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy
2022-02-03T16:19:56.477-0800 [DEBUG] waypoint.runner.app.main.platform.stdio: received EOF, stopping recv loop: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy err="rpc error: code = Unavailable desc = error reading from server: EOF"
2022-02-03T16:19:56.486-0800 [DEBUG] waypoint.runner.app.main.platform: plugin process exited: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy path=/usr/bin/waypoint pid=2130790
2022-02-03T16:19:56.486-0800 [DEBUG] waypoint.runner.app.main.platform: plugin exited: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy
2022-02-03T16:19:56.487-0800 [INFO]  waypoint.runner.app.main.platform: launching plugin: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy type=Platform path=/usr/bin/waypoint args=["/usr/bin/waypoint", "plugin", "nomad-jobspec"]
2022-02-03T16:19:56.487-0800 [DEBUG] waypoint.runner.app.main.platform: starting plugin: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy path=/usr/bin/waypoint args=["/usr/bin/waypoint", "plugin", "nomad-jobspec"]
2022-02-03T16:19:56.487-0800 [DEBUG] waypoint.runner.app.main.platform: plugin started: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy path=/usr/bin/waypoint pid=2130803
2022-02-03T16:19:56.487-0800 [DEBUG] waypoint.runner.app.main.platform: waiting for RPC address: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy path=/usr/bin/waypoint
2022-02-03T16:19:56.552-0800 [DEBUG] waypoint.runner.app.main.platform.waypoint: plugin address: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy address=/tmp/plugin2078468760 network=unix @module=plugin timestamp=2022-02-03T16:19:56.552-0800
2022-02-03T16:19:56.552-0800 [DEBUG] waypoint.runner.app.main.platform: using plugin: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy version=1
2022-02-03T16:19:56.553-0800 [INFO]  waypoint.runner.app.main.platform: platform plugin capable of destroy: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy
2022-02-03T16:19:56.553-0800 [INFO]  waypoint.runner.app.main.platform: platform plugin capable of generation ID creation: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy
2022-02-03T16:19:56.554-0800 [DEBUG] waypoint.runner.app.main.platform: plugin successfully launched and connected: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy
2022-02-03T16:19:56.554-0800 [INFO]  waypoint.runner.app.main.platform: initialized component: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy type=Platform
2022-02-03T16:19:56.554-0800 [DEBUG] waypoint.runner.app.main: evaluating config vars for syncing: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy
2022-02-03T16:19:56.554-0800 [DEBUG] waypoint.runner.app.main: no file-based config vars, not syncing config: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy
-> Initializing the Nomad client...
-> Parsing the job specification...
-> Registering job "example-periodic-failure"...
Monitoring evaluation ""
2022-02-03T16:19:59.815-0800 [WARN]  waypoint.runner.app.main.deploy: error during local operation: id=01FV11M63VCDS68BDY87QDR04V job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy err="rpc error: code = Unknown desc = No evaluation with id "" found"
2022-02-03T16:20:00.427-0800 [DEBUG] waypoint.runner.app.main.deploy: metadata marked as complete: id=01FV11M63VCDS68BDY87QDR04V job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy
2022-02-03T16:20:00.428-0800 [DEBUG] waypoint.runner.app.main.platform.stdio: received EOF, stopping recv loop: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy err="rpc error: code = Unavailable desc = error reading from server: EOF"
2022-02-03T16:20:00.435-0800 [DEBUG] waypoint.runner.app.main.platform: plugin process exited: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy path=/usr/bin/waypoint pid=2130803
2022-02-03T16:20:00.435-0800 [DEBUG] waypoint.runner.app.main.platform: plugin exited: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy
2022-02-03T16:20:00.435-0800 [DEBUG] waypoint.runner: closing project: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy
2022-02-03T16:20:00.435-0800 [DEBUG] waypoint.runner: job finished: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy error="rpc error: code = Unknown desc = No evaluation with id "" found"
2022-02-03T16:20:00.435-0800 [WARN]  waypoint.runner: error during job execution: job_id=01FV11M3ZAN76JFGTV27ESDX7D job_op=*gen.Job_Deploy err="rpc error: code = Unknown desc = No evaluation with id "" found"
2022-02-03T16:20:01.169-0800 [DEBUG] waypoint.runner: opening job stream
2022-02-03T16:20:01.169-0800 [WARN]  waypoint: unknown stream event: job_id=01FV11M3ZAN76JFGTV27ESDX7D event="&{job:{id:"01FV11M3ZAN76JFGTV27ESDX7D"  application:{application:"main"  project:"example-periodic-failure"}  workspace:{workspace:"default"}  target_runner:{id:{id:"01FV11M1Y8F2PND65B201B8NF7"}}  data_source:{local:{}}  deploy:{artifact:{application:{application:"main"  project:"example-periodic-failure"}  workspace:{workspace:"default"}  sequence:3  id:"01FV11D0W6HTCMFHZGTD3820AG"  status:{state:SUCCESS  start_time:{seconds:1643933761  nanos:105711834}  complete_time:{seconds:1643933762  nanos:65342624}}  component:{type:BUILDER  name:"docker"}  artifact:{artifact:{[type.googleapis.com/docker.Image]:{image:"waypoint.local/main"  tag:"latest"  docker:{}}}}  build_id:"01FV11CY3VBAQQ0J3P0CAVWSX3"  labels:{key:"waypoint/workspace"  value:"default"}  job_id:"01FV11CXB0D4WHQ8YG9TPQZ4SK"  preload:{}}}  state:ERROR  assigned_runner:{id:"01FV11M1Y8F2PND65B201B8NF7"}  queue_time:{seconds:1643933994  nanos:986308762}  assign_time:{seconds:1643933994  nanos:988701935}  ack_time:{seconds:1643933995  nanos:438218706}  complete_time:{seconds:1643934001  nanos:766542813}  config:{source:FILE}  error:{code:2  message:"No evaluation with id \"\" found"}  expire_time:{seconds:1643934024  nanos:986288566}}}"
2022-02-03T16:20:01.169-0800 [INFO]  waypoint.runner: waiting for job assignment
2022-02-03T16:20:01.170-0800 [WARN]  waypoint: job failed: job_id=01FV11M3ZAN76JFGTV27ESDX7D code=Unknown message="No evaluation with id "" found"
! No evaluation with id "" found
2022-02-03T16:20:01.170-0800 [WARN]  waypoint.runner.watch_config: exiting due to context ended
@acaloiaro acaloiaro added the new label Feb 4, 2022
@briancain briancain added bug Something isn't working plugin/nomad-jobspec labels Feb 8, 2022
@xiaolin-ninja
Copy link
Contributor

At the moment, we don't support periodic job tasks. We wait for the job to start, but Waypoint exits due to no tasks starting because periodic jobs don't start immediately. We intend to support this in the future.

@xiaolin-ninja xiaolin-ninja added enhancement New feature or request and removed bug Something isn't working new labels Feb 9, 2022
@acaloiaro
Copy link
Contributor Author

Hi @xiaolin-ninja, thanks for the response.

I'm wondering if you can comment on the team's thinking about expected behavior for periodic Jobspec tasks in the future.

If you can share that, maybe I or someone else in the community can start working on a pull request.

Cheers

@acaloiaro acaloiaro mentioned this issue Apr 24, 2022
Open
@edegenetais-nx
Copy link

Hi ! We are also impacted by this limitation, as we used waypoint to deploy projects involving periodic nomad jobs.
We are currently using waypoint 0.5.2, and using this version periodic jobs get deployed. However, while testing 0.9.1 we discovered that it does not work anymore.
@acaloiaro had asked if @xiaolin-ninja could comment on the team's thinkingg about expected behavior for periodic Jobspec tasks. Are there some news about this ?

Thanks

@edegenetais-nx
Copy link

To sum up tests we have done with various versions :

  • 0.5.2 : deploys
  • 0.7.2 : deploys but ends in error, not checking if deployed (the job is, however, created and works, ie tasks are started as planned)
  • 0.8.2, 0.9.1, 0.10.1 : it gets worse. Deployment ends very quickly with an error message ("error reading from server: EOF") and moreover, no job is created, no task planned.

@paladin-devops paladin-devops self-assigned this Sep 30, 2022
@paladin-devops paladin-devops mentioned this issue Sep 30, 2022
Merged
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@paladin-devops paladin-devops

Labels
enhancement New feature or request plugin/nomad-jobspec
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

nomad-jobspec plugin periodic and system job fixes hashicorp/waypoint
5 participants
@briancain @acaloiaro @xiaolin-ninja @edegenetais-nx @paladin-devops