New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Operation doesn't exist when requesting /operations/[uuid]/wait #4721

Closed
Spunge opened this Issue Jul 3, 2018 · 3 comments

Comments

2 participants
@Spunge

Spunge commented Jul 3, 2018

Required information

  • Distribution: Ubuntu
  • Distribution version: 16.04

driver: lxc
driver_version: 3.0.1
kernel: Linux
kernel_architecture: x86_64
kernel_version: 4.4.0-130-generic
server: lxd
server_pid: 17648
server_version: "3.2"
storage: zfs
storage_version: 0.6.5.6-0ubuntu20
server_clustered: true

Issue description

When creating any 'task' operation, after hitting the /operations/63277c1b-0ecc-460c-9e85-36ba6bf7fd89/wait url on a clustered LXD setup, i get the error: "Error: Operation '63277c1b-0ecc-460c-9e85-36ba6bf7fd89' doesn't exist", Hitting the operation url without /wait at the end returns the operation data as expected (but does not wait). I only experience these issues with 3 node lxd cluster, same lxd version works fine with 1 daemon.

Steps to reproduce

  1. Setup clustered lxd
  2. Create container via API
  3. Hit the returned operation URL to wait for container setup

Information to attach

Daemon output while hitting these urls:

k=1 lvl=dbug msg="Found cert" t=2018-07-03T08:20:02+0000
ip=10.254.254.254:42024 lvl=dbug method=POST msg=handling t=2018-07-03T08:20:02+0000 url=/1.0/containers
lvl=dbug msg="Responding to container create" t=2018-07-03T08:20:02+0000
lvl=dbug msg="Connecting to a remote LXD over HTTPs" t=2018-07-03T08:20:02+0000
lvl=dbug msg="Forward container post request to 10.254.0.3:8443" t=2018-07-03T08:20:02+0000
lvl=dbug msg="Connected to the websocket" t=2018-07-03T08:20:02+0000
etag= lvl=dbug method=POST msg="Sending request to LXD" t=2018-07-03T08:20:02+0000 url="https://10.254.0.3:8443/1.0/containers?target=forest-a24a0e08-a127-45f3-841d-dcca7d3517c0"
lvl=dbug msg="\n\t{\n\t\t\"architecture\": \"x86_64\",\n\t\t\"config\": {},\n\t\t\"devices\": null,\n\t\t\"ephemeral\": false,\n\t\t\"profiles\": [\n\t\t\t\"default\"\n\t\t],\n\t\t\"stateful\": false,\n\t\t\"description\": \"\",\n\t\t\"name\": \"test\",\n\t\t\"source\": {\n\t\t\t\"type\": \"image\",\n\t\t\t\"certificate\": \"\",\n\t\t\t\"alias\": \"application-php\"\n\t\t},\n\t\t\"instance_type\": \"\"\n\t}" t=2018-07-03T08:20:02+0000
lvl=dbug msg="Got operation from LXD" t=2018-07-03T08:20:02+0000
lvl=dbug msg="\n\t{\n\t\t\"id\": \"c77f4bb3-9e11-4860-b669-bbb032122ba7\",\n\t\t\"class\": \"task\",\n\t\t\"description\": \"Creating container\",\n\t\t\"created_at\": \"2018-07-03T08:20:02.30393348Z\",\n\t\t\"updated_at\": \"2018-07-03T08:20:02.30393348Z\",\n\t\t\"status\": \"Running\",\n\t\t\"status_code\": 103,\n\t\t\"resources\": {\n\t\t\t\"containers\": [\n\t\t\t\t\"/1.0/containers/test\"\n\t\t\t]\n\t\t},\n\t\t\"metadata\": null,\n\t\t\"may_cancel\": false,\n\t\t\"err\": \"\"\n\t}" t=2018-07-03T08:20:02+0000
k=1 lvl=dbug msg="Found cert" t=2018-07-03T08:20:02+0000
ip=10.254.254.254:42030 lvl=dbug method=GET msg=handling t=2018-07-03T08:20:02+0000 url=/1.0/operations/c77f4bb3-9e11-4860-b669-bbb032122ba7/wait
k=0 lvl=dbug msg="Found cert" t=2018-07-03T08:20:02+0000

Please tell me if you need more information.

@Spunge

This comment has been minimized.

Spunge commented Jul 3, 2018

I worked around it by using the events route like the go client does.

@Spunge Spunge changed the title from Operation doesn't exist to Operation doesn't exist when requesting /operations/[uuid]/wait Jul 3, 2018

@stgraber

This comment has been minimized.

Member

stgraber commented Jul 3, 2018

I'll take a look, we may be missing the usual cluster lookup and redirection logic for this specific api endpoint.

@stgraber stgraber self-assigned this Jul 3, 2018

@stgraber stgraber added the Bug label Jul 4, 2018

@stgraber stgraber added this to the lxd-3.3 milestone Jul 4, 2018

@stgraber

This comment has been minimized.

Member

stgraber commented Jul 6, 2018

Got a fix for this particular endpoint, but found a lot of other issues with operations in the cluster case so working through all the API endpoints making sure they're sane. Only one left now is the operation list (GET /1.0/operations) which needs to return all operations in the cluster.

@brauner brauner closed this in 024c71f Jul 6, 2018

stgraber added a commit that referenced this issue Jul 12, 2018

lxd/operations: Forward to right cluster node
Closes #4721

Signed-off-by: Stéphane Graber <stgraber@ubuntu.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment