Save and Restore - Cannot delete a snapshot

[rosstitmarsh]

I have created several configurations each containing a single snapshots using the REST API, then added the snapshots to a composite snapshot. I then deleted the composite snapshot and tried to delete all of the previously created configurations and snapshots. This was successful for all of them except for one of snapshots them which gives this error when trying to delete it.

java.util.concurrent.ExecutionException:` java.lang.RuntimeException: org.phoebus.applications.saveandrestore.SaveAndRestoreClientException: Failed to delete nodes: {"timestamp":1760360793102,"status":500,"error":"Internal Server Error","path":"/save-restore/node"}
	at java.base/java.util.concurrent.FutureTask.report(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.get(Unknown Source)
	at org.phoebus.applications.saveandrestore.ui.SaveAndRestoreService.deleteNodes(SaveAndRestoreService.java:129)
	at org.phoebus.applications.saveandrestore.ui.SaveAndRestoreController.lambda$deleteTreeItems$21(SaveAndRestoreController.java:480)
	at org.phoebus.framework.jobs.Job.execute(Job.java:50)
	at org.phoebus.framework.jobs.JobManager.execute(JobManager.java:54)
	at org.phoebus.framework.jobs.JobManager.lambda$schedule$0(JobManager.java:45)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: org.phoebus.applications.saveandrestore.SaveAndRestoreClientException: Failed to delete nodes: {"timestamp":1760360793102,"status":500,"error":"Internal Server Error","path":"/save-restore/node"}
	at org.phoebus.applications.saveandrestore.client.SaveAndRestoreClientImpl.deleteNodes(SaveAndRestoreClientImpl.java:201)
	at org.phoebus.applications.saveandrestore.ui.SaveAndRestoreService.lambda$deleteNodes$4(SaveAndRestoreService.java:129)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	... 4 more
Caused by: org.phoebus.applications.saveandrestore.SaveAndRestoreClientException: Failed to delete nodes: {"timestamp":1760360793102,"status":500,"error":"Internal Server Error","path":"/save-restore/node"}
	at org.phoebus.applications.saveandrestore.client.SaveAndRestoreClientImpl.deleteNodes(SaveAndRestoreClientImpl.java:198)
	... 6 more

I have a attached copy of the server logs here Phoebus_-_Save_and_restore.log

[georgweiss]

@rosstitmarsh , could you please be more specific on the actions you took in the UI? E.g. what UI elements did you select, what context menu selections etc.

[georgweiss]

@rosstitmarsh, I am unable to reproduce your issue.
Based on the stack trace and server log, seems client attempts to delete something that does not exist.

Also:

I have created several configurations each containing a single snapshots using the REST API

Does that mean you have another client using the REST API?

[rosstitmarsh]

I have been trying to reproduce the circumstances but have been unsuccessful.

The REST API was just used to create the configs and snapshots.

I have 4 undeletable snapshots, I have tried deleting them from 2 difference PCs using Phoebus and using the REST API, all return the same 500 error.

[georgweiss]

Can you see those snapshots in the UI? Do they have a unique ID (use context menu to copy unique id to clipboard)?

[rosstitmarsh]

Yes they are visible in Phoebus, I can interact with them and change the name and comments.

Here is the responses when using REST API with a Jupyter notebook

snapshot_id = "ca122d9f-baa3-4cc8-8fca-baae8d95ec9e"

response = client.get(f"{sar_url}/node/{snapshot_id}")
print(json.dumps(response.json(), indent=4))

response = client.request("delete", f"{sar_url}/node", json=[snapshot_id])
print(json.dumps(response.json(), indent=4))

{
    "uniqueId": "ca122d9f-baa3-4cc8-8fca-baae8d95ec9e",
    "name": "renamed snapshot",
    "description": "cannot delete",
    "created": 1760022717704,
    "lastModified": 1760445896708,
    "nodeType": "SNAPSHOT",
    "userName": "admin",
    "tags": [],
    "properties": null
}
{
    "timestamp": 1760446039411,
    "status": 500,
    "error": "Internal Server Error",
    "path": "/save-restore/node"
}

[georgweiss]

Deleting a node with curl like so:
>curl -u <user> -XDELETE http(s)://<server>/save-restore/node/<node-id>

Looks like the python code is sending the node id as json body

[rosstitmarsh]

That returns the same error.

$ curl -u admin -XDELETE http://save-and-restore-service:8001/save-restore/node/ca122d9f-baa3-4cc8-8fca-baae8d95ec9e
Enter host password for user 'admin':
{"timestamp":1760446576814,"status":500,"error":"Internal Server Error","path":"/save-restore/node/ca122d9f-baa3-4cc8-8fca-baae8d95ec9e"}

Is the json list method incorrect? It works for deleting any other node.

[georgweiss]

You're right, my bad. You may also define a list of node ids in a json body.
Do you have a server log for the curl attempt?

[rosstitmarsh]

This is the result of the curl I believe.

Phoebus_-_Save_and_restore.log

[georgweiss]

@rosstitmarsh, based on the server log it looks like you have stale or inconsistent data for composite snapshots in your Elasticsearch index. This I suspect may be due to incorrectly created composite snapshots when using the REST API.

You may need to interact directly with Elasticsearch to check your data. For instance, to retrieve composite snapshot data you can do it using an URL like http(s)://:9200/saveandrestore_composite_snapshot/_search, which returns up to 10 documents. See below for an example and please note that the "uniqueId" of each document must be set. If it is not, then that could explain the 500 error.

{
  "took": 15,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 21,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "saveandrestore_composite_snapshot",
        "_id": "38e2d387-7231-49a7-9d5f-42316088d46c",
        "_score": 1,
        "_source": {
          "uniqueId": "38e2d387-7231-49a7-9d5f-42316088d46c",
          "referencedSnapshotNodes": [
            "317592e9-5439-41ee-a434-e2dbc154f481",
            "56be4a6a-e772-4e0a-ad74-64b8b02686f8",
            "1393c99e-3e41-44e3-a5a9-6fd67a03df89",
            "cecf74a8-53b6-4acf-9f55-e87ca35044d8",
            "77d74576-b3ea-45d1-82ba-a407137319e7",
            "ffcc147e-8288-4792-ad22-735baa73309f",
            "9b91a975-12b6-4fba-8cb8-d29bc69e38b5"
          ]
        }
      },
      {
        "_index": "saveandrestore_composite_snapshot",
        "_id": "29278d0b-6e86-4e9c-9526-85e68a9f8d43",
        "_score": 1,
        "_source": {
          "uniqueId": "29278d0b-6e86-4e9c-9526-85e68a9f8d43",
          "referencedSnapshotNodes": [
            "ca1724ec-546a-4f1a-b835-8847e3d0a8d9",
            "ad3d2b8f-a863-48f8-bbe6-b4766d3bcacb",
            "04547a96-9569-42eb-a506-1d80601f7968",
            "82272cc6-c019-46fb-aba5-b17da9cf8303",
            "f3f9a788-3a93-4a1b-8024-826f66a8ca74"
          ]
        }
      },
      {
        "_index": "saveandrestore_composite_snapshot",
        "_id": "4e1c913c-a43f-4ef5-881e-869c2882861a",
        "_score": 1,
        "_source": {
          "uniqueId": "4e1c913c-a43f-4ef5-881e-869c2882861a",
          "referencedSnapshotNodes": [
            "324ee3d8-03e0-4f96-a09a-e827c795c5ec",
            "c679daba-0982-4218-8727-d5047b8b257d",
            "28c07ff1-7226-4e49-9340-bca0ff75bc24"
          ]
        }
      },
      {
        "_index": "saveandrestore_composite_snapshot",
        "_id": "2bcaead6-e394-48a3-bc0b-551fe6a8da8a",
        "_score": 1,
        "_source": {
          "uniqueId": "2bcaead6-e394-48a3-bc0b-551fe6a8da8a",
          "referencedSnapshotNodes": [
            "57bf92bc-0e6f-494c-b05a-eebe18352100",
            "c93696e8-4994-4f2e-a57f-43931e1216d7",
            "3676e4c4-e41c-4b4f-a646-a49876ae1454"
          ]
        }
      },
      {
        "_index": "saveandrestore_composite_snapshot",
        "_id": "81104977-ed0c-4f1c-803f-f33022848d93",
        "_score": 1,
        "_source": {
          "uniqueId": "81104977-ed0c-4f1c-803f-f33022848d93",
          "referencedSnapshotNodes": [
            "178e4104-eddb-4da6-9385-935e178d74d2",
            "351f136d-03e5-4e54-b830-6f7dc7102d22",
            "a53e02f6-60fd-4972-b7b2-087f75540098"
          ]
        }
      },
      {
        "_index": "saveandrestore_composite_snapshot",
        "_id": "c93696e8-4994-4f2e-a57f-43931e1216d7",
        "_score": 1,
        "_source": {
          "uniqueId": "c93696e8-4994-4f2e-a57f-43931e1216d7",
          "referencedSnapshotNodes": [
            "890d0da2-f521-48fd-8719-0141c0ef956a",
            "2ff17e07-77c5-41dd-bae6-49dcff87b07e"
          ]
        }
      },
      {
        "_index": "saveandrestore_composite_snapshot",
        "_id": "3676e4c4-e41c-4b4f-a646-a49876ae1454",
        "_score": 1,
        "_source": {
          "uniqueId": "3676e4c4-e41c-4b4f-a646-a49876ae1454",
          "referencedSnapshotNodes": [
            "3b60fc64-bcf0-494c-8886-0d53e836ba60",
            "908ded09-2c22-404f-9928-6256fbbe73e6"
          ]
        }
      },
      {
        "_index": "saveandrestore_composite_snapshot",
        "_id": "8d1a0762-6e88-4cde-a0c9-45f5cc145cbf",
        "_score": 1,
        "_source": {
          "uniqueId": "8d1a0762-6e88-4cde-a0c9-45f5cc145cbf",
          "referencedSnapshotNodes": []
        }
      },
      {
        "_index": "saveandrestore_composite_snapshot",
        "_id": "202957f8-37e7-435c-8919-e8ab49042210",
        "_score": 1,
        "_source": {
          "uniqueId": "202957f8-37e7-435c-8919-e8ab49042210",
          "referencedSnapshotNodes": [
            "8335ccf7-a71a-4206-90c5-eb29118b4e28",
            "30779562-62de-413c-b0df-314af874cedc",
            "84911777-e025-492b-818f-042f5b4f10b8",
            "8f3fdc94-17aa-4a26-a768-6d9bfc18e49a",
            "8ae836f0-4d8c-43dd-acf6-fcb4e0ec2e06",
            "d8a4a5ba-d66b-4d9e-b13f-89f6f9511ade",
            "db9c3f89-cdb4-49b4-9295-1f9117b211d5",
            "6ae33289-f046-49f6-a33b-9a9345239b66",
            "b410f2b4-3ab8-404e-8e71-62888a7b363f",
            "55dfaf31-fa39-44ac-909b-8778b99fdbd4",
            "d1c4369d-cf5b-4a91-8ab3-b90b4c57ed92",
            "88c5b76f-c176-4548-84a3-326c0ae7af85",
            "9ba3489d-484a-4b7a-ae60-43cb6a9104bf",
            "42cfeae4-1fec-4bf8-89f8-5f0694cc2c5a",
            "393461d5-52fa-43d8-a8a5-1c52dc9985b1"
          ]
        }
      },
      {
        "_index": "saveandrestore_composite_snapshot",
        "_id": "352d0262-c803-4d3c-9037-7ad411cff8a0",
        "_score": 1,
        "_source": {
          "uniqueId": "352d0262-c803-4d3c-9037-7ad411cff8a0",
          "referencedSnapshotNodes": [
            "95af6324-bb64-4eac-b250-197a600df9a8",
            "cb93d99d-79c8-4713-b2a3-ca0a04958db1",
            "1602beb9-237a-47f7-8100-b02923fa7b6a",
            "3945a4a7-3c2e-4d0b-a033-da7d6bf6b4a4",
            "99c60875-dae7-479c-978e-abf609561fe7",
            "03a28ffc-d6e3-4cd5-8b0f-cfbd5368e7aa",
            "cb56f376-0880-4133-bd03-d7c9609c0410",
            "81104977-ed0c-4f1c-803f-f33022848d93",
            "c93696e8-4994-4f2e-a57f-43931e1216d7",
            "3676e4c4-e41c-4b4f-a646-a49876ae1454",
            "4e1c913c-a43f-4ef5-881e-869c2882861a"
          ]
        }
      }
    ]
  }
}

[rosstitmarsh]

That looks promising, there are entries missing uniqueId, the _id fields are not UUIDs like in your example.

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 47,
      "relation": "eq"
    },
    "max_score": 1.0,
    "hits": [
      {
        "_index": "saveandrestore_composite_snapshot",
        "_id": "kL_d3ZkBjKls7HwZrGmA",
        "_score": 1.0,
        "_source": {
          "referencedSnapshotNodes": []
        }
      },
      {
        "_index": "saveandrestore_composite_snapshot",
        "_id": "Br9a3ZkBjKls7HwZc0BZ",
        "_score": 1.0,
        "_source": {
          "referencedSnapshotNodes": []
        }
      },
      {
        "_index": "saveandrestore_composite_snapshot",
        "_id": "b79b3ZkBjKls7HwZ0EBr",
        "_score": 1.0,
        "_source": {
          "referencedSnapshotNodes": []
        }
      },
      {
        "_index": "saveandrestore_composite_snapshot",
        "_id": "er9c3ZkBjKls7HwZWUCw",
        "_score": 1.0,
        "_source": {
          "referencedSnapshotNodes": []
        }
      },
      {
        "_index": "saveandrestore_composite_snapshot",
        "_id": "e79c3ZkBjKls7HwZW0Ct",
        "_score": 1.0,
        "_source": {
          "referencedSnapshotNodes": []
        }
      },
      {
        "_index": "saveandrestore_composite_snapshot",
        "_id": "gL9d3ZkBjKls7HwZi0BU",
        "_score": 1.0,
        "_source": {
          "referencedSnapshotNodes": []
        }
      },
      {
        "_index": "saveandrestore_composite_snapshot",
        "_id": "jr9f3ZkBjKls7HwZYEAt",
        "_score": 1.0,
        "_source": {
          "referencedSnapshotNodes": []
        }
      },
      {
        "_index": "saveandrestore_composite_snapshot",
        "_id": "nb_D3ZkBjKls7HwZ-2D2",
        "_score": 1.0,
        "_source": {
          "referencedSnapshotNodes": []
        }
      },
      {
        "_index": "saveandrestore_composite_snapshot",
        "_id": "j7_d3ZkBjKls7HwZDGnh",
        "_score": 1.0,
        "_source": {
          "referencedSnapshotNodes": []
        }
      },
      {
        "_index": "saveandrestore_composite_snapshot",
        "_id": "OrqHyZkBjKls7HwZaamv",
        "_score": 1.0,
        "_source": {
          "referencedSnapshotNodes": [
            "ca122d9f-baa3-4cc8-8fca-baae8d95ec9e"
          ]
        }
      }
    ]
  }
}

[rosstitmarsh]

So it seams the cause was that my earlier PUT requests to the composite snapshots endpoint were missing the uniqueId field under the data section.

[georgweiss]

Can you confirm my hypothesis about the missing uniqueId? Asking as the service adds the uniqueId when creating a composite snapshot. So I'm a bit confused...

[rosstitmarsh]

Yes I have reproduced it.

First I created a new configuration and snapshot through the Phoebus GUI.

Then in a notebook:

First get the composite snapshot.

snapshot_id = "b70d763e-dd4b-4502-9e5c-7b3ff92d84ea"
composite_snapshot_id = "86765ace-f2b1-47da-9d18-df5e554881be"

node = client.get(f"{sar_url}/node/{composite_snapshot_id}").json()
data = client.get(f"{sar_url}/composite-snapshot/{composite_snapshot_id}").json()

composite_snapshot = {"compositeSnapshotNode": node, "compositeSnapshotData": data}
composite_snapshot

{'compositeSnapshotNode': {'uniqueId': '86765ace-f2b1-47da-9d18-df5e554881be',
  'name': 'test-composite-snapshot',
  'description': 'Test composite snapshot',
  'created': 1760460816966,
  'lastModified': 1760460816966,
  'nodeType': 'COMPOSITE_SNAPSHOT',
  'userName': 'admin',
  'tags': None,
  'properties': None},
 'compositeSnapshotData': {'uniqueId': '86765ace-f2b1-47da-9d18-df5e554881be',
  'referencedSnapshotNodes': []}}

Then replace the data section but miss out the uniqueId.

composite_snapshot["compositeSnapshotData"] = {"referencedSnapshotNodes": [snapshot_id]}
composite_snapshot

{'compositeSnapshotNode': {'uniqueId': '86765ace-f2b1-47da-9d18-df5e554881be',
  'name': 'test-composite-snapshot',
  'description': 'Test composite snapshot',
  'created': 1760460816966,
  'lastModified': 1760460816966,
  'nodeType': 'COMPOSITE_SNAPSHOT',
  'userName': 'admin',
  'tags': None,
  'properties': None},
 'compositeSnapshotData': {'referencedSnapshotNodes': ['b70d763e-dd4b-4502-9e5c-7b3ff92d84ea']}}

Then POST this to the server. It returns a null uniqueId.

composite_snapshot = client.post(f"{sar_url}/composite-snapshot", json=composite_snapshot).json()
composite_snapshot

{'compositeSnapshotNode': {'uniqueId': '86765ace-f2b1-47da-9d18-df5e554881be',
  'name': 'test-composite-snapshot',
  'description': 'Test composite snapshot',
  'created': 1760460816966,
  'lastModified': 1760461342889,
  'nodeType': 'COMPOSITE_SNAPSHOT',
  'userName': 'admin',
  'tags': None,
  'properties': None},
 'compositeSnapshotData': {'uniqueId': None,
  'referencedSnapshotNodes': ['b70d763e-dd4b-4502-9e5c-7b3ff92d84ea']}}

Then back in Phoebus I deleted the composite snapshot, no errors. Trying to delete the referenced snapshots produces the 500 error.

[georgweiss]

So it seems you are calling the POST endpoint which is intended for updates of composite snapshots. In doing so you need to add the uniqueId in the compositeSnapshotData portion. If not, then the snapshot data will be saved without uniqueId. This has a side effect: when deleting snapshots, the service checks if they are referenced by any composite snapshot. In that check the uniqueId is needed to identify a composite snapshot node.

I will update the service to make sure the uniqueId is properly set when updating a composite snapshot.