Cancel upcoming activities

[marko-lisica]

Goal

User story
As an IT admin,
I want to cancel upcoming activities
so that I can prioritize important activities or cancel wrong actions.

Key result

One queue for MDM commands, software, and scripts

Original requests

#20712

Context

• Product designer: @marko-lisica

Changes

Product

• UI changes: Figma link
• CLI (fleetctl) usage changes: No changes.
• YAML changes: No changes.
• REST API changes: [API design] Cancel upcoming activities #27190
• Fleet's agent (fleetd) changes: No changes.
• GitOps mode changes: No changes
• Activity changes: No changes. We're tracking canceled activities on the host level, so no need to update Audit logs page
• Permissions changes:
  • [API design] Cancel upcoming activities #27190
  • Additional update to roles for cancel upcoming activities #27762
• Changes to paid features or tiers: Available in Fleet Free and Premium.
• Transparency changes: No changes.
• First draft of test plan added
• Other reference documentation changes: No changes.
• Once shipped, requester has been notified
• Once shipped, dogfooding issue has been filed. Dogfooding issue: fleetdm/confidential#10421

Engineering

• Test plan is finalized
• Feature guide changes: CUA: Guide #27534
• Database schema migrations: N/A
• Load testing: N/A

ℹ️  Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

QA

Risk assessment

• Requires load testing: no
• Risk level: Low / High: low
• Risk description: Introducing the unified queue was high risk and required load testing, but since those are now complete additional load testing is not required for the cancelation feature.

Test plan

Make sure to go through the list and consider all events that might be related to this story, so we catch edge cases earlier.

UI

• Make sure that user can cancel upcoming activities on the host details page, by hovering over the upcoming activity and clicking ❌ icon.
• Make sure that confirmation modal with details about activity that's being canceled pops up when user hit cancel icon
• Once the user confirms the cancel action, a success or error flash message should appear on the screen and upcoming activities should be refetched.
• Make sure that when user cancels activity it's tracked in host's past activity (and global past activities) as one of canceled_run_script, canceled_install_app_store_app, canceled_install_software, canceled_uninstall_software, or canceled_uninstall_app_store_app activity.
• Make sure that when Windows or Linux host is locked and the lock script is canceled, LOCK PENDING badge from host details is cleared and lock activity isn't logged to host's past and global activity feed.
• Make sure that when Windows or Linux host is wiped (Windows wipe is via MDM, not in the upcoming queue at the moment) and the wipe script is canceled, WIPE PENDING badge from host details is cleared and lock activity isn't logged to host's past and global activity feed.
• Make sure that if the user cancels installed_software or uninstalled_software activity, on the host details > software, the status of that software is changed from pending to whatever it was before.
• Make sure that if the user cancels installed_app_store_app or uninstalled_app_store_app activity, on the host details > software, the status of that software is changed from pending to whatever it was before.
• Make sure that if the user cancels ran_script activity, on the host details > run script modal, status of that script is changed from pending to whatever it was before.
• Make sure that pending status count on the software title details page is updated after installed_software or installed_app_store_app is canceled.
• Make sure that only users with maintainer or admin role can cancel host's upcoming activities.
• Make sure that when 1st activity in the upcoming queue is canceled, results from that activity isn't tracked in the host's past or global feed, just activity about cancelation is tracked in host's past feed.
• Make sure that when installing software if the host isn't in the target anymore (team change or labels change) software install upcoming activity is canceled by Fleet and not tracked as canceled activity in the host's past feed.
• Make sure that if host record is deleted from Fleet, and it's enrolled again, all previous upcoming and past activities are deleted and don't affect new enrollment.
• If user cancel upcoming activity and it's running or is about to run in setup experience flow, Fleet should show failed status for those activities (software or script) and proceed to next one.
• Make sure that if a Lock or Wipe activity is next to run (first in queue), the server prevents cancelling it and the expected error message is displayed.

API

• Make sure that user can cancel upcoming activity via API: DELETE /api/v1/fleet/hosts/:id/activities/upcoming/:activity_id
• Make sure that cancelable flag in the upcoming activity response is deleted.

Happy path ✅

1. Go to host details page, scroll to activities card, then select upcoming activities (assuming there's a few upcoming activities feed)
2. Cancel 3rd upcoming activity in the list.
3. It was successful, activities are refetched, and cancel is tracked in the host's past activity feed (and the global feed).
4. Now, cancel 1st activity in the upcoming queue
5. It was successful, activities are refetched, and canceled activity isn't tracked in the host's past feed, nor in the global feed. Only activity about cancelation is tracked in the host's past activity feed (and the global feed).

Testing notes

Confirmation

1. Engineer: Added comment to user story confirming successful completion of test plan.
2. QA: Added comment to user story confirming successful completion of test plan.

[noahtalerman]

@mna can you please add the SQL commands to unstick pending activities?

[noahtalerman]

@marko-lisica I just chatted with Martin re this Slack thread and learned that when we ship #22866 it's possible that some commands might get stuck and block all other commands due to unanticipated edge cases. In the next release (4.65) the work around will be to run some SQL command to unstick activities. Not the best.

So, I think we want to prioritize this story so we can get to canceling as soon as we can (current sprint, 4.66, if possible). I added P2 to it and assigned @lukeheath.

If Luke agrees let's prioritize drafting this one above all other stories (including IdP host vitals). If Luke says we need to wait, I think we wrap up the IdP host vitals story and then shift to this one to get it ready for next sprint (4.67).

[mna]

UPDATE: Moved these instructions to a public Google doc here: https://docs.google.com/document/d/1yTC-CHHEV7eoFs0AQb1QxNo6fFafq8fk78SRCFeKank/edit?tab=t.0

---

@noahtalerman Unblocking the unified queue via the database is a 3-step process :

1. Identify the activity (in rare cases, possibly activities) that are stuck, along with its type;
2. Insert a bogus result for that activity, how to do so varies based on its activity type;
3. Send a bogus "run script" command to force the upcoming queue to make progress (activate any subsequent activity).

Details for each step follow.

1. Identify the activity

Run the following query to identify the activity's execution_id and activity_type. The host_id value (the ? in the query) can be easily obtained by the URL of the target host's page (e.g. the 2 in https://server.fleetdm.com/hosts/2):

SELECT execution_id, activity_type FROM upcoming_activities WHERE host_id = ? AND activated_at IS NOT NULL;

Example:

mysql> SELECT execution_id, activity_type FROM upcoming_activities WHERE host_id = 2 AND activated_at IS NOT NULL;
+--------------------------------------+---------------+
| execution_id                         | activity_type |
+--------------------------------------+---------------+
| 54a0058d-0001-4756-8de9-309b88456c72 | script        |
+--------------------------------------+---------------+
1 row in set (0.00 sec)

2. Insert a bogus result

Based on the activity type, insert a fake result so that the blocking activity is not pending anymore. The following sub-sections show how to do this based on the activity type.

script activity type

Replace the ? with the execution_id (in single quotes) of the activity identified in the previous step:

UPDATE host_script_results 
SET exit_code = 1, output = 'fake result'
WHERE execution_id = ?;

And then delete the upcoming activity row, again replacing ? with the execution_id in single quotes:

DELETE FROM upcoming_activities WHERE execution_id = ?;

Example:

mysql> UPDATE host_script_results SET exit_code = 1, output = 'fake result' WHERE execution_id = '54a0058d-0001-4756-8de9-309b88456c72';
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> DELETE FROM upcoming_activities WHERE execution_id = '54a0058d-0001-4756-8de9-309b88456c72';
Query OK, 1 row affected (0.02 sec)

vpp_app_install activity type

For the vpp_app_install, we need to first get the uuid of the host, replace ? with the host's id:

SELECT uuid FROM hosts WHERE id = ?

With that host uuid and the execution_id of the activity, we have all that we need to insert a bogus result for the VPP app install, replace the first ? with the host's uuid and the second ? with the activity's execution_id:

INSERT INTO nano_command_results 
(id, command_uuid, status, result) 
VALUES
(?, ?, 'Error', '<?xml');

And then delete the upcoming activity row, again replacing ? with the execution_id in single quotes:

DELETE FROM upcoming_activities WHERE execution_id = ?;

Example:

INSERT INTO nano_command_results (id, command_uuid, status, result) VALUES ('2997C0F2-F576-4122-9B11-97C0CCC61BBC', 'b94e7ddd-1956-4eca-a7d4-ac0e1c46062b', 'Error', '<?xml');
Query OK, 1 row affected (0.04 sec)

mysql> DELETE FROM upcoming_activities WHERE execution_id = 'b94e7ddd-1956-4eca-a7d4-ac0e1c46062b';
Query OK, 1 row affected (0.01 sec)

software_install activity type

For the software install activity, we can simulate a result by setting the pre-condition query to an empty string, which will mark it as failed to install. Replace the ? with the activity's execution_id in single quotes:

UPDATE host_software_installs 
SET pre_install_query_output = '' 
WHERE execution_id = ?;

And then delete the upcoming activity row, again replacing ? with the execution_id in single quotes:

DELETE FROM upcoming_activities WHERE execution_id = ?;

Example:

mysql> UPDATE host_software_installs SET pre_install_query_output = '' WHERE execution_id = '0cb1ec54-f789-4eb3-8e55-6abf63d5d090';
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> DELETE FROM upcoming_activities WHERE execution_id = '0cb1ec54-f789-4eb3-8e55-6abf63d5d090';
Query OK, 1 row affected (0.02 sec)

software_uninstall activity type

We need to update two tables to insert a bogus result for uninstalls. First update the uninstall script result, replace ? with the activity's execution_id in single quotes:

UPDATE host_script_results 
SET exit_code = 1, output = 'fake result'
WHERE execution_id = ?;

Next update the software uninstall result, again replacing ? with the activity's execution_id in single quotes:

UPDATE host_software_installs
SET uninstall_script_output = 'fake result', uninstall_script_exit_code = 1
WHERE execution_id = ?;

And then delete the upcoming activity row, again replacing ? with the execution_id in single quotes:

DELETE FROM upcoming_activities WHERE execution_id = ?;

Example:

mysql> UPDATE host_script_results  SET exit_code = 1, output = 'fake result' WHERE execution_id = 'a85530c1-81ee-43a2-ad14-5d3f44573111';
Query OK, 1 row affected (0.02 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> UPDATE host_software_installs SET uninstall_script_output = 'fake result', uninstall_script_exit_code = 1 WHERE execution_id = 'a85530c1-81ee-43a2-ad14-5d3f44573111';
Query OK, 1 row affected (0.08 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> DELETE FROM upcoming_activities WHERE execution_id = 'a85530c1-81ee-43a2-ad14-5d3f44573111';
Query OK, 1 row affected (0.02 sec)

3. Send a bogus script to execute

This last step is necessary to ensure any subsequent upcoming activity gets processed, because until this point all we've done is mark the blocked one as done, this is not enough by itself to start running the next activity.

This can be done either via the UI (select an existing, idempotent/side-effect free script to run on this host, e.g. one that only executes echo "Hello world"), or perhaps easier if no such script is available in the UI, via fleetctl, which I'll show an example here:

$ fleetctl run-script --host host-display-name -- 'echo "hello"'

This will cause the next activity in the upcoming queue to be "activated" (processed, executed).

At this point the upcoming queue for that host should be unblocked.

[noahtalerman]

@mna thanks! I moved those instructions to a public Google doc here: https://docs.google.com/document/d/1yTC-CHHEV7eoFs0AQb1QxNo6fFafq8fk78SRCFeKank/edit?tab=t.0

@lukeheath @georgekarrv these steps to unblock stuck activities is complex! I think it's risk to wait until 4.67 to get to canceling upcoming activities. So, I think we want to prioritize this story so we can get to canceling as soon as we can (current sprint, 4.66, if possible).

Can we do it?

[lukeheath]

@noahtalerman Agreed, since it was a Q1 OKR we want to get to it as soon as possible.

Edit: Sorry, I didn't realize you were proposing disrupting the sprint. I'm inclined not do that, but I'm not sure what kind of impact it would have on the sprint. @georgekarrv what do you think?

[georgekarrv]

I think pulling this in for next sprint would be ideal. In the mean time making customer success aware of these ways to fix stuck issues for any customers that might bring them up would be a good first step.