Commits on Dec 20, 2022

1. [Fleet] refactored bulk update tags retry (elastic#147594) …

   ## Summary
   
   Fixes elastic#144161
   
   As discussed
   [here](elastic#144161 (comment)),
   the existing implementation of update tags doesn't work well with real
   agents, as there are many conflicts with checkin, even when trying to
   add/remove one tag.
   Refactored the logic to make retries more efficient:
   - Instead of aborting the whole bulk action on conflicts, changed the
   conflict strategy to 'proceed'. This means, if an action of 50k agents
   has 1k conflicts, not all 50k is retried, but only the 1k conflicts,
   this makes it less likely to conflict on retry.
   - Because of this, on retry we have to know which agents don't yet have
   the tag added/removed. For this, added an additional filter to the
   `updateByQuery` request. Only adding the filter if there is exactly one
   `tagsToAdd` or one `tagsToRemove`. This is the main use case from the
   UI, and handling other cases would complicate the logic more (each
   additional tag to add/remove would result in another OR query, which
   would match more agents, making conflicts more likely).
   - Added this additional query on the initial request as well (not only
   retries) to save on unnecessary work e.g. if the user tries to add a tag
   on 50k agents, but 48k already have it, it is enough to update the
   remaining 2k agents.
   - This improvement has the effect that 'Agent activity' shows the real
   updated agent count, not the total selected. I think this is not really
   a problem for update tags.
   - Cleaned up some of the UI logic, because the conflicts are fully
   handled now on the backend.
   - Locally I couldn't reproduce the conflict with agent checkins, even
   with 1k horde agents. I'll try to test in cloud with more real agents.
   
   To verify:
   - Enroll 50k agents (I used 50k with create_agents script, and 1k with
   horde). Enroll 50k with horde if possible.
   - Select all on UI and try to add/remove one or more tags
   - Expect the changes to propagate quickly (up to 1m). It might take a
   few refreshes to see the result on agent list and tags list, because the
   UI polls the agents every 30s. It is expected that the tags list
   temporarily shows incorrect data because the action is async.
   
   E.g. removed `test3` tag and added `add` tag quickly:
   <img width="1776" alt="image"
   src="https://user-images.githubusercontent.com/90178898/207824481-411f0f70-d7e8-42a6-b73f-ed80e77b7700.png">
   <img width="422" alt="image"
   src="https://user-images.githubusercontent.com/90178898/207824550-582d43fc-87db-45e1-ba58-15915447fefd.png">
   
   The logs show the details of how many `version_conflicts` were there,
   and it decreased with retries.
   
   ```
   [2022-12-15T10:32:12.937+01:00][INFO ][plugins.fleet] Running action asynchronously, actionId: 90acd54-19ac-4738-b3d3-db32789233de, total agents: 52000
   [2022-12-15T10:32:12.981+01:00][INFO ][plugins.fleet] Scheduling task fleet:update_agent_tags:retry:check:90acd541-19ac-4738-b3d3-db32789233de
   [2022-12-15T10:32:16.477+01:00][INFO ][plugins.fleet] Running action asynchronously, actionId: 29e9da7-7194-4e52-8004-2c1b19f6dfd5, total agents: 52000
   [2022-12-15T10:32:16.537+01:00][INFO ][plugins.fleet] Scheduling task fleet:update_agent_tags:retry:check:29e9da70-7194-4e52-8004-2c1b19f6dfd5
   [2022-12-15T10:32:22.893+01:00][DEBUG][plugins.fleet] {"took":9886,"timed_out":false,"total":52000,"updated":41143,"deleted":0,"batches":52,"version_conflicts":10857,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1,"throttled_until_millis":0,"failures":[]}
   [2022-12-15T10:32:26.066+01:00][DEBUG][plugins.fleet] {"took":9518,"timed_out":false,"total":52000,"updated":25755,"deleted":0,"batches":52,"version_conflicts":26245,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1,"throttled_until_millis":0,"failures":[]}
   [2022-12-15T10:32:27.401+01:00][ERROR][plugins.fleet] Action failed: version conflict of 10857 agents
   [2022-12-15T10:32:27.461+01:00][INFO ][plugins.fleet] Scheduling task fleet:update_agent_tags:retry:90acd541-19ac-4738-b3d3-db32789233de
   [2022-12-15T10:32:27.462+01:00][INFO ][plugins.fleet] Retrying in task: fleet:update_agent_tags:retry:90acd541-19ac-4738-b3d3-db32789233de
   [2022-12-15T10:32:29.274+01:00][ERROR][plugins.fleet] Action failed: version conflict of 26245 agents
   [2022-12-15T10:32:29.353+01:00][INFO ][plugins.fleet] Scheduling task fleet:update_agent_tags:retry:29e9da70-7194-4e52-8004-2c1b19f6dfd5
   [2022-12-15T10:32:29.353+01:00][INFO ][plugins.fleet] Retrying in task: fleet:update_agent_tags:retry:29e9da70-7194-4e52-8004-2c1b19f6dfd5
   [2022-12-15T10:32:31.480+01:00][INFO ][plugins.fleet] Running bulk action retry task
   [2022-12-15T10:32:31.481+01:00][DEBUG][plugins.fleet] Retry elastic#1 of task fleet:update_agent_tags:retry:90acd541-19ac-4738-b3d3-db32789233de
   [2022-12-15T10:32:31.481+01:00][INFO ][plugins.fleet] Running action asynchronously, actionId: 90acd54-19ac-4738-b3d3-db32789233de, total agents: 52000
   [2022-12-15T10:32:31.481+01:00][INFO ][plugins.fleet] Completed bulk action retry task
   [2022-12-15T10:32:31.485+01:00][INFO ][plugins.fleet] Scheduling task fleet:update_agent_tags:retry:check:90acd541-19ac-4738-b3d3-db32789233de
   [2022-12-15T10:32:33.841+01:00][DEBUG][plugins.fleet] {"took":2347,"timed_out":false,"total":10857,"updated":9857,"deleted":0,"batches":11,"version_conflicts":1000,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1,"throttled_until_millis":0,"failures":[]}
   [2022-12-15T10:32:34.556+01:00][INFO ][plugins.fleet] Running bulk action retry task
   [2022-12-15T10:32:34.557+01:00][DEBUG][plugins.fleet] Retry elastic#1 of task fleet:update_agent_tags:retry:29e9da70-7194-4e52-8004-2c1b19f6dfd5
   [2022-12-15T10:32:34.557+01:00][INFO ][plugins.fleet] Running action asynchronously, actionId: 29e9da7-7194-4e52-8004-2c1b19f6dfd5, total agents: 52000
   [2022-12-15T10:32:34.557+01:00][INFO ][plugins.fleet] Completed bulk action retry task
   [2022-12-15T10:32:34.560+01:00][INFO ][plugins.fleet] Scheduling task fleet:update_agent_tags:retry:check:29e9da70-7194-4e52-8004-2c1b19f6dfd5
   [2022-12-15T10:32:35.388+01:00][ERROR][plugins.fleet] Retry elastic#1 of task fleet:update_agent_tags:retry:90acd541-19ac-4738-b3d3-db32789233de failed: version conflict of 1000 agents
   [2022-12-15T10:32:35.468+01:00][INFO ][plugins.fleet] Scheduling task fleet:update_agent_tags:retry:90acd541-19ac-4738-b3d3-db32789233de
   [2022-12-15T10:32:35.468+01:00][INFO ][plugins.fleet] Retrying in task: fleet:update_agent_tags:retry:90acd541-19ac-4738-b3d3-db32789233de
   {"took":5509,"timed_out":false,"total":26245,"updated":26245,"deleted":0,"batches":27,"version_conflicts":0,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1,"throttled_until_millis":0,"failures":[]}
   [2022-12-15T10:32:42.722+01:00][INFO ][plugins.fleet] processed 26245 agents, took 5509ms
   [2022-12-15T10:32:42.723+01:00][INFO ][plugins.fleet] Removing task fleet:update_agent_tags:retry:check:29e9da70-7194-4e52-8004-2c1b19f6dfd5
   [2022-12-15T10:32:46.705+01:00][INFO ][plugins.fleet] Running bulk action retry task
   [2022-12-15T10:32:46.706+01:00][DEBUG][plugins.fleet] Retry elastic#2 of task fleet:update_agent_tags:retry:90acd541-19ac-4738-b3d3-db32789233de
   [2022-12-15T10:32:46.707+01:00][INFO ][plugins.fleet] Running action asynchronously, actionId: 90acd54-19ac-4738-b3d3-db32789233de, total agents: 52000
   [2022-12-15T10:32:46.707+01:00][INFO ][plugins.fleet] Completed bulk action retry task
   [2022-12-15T10:32:46.711+01:00][INFO ][plugins.fleet] Scheduling task fleet:update_agent_tags:retry:check:90acd541-19ac-4738-b3d3-db32789233de
   [2022-12-15T10:32:47.099+01:00][DEBUG][plugins.fleet] {"took":379,"timed_out":false,"total":1000,"updated":1000,"deleted":0,"batches":1,"version_conflicts":0,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1,"throttled_until_millis":0,"failures":[]}
   [2022-12-15T10:32:47.623+01:00][INFO ][plugins.fleet] processed 1000 agents, took 379ms
   [2022-12-15T10:32:47.623+01:00][INFO ][plugins.fleet] Removing task fleet:update_agent_tags:retry:check:90acd541-19ac-4738-b3d3-db32789233de
   ```
   
   ### Checklist
   
   - [x] [Unit or functional
   tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
   were updated or added to match the most common scenarios
   
   Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
   (cherry picked from commit 687987a)

   

   juliaElastic committed Dec 20, 2022
   Configuration menu

   • View commit details
   • Copy full SHA for 6e620a1
   • Browse repository at this point

   Copy the full SHA

   6e620a1 View commit details

   Browse the repository at this point in the history