[FR] Command to retro-actively apply history_filter on history

[xvello]

As discussed offline, i have a lot of kubectl delete lines in my history after operations on the dev cluster. I really want to avoid executing these on a prod cluster, so I'd like to make sure they are never shown in the search.

I set history_filter to avoid capturing more of these, but I'd really like to remove them from the DB. I am aware that one can remove commands with ad-hoc calls to atuin search --delete, but I feel like a command to apply the filters would give me the incentive to keep adding to history_filter and avoid a repeat.

I'm happy to implement this once we agree on the design for it. My suggestion:

• a new atuin history prune subcommand will scan the whole history, match history_filter on it, and issue the appropriate deletions
• the subcommand can be extended later to pass arbitrary patterns, but by default will use the values from the config
• as a destructive operation, we'll want to allow to preview the changes before they are executed (to make sure a buggy patterns matches everything). We could either:
  • add a unix-style -n / --dry-run argument and default to delete (easier)
  • list the commands and ask for confirmation before deleting (and add a -f / --force argument to skip the confirmation)
• the most naive implementation would reuse History::should_save, so that all filtering config is applied without increasing maintenance cost
  • as a once-a-quarter operation, I think it's fine to have it be CPU-intensive and slow. We could try delegating the filtering to the Database but it would add more complexity, and possibly bugs if the regexp behaviours don't align perfectly
  • we need to inspect every single history record. Calling Database::list with unique = true would give us this, at the cost of unbounded memory usage. It might be good enough for a first iteration
  • to keep the memory usage bounded, we'd need a new access method from Database that would allow to iterate or paginate. Alternatively, we can use Database::range on a user-provided timestamp range (default on last 7 days maybe?)

What are your thoughts @ellie?

[ellie]

Hey! Thanks for writing this one up 😊

a new atuin history prune subcommand will scan the whole history, match history_filter on it, and issue the appropriate deletions

I think that sounds great to me, both in description + name

the subcommand can be extended later to pass arbitrary patterns, but by default will use the values from the config

I'm not sure that would be the best idea, I'd rather keep deleting stuff elsewhere. Prune makes sense for enforcing config, and it would be great to keep the "edit your config" incentive

as a destructive operation, we'll want to allow to preview the changes before they are executed...

I think the unix style dry run makes sense, and would be familiar to users. Otherwise, I think stating how many records will be deleted + asking for confirmation would work, without requiring confirmation + listing them.

Kinda like zsh with an rm *

the most naive implementation...

Reusing should_save would be best for now. Unbounded memory use is generally not a good thing, though in the case of Atuin it's a premature optimisation to be concerned with at this point. Even the largest databases are only a few 10s of mb on disk, and less when loaded into memory. It's definitely something to sort in the future, but not something to be overly concerned with right now.

The only times when we actively need to consider this are for UI/search cases, but for maintenance operations I think this is totally reasonable.

Otherwise - pagination is something I could do with implementing anyway, for some stats/UI use cases. So don't worry about it here