v1.7.0: Memories compaction

Memories compact themselves

Every write to a memory adds fragments, and a memory written turn by turn piles up thousands of them. Recall slows down as they accumulate, and a remote memory heads for the Hub's limit of 10k files per folder. funes now compacts a memory: it merges the small fragments, adds their rows to the search indexes, and deletes the old versions.

• funes index compacts the local memory at the end of each run that writes, and deletes the versions older than ten minutes. Past 256 small indexed fragments, it rewrites them all.
• funes push compacts the remote once 500 rows sit unindexed there, and deletes every old version. On a real Hub memory, the first compaction went from 1,111 manifests to 2, and from 1,534 data files to 293.
• funes compact [memory] runs the same steps on demand, on your local memory, a local path, or a remote memory (it needs write access).

funes push --compact forces a compaction. --force-reindex is its old name and still works.

Also

• funes recall --role and the MCP role parameter restrict a search to one role, as the turns carry it. In Claude's sessions tool output arrives on user turns, so the user's own words are --role user --type text.
• The writer lock lives in the memory (<memory>/store.lock), so writers to one memory exclude each other whatever funes home they run from.

Upgrading from 1.6.x

No reindex. Update with funes update: funes 1.6 locks the memory from the funes home, so if one of its index runs is still going, the update waits for it to finish.

The first index after the upgrade (a hook's or yours) can take longer on a memory grown turn by turn, since its first compaction rewrites every fragment. A remote memory compacts at its next push past 500 unindexed rows. To do it now, run funes compact <org>/<repo>. For a remote over 10 GB, hf download it first, so the compaction reads from the cache instead of fetching its files one at a time (docs/index.md).

What's Changed

• Allow to filter recall by role by @dacorvo in #214
• Compact local and remote datasets for faster recall by @dacorvo in #216
• Fix the first cleanup of a large remote memory by @dacorvo in #217
• Add "funes compact" by @dacorvo in #218

Full Changelog: v1.6.0...v1.7.0