Repository navigation
TUFF 9.4.1
TUFF 9.4.1 fixes two bugs: the reload prompt that kept showing up after ordinary answers, and the Background API ignoring the context size set in Settings.
- No more "Settings changed. Reload the model to continue." after a simple answer (#54). With Auto on, TUFF re-planned the model's memory after every answer and changed the settings of the model that was already loaded, so it asked for a reload even though nothing had been changed. A memory warning, or any app on the Mac swapping a little during an answer, was enough to set it off. Auto still learns from every answer, but its new plan now waits for the next time you load the model. Paging under 64 MB no longer counts against a plan, and if every plan has been seen under pressure, Auto plans as if nothing had been observed instead of failing to load. Thanks to @arker2 for the report and @Osiriss664 for tracking down the cause.
- The Background API and
tuff serveuse each model's memory settings from Settings > Models (#55). The server always planned with Auto's Balanced profile, which capped most models at 16K. Now, with Auto on, it uses the profile you picked. With Auto off, it uses the Context you chose, as long as that fits your Mac's memory, and otherwise falls back to Balanced./v1/modelsreports the new limit. On a 16 GB M2 Air with Ornith 1.5 9B set to 64K, a 31K-token prompt that used to be refused now gets an answer. Changes apply the next time the server loads the model. Thanks to @betamuh for the detailed report and patch.