Skip to content

b11758

  • b11758
  • 83c2d01
  • Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
  • Choose a tag to compare

  • b11758
  • 83c2d01
  • Choose a tag to compare

  • Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
@petersid2022 petersid2022 tagged this 20 Oct 11:31
* spec : save the dynamic ngram cache file

* save the cache at the common_speculative_impl_ngram_cache destruction
* remove save_static/save_dynamic flags as well as the relevant TODO
* if the dynamic cache file does not exist, start with an empty cache
  that is written on exit. if it exists but cannot be read, abort
  instead of overwriting it.

* spec : fix draft_one not updating the ngram cache history correctly

* common_ngram_cache_update received only the new tokens with none of
  the tokens before them causing ngrams ending at the new token to not
  being formed, leading the context cache to only ever hold the prompt

* fix: pass the full history to common_ngram_cache_update with nnew set
  to the number of new tokens the way llama-lookup does and also reset
  cache_size to 0 on each new request

---------

Co-authored-by: Petros Sideris <petros.sideris@nokia.com>
Assets 2
Loading