readme : add link to Autopen under UIs #11684
Conversation
Autopen (https://github.com/blackhole89/autopen) is a graphical text editor that uses llama.cpp to tokenize the buffer on the fly, score the buffer, visualise token logits and allow you to switch back and forth between different possible completions at any point. It hopefully meets the criteria for inclusion, as the dependency on llama.cpp is stated prominently.
Indeed there are some subtle effects that lead to differences when processing the same input with different batch sizes (see for example #7745). Not clear how to make this fully deterministic. There are at least 2 points in the computation that are problematic in this regard:
|
|
Ah, good to know that it's a known problem. I've observed discrepancies on the order of 0.5 logit units (usually less), though my use case may be a bit pathological since I frequently reevaluate the same run of ~10 tokens (the window between two snapshots, with default params, in my program) batched up differently, in a context where the ordering of two choices suddenly flipping is quite conspicuous. Thanks for putting in my link! |
Autopen (https://github.com/blackhole89/autopen) is a graphical text editor that uses llama.cpp to tokenize the buffer on the fly, score the buffer, visualise token logits and allow you to switch back and forth between different possible completions at any point. It hopefully meets the criteria for inclusion, as the dependency on llama.cpp is stated prominently.
Autopen (https://github.com/blackhole89/autopen) is a graphical text editor that uses llama.cpp to tokenize the buffer on the fly, score the buffer, visualise token logits and allow you to switch back and forth between different possible completions at any point. It hopefully meets the criteria for inclusion, as the dependency on llama.cpp is stated prominently.
Autopen (https://github.com/blackhole89/autopen) is a graphical text editor that uses llama.cpp to tokenize the buffer on the fly, score the buffer, visualise token logits and allow you to switch back and forth between different possible completions at any point. It hopefully meets the criteria for inclusion, as the dependency on llama.cpp is stated prominently.
Autopen (https://github.com/blackhole89/autopen) is a graphical text editor that uses llama.cpp to tokenize the buffer on the fly, score the buffer, visualise token logits and allow you to switch back and forth between different possible completions at any point. It hopefully meets the criteria for inclusion, as the dependency on llama.cpp is stated prominently.
3 participants
Sorry that after all the months of AWOL this triviality is all I come back with, but I'd be grateful if my little project could be added to the README list.
Autopen is a graphical text editor that uses llama.cpp to tokenize the buffer on the fly, score the buffer, visualise token logits and allow you to switch back and forth between different possible completions at any point. There's a demo video here. I hope the criteria for inclusion are met, as I'm stating the dependency prominently.
(I think I might also have found some subtle issues (of numerical stability, snapshot save/restore, and/or the CUDA kernels) in the process of working on it, where different sequences of decode calls that should give identical results don't, but I'm still working on isolating the exact conditions in a way that is actionable.)