Min P sampler implementation [alternative to Top P/Top K]

[kalomaze]

The way that this sampler works is:

• Every possible token has a probability percentage attached to it that we will be measuring for consideration.
• The base min p value represents the starting required percentage. (For example, 0.05 = only include tokens that are at least 5% probable)
• This gets scaled by the top token in the entire list's probability. So if your top token is 90%, then that 5% is multiplied by 0.9x (4.5%)
• So if the top token is 90% probable, and your base_min_p is set to 0.05, then only tokens that are at least 4.5% probable will be sampled from before temperature is applied.
• This method seems more effective at selecting the reasonable tokens compared to both Top P and Top K.

Top P has a design flaw in that numerous tail end tokens can be considered if the top tokens don't have concentrated enough scores to meet up to the specified Top P value, while TFS and other novel sampler approaches aren't as easily interpretable or consistent as Top P. The primary purpose of the Min P sampler is to accomodate for both of these design flaws.

The current implementation is very rough around the edges code-wise, as I am not very experienced with C++, but I hope to properly polish this implementation to be considered for merging. I have gotten improved results personally and positive feedback from other users, especially in regards to increased coherent creativity.

Mathematically, it is not as complex as TFS or other tail search algorithms, but importantly, it is easily understandable and in how it impacts the probabilities as a result. It is essentially a streamlined linear version of Top A in design. However, it consistently outperforms Top P and Top K for removing tail end tokens.

[kalomaze]

The current implementation:

• Checks if Top P is set to 0.02, and if it has this value, triggers an override to use Min P sampling
• Creates a .txt file with the base_min_p value, SamplerBaseMinP.txt if it doesn't already exist
• Loads the value from the .txt file and performs Min P calculations

This is of course suboptimal in a lot of ways, but when drafting sampler ideas, I wanted to avoid touching the sampler stack order as it currently existed before I found a solution. What would be the best way to integrate this if the objective was to avoid Top P and Top K's flaws via an improved single sampler, where it's not intended to be used in tandem with them? (Maybe they should be disabled like how Mirostat disables samplers when this is enabled?)

[kalomaze]

A comparison between Top P and Min P when faced with absurdly high temperature scaling (no prompt formatting or anything, so not ideal model conditions, just a quick test)

[ivanstepanovftw]

the order of the samplers became a command-line parameter

I like this. It is easy to implement -a 1 -b 2 --order A,B but it better be just -a 1 -b 2 in order to apply already applied sampler second time, i.e. -a 1 -b 2 -a 3.

Regarding defaults - I believe that all disabled samplers and penalties, with temperature == 1 is the best way to avoid token and context repetitions, and preserve statistically human-like prediction, because that shows model abilities.

[cebtenzzre]

I like this. It is easy to implement -a 1 -b 2 --order A,B but it better be just -a 1 -b 2 in order to apply already applied sampler second time, i.e. -a 1 -b 2 -a 3.

If we did something like that, the defaults would have to be cleared as soon as you override any of them. Which might be confusing if a user just wants to override e.g. top-p and leave the other parameters alone.

I've tried using models with minimal samplers (e.g. just min-p, or min-p and top-p=.9x), but I had to fall back on my favorite default preset of Midnight Enigma (temp=1 top-p=0.37 rep-pen=1.18 top-k=100) after the models I tried seemed to have a hard time staying on topic and remaining coherent.

[kalomaze]

I like this. It is easy to implement -a 1 -b 2 --order A,B but it better be just -a 1 -b 2 in order to apply already applied sampler second time, i.e. -a 1 -b 2 -a 3.

If we did something like that, the defaults would have to be cleared as soon as you override any of them. Which might be confusing if a user just wants to override e.g. top-p and leave the other parameters alone.

I've tried using models with minimal samplers (e.g. just min-p, or min-p and top-p=.9x), but I had to fall back on my favorite default preset of Midnight Enigma (temp=1 top-p=0.37 rep-pen=1.18 top-k=100) after the models I tried seemed to have a hard time staying on topic and remaining coherent.

Top P 0.37 seems aggressively deterministic to me, you're picking like two or three choices max 99% of the time. Your repetition penalty is pretty high as well, which probably helps counteract the determinism, but I try to avoid rep pen because it's a bit of a 'hacky' solution to the problem of overly high determinism compared to turning up Temp.

I would try lowering Rep Pen a bit, turning Top P off (to 1.0), and using a Min P that is on the deterministic side. (e.g 0.25 Min P) to get similar effects to what you want.

[cebtenzzre]

In my experience, the TGWUI preset called "Midnight Enigma" (possibly with top-p increased to .57) is very good for Alpaca-style instruction-based prompting with certain models. Free-form prompting with models like Chronoboros does seem to benefit from using e.g. min-p=.25 instead.

[ivanstepanovftw]

rep-pen=1.18

Some languages does not have full word tokens, and you will penalize sub words or characters.

[ZoomRmc]

Having experimented with using strictly min-p sampler (all others turned off) for creative but structured writing with llama3-7b, I see very little sense in using the currently default order (temperature last).

• --samplers min_p;temperature (current default if top-k and top-p turned off)
  In this configuration, it seems there's very little direct influence on the creativity by setting various temp values. The output follows the prompted structure correctly and initially generates rather cohesive output well into the very high temp values (t:4.0 min-p: 0.01) until some extremely unlikely garbage tokens slip through which immediately destabilizes the output.
  In other words, I couldn't achieve any gradual control over the creativity of the output. Moreover, tweaking min-p alone doesn't help with changing the creativity much, abruptly starting to produce gibberish.
• --samplers temperature;min_p
  This seems far more controllable. The gradual increase of the temperature up to the extreme values work as you'd expect: the output follows the bland>unremarkable>creative>typo-ridden>broken grammar>unstable curve in a more or less predictable way.
  On the other hand, this sampler order is more interdependent and makes it easier to subtly influence the results in an unintended way.

My impression after a brief testing: Current order defaults certainly provide more deterministic output for most cases, including uninformed tweaking of the sampler settings, but probably limit the user's control.

Relevant: #4091

[ivanstepanovftw]

Also NMS (Non Max Suppression) from image object detection task uses probability threshold too, similarly to Min P.