fix: correct auto ntk scaling_factor for 4k ctx case

[sandwichdoge]

The correct NTK alpha (at least for llama2) isn't actually that linear. For example if you want 8192 context size, the most optimal value is around 2.6, not 2.

[AlpinDale]

Thanks for the contribution! Please run ./formatting.sh --all to properly format the code, then check the pylint output for minor issues that need to be resolved.

[sandwichdoge]

Alright it's done. I had a lot of trouble running the formatting.sh script on Kubuntu 23 btw. It demanded very specific versions of dependencies, and it threw an error like this:
ImportError: cannot import name 'formatargspec' from 'inspect' (/usr/lib/python3.11/inspect.py)
Which I had to install this to fix:
pip3 install wrapt==1.14.0

[AlpinDale]

Ah sorry, I forgot to mention you'll need to run pip install -r requirements-dev.txt first. I should probably add a contributor readme file. I'll test this PR in a bit and get back to you.

[AlpinDale]

LGTM, thanks for the contribution

[Pyroserenus]

@sandwichdoge sorry to dig this up from the grave, but are you sure this is implemented correctly? I have a user situation where 4x scaling is breaking with this automatic implementation but not with dynamic alpha 2 being manually set.

For context, dynamic NTK doesn't work quite the same way as standard NTK alpha scaling. I had just assumed that this was suboptimal when i decided to rescind my concerns. https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases/

[1Tomber]

I add my findings to get to 16k:
I download the model (tested psyconic 20b and xwin lewd 13b) manually (not using the hf_downloader) and start it via this command line

python -m aphrodite.endpoints.kobold.api_server --port 5000 --host 127.0.0.1 --max-log-len 0 --quantization gptq --dtype float16 --model /workspace/models/TheBloke_psyonic-cetacean-20B-GPTQ --served-model-name jebcarter/psyonic-cetacean-20B --gpu-memory-utilization 0.90 --download-dir /workspace/models --max-model-len 16384

On a full context I get non coherent outputs, stories ending abruptly, output asking me how to continue, gibberish, misplaced words and double quotes in dialogues.

When I change the config.json to "rope_scaling": { "factor": 4.0, "type": "dynamic"},, same command line, the output behaves much more coherent and continues the story in its context (I can't benchmark the quality, but it feels much more coherent).

[AlpinDale]

Perhaps we can only use this scaling for when the user-specified max-model-len is 8192.

[sandwichdoge]

@Pyroserenus I see, looks like we're using the dynamically-scaled NTK version and not the standard NTK RoPE. I dug a little deeper and indeed we're using the DynamicNTKScalingRotaryEmbedding method, which already dynamically calculates the alpha value for us.

elif scaling_type == "dynamic":
    rotary_emb = DynamicNTKScalingRotaryEmbedding(
        head_size, rotary_dim, max_position, base, is_neox_style,
        scaling_factor)

And the target length calculation was made as simple as the original max position multiplied by the scaling factor parameter in the implementation!

max_len = self.max_position_embeddings * self.scaling_factor
base = self.base * (
    (self.scaling_factor * max_len / self.max_position_embeddings) -
    (self.scaling_factor - 1))**(self.rotary_dim /
                                 (self.rotary_dim - 2))

For standard NTK scaling, if you scale the factor a little too high, your general perplexity will suffer a bit but you won't get total gibberish. For dynamic NTK becomes a problem indeed.

I can at least confirm the scaling factor is as simple as target_max_len / max_position_embeddings now.

@AlpinDale I think github supports easy PR reverse. Please revert this change.

[AlpinDale]

Opened #149
will merge after someone (or myself) runs tests