Auto-YaRN for Qwen3? Maybe even set it dynamically?

[Downtown-Case]

It appears Qwen3 is using YaRN for long contexts, like Qwen 2.5:

The maximum context length in pre-training for Qwen3 models is 32,768 tokens. It can be extended to 131,072 tokens with RoPE scaling techniques. We have validated the performance with YaRN.

https://github.com/QwenLM/Qwen3/blob/main/docs/source/inference/transformers.md#enabling-long-context

Editing the config file for the desired context is... less than ideal for users.

However, they mention this little nugget:

Transformers implements static YaRN, which means the scaling factor remains constant regardless of input length, potentially impacting performance on shorter texts. We advise adding the rope_scaling configuration only when processing long contexts is required. It is also recommended to modify the factor as needed. For example, if the typical context length for your application is 65,536 tokens, it would be better to set factor as 2.0

The default factor is 4.0, and currently the exllama code will stick to that if the 'default' long context configuration is present:

exllamav3/exllamav3/util/rope.py

Lines 130 to 159 in 0ff9fd2

	def _rope_params_yarn(self):
	rs = self.rope_settings
	assert rs.max_position_embeddings is not None, \
	"YaRN scaling requires explicit max_position_embeddings"
	base = rs.rope_theta
	dim = int(rs.head_dim * rs.partial_rotary_factor)
	factor = rs.rope_scaling.get("factor")
	attn_factor = rs.rope_scaling.get("attention_factor", 0.1 * math.log(factor) + 1.0)
	beta_fast = rs.rope_scaling.get("beta_fast", 32)
	beta_slow = rs.rope_scaling.get("beta_slow", 1)
	def find_correction_dim(num_rotations):
	return (dim * math.log(rs.max_position_embeddings / (num_rotations * 2 * math.pi))) / (2 * math.log(base))
	def find_correction_range(low_rot, high_rot):
	_low = math.floor(find_correction_dim(low_rot))
	_high = math.ceil(find_correction_dim(high_rot))
	return max(_low, 0), min(_high, dim - 1)
	def linear_ramp_factor(_min, _max, _dim):
	if _min == _max:
	_max += 0.001
	linear_func = (torch.arange(_dim, dtype = torch.float32, device = self.device) - _min) / (_max - _min)
	ramp_func = torch.clamp(linear_func, 0, 1)
	return ramp_func
	pos_freqs = base ** (torch.arange(0, dim, 2, device = self.device).float() / dim)
	inv_freq_extrapolation = 1.0 / pos_freqs
	inv_freq_interpolation = 1.0 / (factor * pos_freqs)
	low, high = find_correction_range(beta_fast, beta_slow)
	inv_freq_extrapolation_factor = 1 - linear_ramp_factor(low, high, dim // 2).float()
	inv_freq = inv_freq_interpolation * (1 - inv_freq_extrapolation_factor)
	inv_freq += inv_freq_extrapolation * inv_freq_extrapolation_factor
	return inv_freq, attn_factor

But it appears a more optimal thing to do is ignore the rope_scaling factor, and statically set it with (context length of the loaded model)/original_max_position_embeddings, which would be 2.0 for 64K, 3.0 for 96K, and so on.

This is trivially easy to implement, but the question is if exllamav3 should set it automatically like that. I say it should, as most users aren't going to dig into how YaRN scaling works and match the scaling factor to their specified context length.

---

...But what would be even more optimal would be to dynamically set the scaling factor per request. For instance, if a 64K request was recieved, set it to 2.0. A 96K one, 3.0. So each request gets the optimal factor.

Even better would be to disable YaRN entirely for requests under 32K.

...Is this practical to do? I seem to remember it not being practical in exllamav2.

[turboderp]

The RoPE parameters has an override that could be used at load time, though it's currently only used for longrope:

@dataclass
class RopeSettings:
    head_dim: int = 128
    rope_theta: float = 10000.0
    rope_scaling: dict | None = None
    partial_rotary_factor: float = 1.0
    max_position_embeddings: int | None = None
    original_max_position_embeddings: int | None = None
    rope_style: RopeStyle = RopeStyle.NEOX
    override_max_position_embeddings: int | None = None

It's a total nightmare to deal with, though, honestly. Changing the scaling factor dynamically breaks just about every assumption made in the generator, and I'm not sure how to even approach stuff like a new set of frequencies per request, since the generator can batch together sequences of different length. It can also reuse cached values from a previous sequence, and that would have to go as well if position IDs have to be reinterpreted based on the overall max sequence length.

[Downtown-Case]

Ah, yeah, it's not worth breaking anything. Forget per-request dynamism. It doesn't seem that great past 32K it in my few API tests anyway.

How about a log warning? Maybe a print statement along the lines of "WARNING: A scaling factor of 2.0 is recommended for a context size of 65536!"

[Downtown-Case]

An interesting datapoint: https://huggingface.co/Qwen/Qwen3-32B/discussions/18

TL;DR: Qwen3 32B 3.45bpw, squeezed into 24GB, got a math/reasoning question from a 107K prompt! Exllamav2/Qwen2.5 (with YaRN) wasn't even close to that in my testing, though admittedly I never tested QwQ like that.

In other words, YaRN is surprisingly effective with Qwen3, and setting the correct scaling factor seems to be important.

[Lissanro]

I am interested in this as well, so decided to share my thoughts below in case they are useful.

what would be even more optimal would be to dynamically set the scaling factor per request. For instance, if a 64K request was recieved, set it to 2.0. A 96K one, 3.0. So each request gets the optimal factor.

I think request length does not include reply that will be generated, so setting it dynamically would need setting it after every generated token (but please correct me if I misunderstand how YaRN supposed to work with growing context while the model generates reply for the given request).

ignore the rope_scaling factor, and statically set it with (context length of the loaded model)/original_max_position_embeddings

I like this idea, but if it will be considered for an implementation and enabled by default, I suggest adding command-line option that allows to disable it if needed.

[turboderp]

Closing some stale issues. (I believe the YaRN implementation should be complete, although the K/V cache is static and RoPE scaling factors can't change dynamically in response to context length.)