Skip to content

Conversation

vasqu
Copy link
Contributor

@vasqu vasqu commented Sep 1, 2025

Rebase/merge didnt remove the logic in #40490 because they were done at 2 different lines in the code

cc @zucchini-nlp

):
mrope_section = config.rope_scaling["mrope_section"]
mutiplier = max(requested_dim // head_dim, 1)
config.rope_scaling = {"type": "default", "mrope_section": [i * mutiplier for i in mrope_section]}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other logic is a few lines down

# 3d rope also depends on the head dim
# (we assume easy shapes here where we get to the requested head dim at least)
if hasattr(config, "rope_scaling") and len(config.rope_scaling.get("mrope_section", None)) > 0:
scaling_factor = max(requested_dim // (sum(config.rope_scaling["mrope_section"]) * 2), 1)
config.rope_scaling["mrope_section"] = [
section * scaling_factor for section in config.rope_scaling["mrope_section"]
]

@vasqu
Copy link
Contributor Author

vasqu commented Sep 1, 2025

Seems like the failed tests are the same as in main, ie CI is currently broken

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Copy link
Member

@zucchini-nlp zucchini-nlp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, i blindly trusted GH rebase process 🙈

@vasqu vasqu merged commit abf5900 into main Sep 1, 2025
25 checks passed
@vasqu vasqu deleted the vas-clean-mrope-expansion branch September 1, 2025 15:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants