Add GreedyLR adaptive learning rate scheduler by balak4 · Pull Request #44271 · huggingface/transformers

balak4 · 2026-02-25T01:40:57Z

Summary

Add GreedyLR, a metric-based adaptive learning rate scheduler that adjusts the learning rate during training based on the current loss
Based on "Dynamic Learning Rate Scheduling based on Loss Changes Leads to Faster Convergence"
Integrated with Trainer following the existing ReduceLROnPlateau pattern

Changes

optimization.py: Add GreedyLR class, StreamingAverage helper, and get_greedy_schedule() factory
trainer.py: Add GreedyLR alongside ReduceLROnPlateau for metric-based scheduler stepping
training_args.py: Add validation requiring eval strategy for greedy scheduler
trainer_utils.py: Add GREEDY to SchedulerType enum
__init__.py: Export GreedyLR and get_greedy_schedule
tests/optimization/test_greedy_lr.py: Comprehensive test suite (30 tests)
docs/source/en/main_classes/optimizer_schedules.md: Autodoc entries
examples/greedy-lr/: Example script and documentation

How it works

GreedyLR adaptively adjusts the learning rate during training based on the current loss:

Increases LR (divides by factor) when loss improves beyond patience
Decreases LR (multiplies by factor) when loss stops improving beyond patience
Supports warmup/cooldown periods, metric smoothing via streaming average, and automatic reset when LR hits min

Usage

training_args = TrainingArguments(
    lr_scheduler_type="greedy",
    lr_scheduler_kwargs={"patience": 10, "factor": 0.95, "min_lr": 1e-5},
    eval_strategy="steps",
    eval_steps=200,
    ...
)

Test plan

pytest tests/optimization/test_greedy_lr.py — 30/30 pass
pytest tests/optimization/test_optimization.py — 4/4 pass (no regressions)
make check-repo — no new errors introduced
from transformers import GreedyLR, get_greedy_schedule works

balak4 · 2026-02-25T16:28:17Z

@Rocketknight1 could you provide specific feedback on what led to closing this? I'd like to address any concerns.

For context: I'm one of the authors of this paper from our team at AWS (https://arxiv.org/abs/2512.14527). GreedyLR is a new adaptive LR scheduler that adjusts the learning rate during training based on the current loss. We validated it across NLP, CV, and LLM tasks up to 7B parameters, showing faster convergence without performance loss for both fine-tuning and pre-training.

The implementation follows the existing ReduceLROnPlateau integration pattern, includes 30 unit tests, and has been E2E tested on single and multi-GPU setups on AWS. All CI checks pass.

Happy to address any specific feedback or make changes to better fit the library's conventions.

cc: @w601sxs @pranavvm26 (co-authors)

Rocketknight1 · 2026-02-26T14:23:18Z

Hi @balak4, I'm sorry! We've been getting swamped with code agent PRs lately, and so we've been trying to filter out code agent PRs from first-time contributors because a lot of them are low-quality spam and our reviewer time is too limited to read them all in detail.

In this case the PR was more legitimate than I realized at first - I'll reopen it. Sorry for the inconvenience!

balak4 · 2026-02-26T16:47:16Z

Hi @balak4, I'm sorry! We've been getting swamped with code agent PRs lately, and so we've been trying to filter out code agent PRs from first-time contributors because a lot of them are low-quality spam and our reviewer time is too limited to read them all in detail.

In this case the PR was more legitimate than I realized at first - I'll reopen it. Sorry for the inconvenience!

@Rocketknight1 Gotcha, I understand, thanks for re-opening. Please let me know if you have any questions / need clarification. We're eager to share this work with the community!

balak4 · 2026-03-09T18:08:27Z

Hey @Rocketknight1 , checking in since it's been 2 weeks since we submitted this PR - any update on when the PR could be reviewed?

Rocketknight1 · 2026-03-10T12:10:33Z

Ah, sorry! I should have pinged a reviewer when I reopened it. cc @SunMarc!

SunMarc

Thanks, just a few ntis

SunMarc · 2026-03-10T14:34:00Z

examples/scheduler/README.md

+# GreedyLR: Adaptive Learning Rate Scheduler
+
+GreedyLR monitors training metrics and adaptively adjusts the learning rate -- increasing when improving, decreasing when plateauing. It works for both pre-training and fine-tuning.
+
+## Paper
+
+> **GreedyLR: A Novel Adaptive Learning Rate Scheduler**
+>
+> Despite significant advances in optimizers for training, most research works use common scheduler choices like Cosine or exponential decay. In this paper, we study GreedyLR, a novel scheduler that adaptively adjusts the learning rate during training based on the current loss. To validate the effectiveness of our proposed scheduler, we conduct experiments on several NLP, CV, and LLM tasks with up to 7B parameters, including both fine-tuning and pretraining experiments. The results show that our approach outperforms several state-of-the-art schedulers in terms of accuracy, speed, and convergence.
+


maybe create a new folder called examples/scheduler and put that there ? It will be better ! Also, maybe it will be interesting to have a section for scheduler in trainer docs cc @stevhliu

SunMarc · 2026-03-10T14:34:15Z

examples/greedy-lr/.gitignore

+CLAUDE.md
+archive/


SunMarc · 2026-03-10T14:40:57Z

tests/optimization/test_greedy_lr.py

+# Copyright 2026 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import unittest
+
+from transformers import is_torch_available
+from transformers.testing_utils import require_torch
+


add some trainer tests also to make sure the integration is correctly done. You need to update the test_trainer_optimizers.py file

balak4 · 2026-03-11T20:18:41Z

Thanks, just a few ntis

@SunMarc thanks for the feedback! Will get these addressed

Add GreedyLR, a metric-based scheduler that increases LR on improvement and decreases on plateau, based on arxiv.org/abs/2512.14527. - Add GreedyLR class and get_greedy_schedule() to optimization.py - Add StreamingAverage helper for metric smoothing - Integrate with Trainer via ReduceLROnPlateau-style metric stepping - Add GREEDY to SchedulerType enum and TrainingArguments validation - Add comprehensive tests in tests/optimization/test_greedy_lr.py - Add example script and documentation

…ler, delete .gitignore, add trainer integration tests

balak4 · 2026-03-11T22:15:32Z

@SunMarc comments addressed. Please review and let me know if there's anything else you'd like updated.

cc: @Rocketknight1

balak4 · 2026-03-17T22:06:09Z

@SunMarc @Rocketknight1 hope you're well, just a friendly follow-up. Are we good to get this PR merged now that the comments are addressed?

SunMarc

Thanks a lot ! Sorry for the delay

HuggingFaceDocBuilderDev · 2026-03-18T14:30:26Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

balak4 force-pushed the add-greedylr-scheduler branch from dcd2311 to 1d70c35 Compare February 25, 2026 02:20

redpanda1995 approved these changes Feb 25, 2026

View reviewed changes

Rocketknight1 closed this Feb 25, 2026

Rocketknight1 added the Code agent slop label Feb 25, 2026

Rocketknight1 reopened this Feb 26, 2026

Rocketknight1 removed the Code agent slop label Feb 26, 2026

SunMarc reviewed Mar 10, 2026

View reviewed changes

balak4 added 2 commits March 11, 2026 13:55

Address review comments: rename examples/greedy-lr to examples/schedu…

4e608f1

…ler, delete .gitignore, add trainer integration tests

balak4 force-pushed the add-greedylr-scheduler branch from cadcd23 to 4e608f1 Compare March 11, 2026 22:13

SunMarc approved these changes Mar 18, 2026

View reviewed changes

SunMarc enabled auto-merge March 18, 2026 14:20

SunMarc added this pull request to the merge queue Mar 18, 2026

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Mar 18, 2026

Rocketknight1 added this pull request to the merge queue Mar 18, 2026

Merged via the queue into huggingface:main with commit 981ca7b Mar 18, 2026
28 checks passed

stevhliu mentioned this pull request Mar 18, 2026

[docs] optimizers, hyperparam search, training features #44290

Open

		CLAUDE.md
		archive/

Conversation

balak4 commented Feb 25, 2026

Summary

Changes

How it works

Usage

Test plan

Uh oh!

balak4 commented Feb 25, 2026

Uh oh!

Rocketknight1 commented Feb 26, 2026

Uh oh!

balak4 commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

balak4 commented Mar 9, 2026

Uh oh!

Rocketknight1 commented Mar 10, 2026

Uh oh!

SunMarc left a comment

Choose a reason for hiding this comment

Uh oh!

SunMarc Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

SunMarc Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

SunMarc Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

balak4 commented Mar 11, 2026

Uh oh!

balak4 commented Mar 11, 2026

Uh oh!

balak4 commented Mar 17, 2026

Uh oh!

SunMarc left a comment

Choose a reason for hiding this comment

Uh oh!

HuggingFaceDocBuilderDev commented Mar 18, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

balak4 commented Feb 26, 2026 •

edited

Loading