Customizable Logit Warping Strategies for Generation #40010 by PamelaBha · Pull Request #40403 · huggingface/transformers

PamelaBha · 2025-08-23T23:54:15Z

What does this PR do?

Feature request
Improve the generate() API by supporting custom, declarative logit warping strategies. Make it easier for users to plug in standard and custom LogitProcessors via configuration or arguments without needing to subclass or dive into internals.

Motivation
The generation module already supports rich logit manipulation through LogitProcessorList, but:

It is undocumented and hard to use for casual users
Requires advanced subclassing to customize behaviors (e.g., word bans, domain constraints)
Doesn’t support JSON- or dict-style configuration like many other parts of Transformers
Making logit warping more accessible enables:

Prompt engineers and power users to fine-tune generation behavior
Safer generation via blacklists or probability shifting
Dynamic controls like repetition penalties or temperature annealing

Fixes # (issue): 40010

Before submitting

[No ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[ Yes] Did you read the contributor guideline,
Pull Request section?
[ Yes] Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case. Customizable Logit Warping Strategies for Generation #40010
[ Yes] Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
[ Yes] Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Rocketknight1 · 2025-08-25T14:11:21Z

cc @gante

gante

Very cool PR! I like where this feature is going 🔥

I've added a few comments, but I think they are simple to solve 🤗 Let me know if you have questions/counter-proposals regarding my comments.

Other global comments:

Missing: documentation of logits_processor in GenerationConfig. I would add a link to the example file inside the docs, your examples are quite clear!
Suggestion: in the documentation of logits_processor in generate(), I would add a mention about the new format (and a link to the examples)
Some classes should be directly imported from the top level, i.e. we should be able to do from transformers import LogitProcessorRegistry. This is done by adding the classes to the outermost __init__.py. I think the only class we need to add there is LogitProcessorRegistry (we can already to from transformers import LogitsProcessor)

gante · 2025-08-26T10:16:27Z


 if is_torch_available():
-    from .logits_process import SynthIDTextWatermarkLogitsProcessor, WatermarkLogitsProcessor
+    from ..cache_utils import (


(this diff seems related to a merge conflict, we can probably remove these added lines)

gante · 2025-08-26T10:18:00Z

+
+
+


Suggested change

gante · 2025-08-26T10:21:21Z

+        # Special handling for logit_processors - serialize to JSON string
+        if "logit_processors" in output and output["logit_processors"] is not None:
+            if not isinstance(output["logit_processors"], str):
+                # Convert to JSON string
+                output["logit_processors"] = json.dumps(output["logit_processors"])


Is this needed? if self.logit_processors is a list of dicts, then returning it directly should be fine :)

gante · 2025-08-26T10:25:27Z

            config._original_object_hash = hash(config)  # Hash to detect whether the instance was modified
            return config

+    def get_logit_processors(self):


Can we move this function to utils.py?

Goal: let's keep classes as independent as possible -- the generation config doesn't need to know about the expected format that logit_processors takes in generate. It's mostly a data storage class.

gante · 2025-08-26T10:38:14Z

+
+# Enhanced LogitsProcessorList
+class ConfigurableLogitsProcessorList(LogitsProcessorList):
+    """Extended LogitsProcessorList that supports configuration-based construction."""


missing: an example in the docstring

gante · 2025-08-26T10:39:36Z

+
+        # Add any directly passed processors last
+        if logits_processor is not None:
+            processors.extend(logits_processor)


merging with the custom logits_processor already done in self._merge_criteria_processor_list call a few lines above

gante · 2025-08-26T10:40:22Z

+        if configured_processors is not None:
+            # Check for duplicates and warn
+            existing_types = {type(p).__name__ for p in processors}
+            config_types = {type(p).__name__ for p in configured_processors}
+            duplicates = existing_types & config_types
+
+            if duplicates:
+                warnings.warn(
+                    f"Duplicate LogitProcessors detected: {duplicates}. "
+                    f"Configured processors will be added after standard ones."
+                )
+
+            processors.extend(configured_processors)


Perhaps we can reuse the logic in _merge_criteria_processor_list ?

gante · 2025-08-26T10:41:45Z

        self.assertTrue(delay_pattern_processor.active_batches.all())
        self.assertTrue((delay_pattern_processor.delay_pattern == torch.tensor(delay_pattern) - 1).all())
+
+class TestLogitProcessorRegistry(unittest.TestCase):


Let's have a single class for everything related to the configurable logits processors 🤗

gante · 2025-08-26T10:45:45Z

+    from transformers.generation.logits_process import LogitsProcessor, LogitProcessorRegistry
+    import torch


all the imports in the example should be:
1 - at the top of the file
2 - transformers imports should be top-level imports, i.e. from transformers import LogitsProcessor, LogitProcessorRegistry. See the global PR comment I added about this 🤗

PamelaBha · 2025-08-26T16:42:09Z

Very cool PR! I like where this feature is going 🔥

I've added a few comments, but I think they are simple to solve 🤗 Let me know if you have questions/counter-proposals regarding my comments.

Other global comments:

Missing: documentation of logits_processor in GenerationConfig. I would add a link to the example file inside the docs, your examples are quite clear!

Suggestion: in the documentation of logits_processor in generate(), I would add a mention about the new format (and a link to the examples)

Some classes should be directly imported from the top level, i.e. we should be able to do from transformers import LogitProcessorRegistry. This is done by adding the classes to the outermost __init__.py. I think the only class we need to add there is LogitProcessorRegistry (we can already to from transformers import LogitsProcessor)

thanks a lot for reviewing. I will address these soon and send a new PR

…into Issue40010

PamelaBha · 2025-08-29T00:18:51Z

@gante this test is failing in the latest build but nothing to do with my change I believe: FAILED tests/models/gpt_oss/test_modeling_gpt_oss.py::GptOssModelTest::test_assisted_decoding_matches_greedy_search_1_same - AssertionError: False is not true

any pointers for me?

PamelaBha · 2025-09-02T19:22:27Z

@gante this test is failing in the latest build but nothing to do with my change I believe: FAILED tests/models/gpt_oss/test_modeling_gpt_oss.py::GptOssModelTest::test_assisted_decoding_matches_greedy_search_1_same - AssertionError: False is not true

any pointers for me?

Ping on this @gante

gante · 2025-09-09T15:29:27Z

@PamelaBha don't worry about tests that you believe are unrelated to your changes. That may be temporary issues in our codebase, and I can help you with them when we're ready to merge 🤗

A request: the diff in the PR has many style-related changes, likely the outcome of an automated tool (like the one I paste below). Make sure your environment is up-to-date (pip install -e .[dev]), and make sure the diff only contains strictly needed changes. Minimal diff = we can better track why changes were made = easier long-term maintenance :D

Ping me when the PR is ready for a re-review

PamelaBha · 2025-09-10T23:04:46Z

@gante can you help reopen my PR? I was trying to undo all the styling bot changes and somehow accidentally closed the PR. I am hoping you have a reopen permission

gante · 2025-09-12T09:19:35Z

haha done 👍

PamelaBha · 2025-09-15T17:46:40Z

@gante please take a look at the latest PR

PamelaBha · 2025-09-17T17:25:07Z

@gante ping on this

PamelaBha · 2025-10-09T14:41:38Z

Hi @gante - can you please help review and complete the PR?

zucchini-nlp · 2025-11-05T14:28:20Z

@PamelaBha I will take a look tomorrow, sorry for a delay. With the major v5 release, we might have missed this PR. In the meanwhile, review would be easier if you can revert style changes which aren't related. I see the configuration file has no changes expect for style and shorter line length

It'll help me spot the actual files to review

PamelaBha added 2 commits August 21, 2025 17:19

Declarative logit processor configuration

445fa34

Gitignore conda changes

1f59508

gante reviewed Aug 26, 2025

View reviewed changes

PamelaBha added 17 commits August 27, 2025 16:18

Declarative logit processor configuration

fbd1840

Gitignore conda changes

f90f939

PR comment changes

f3b72a1

More fixes

3a69724

Merge branch 'Issue40010' of https://github.com/PamelaBha/transformers …

c90f0cd

…into Issue40010

Fix merge conflict

bed91c1

Resolve PR comments

5c93e6c

PR comment

e7c342b

syntactic changes

329b1bf

Merge branch 'main' into Issue40010

ffe96a2

syntactic changes

e304ae9

Merge branch 'Issue40010' of https://github.com/PamelaBha/transformers …

957117e

…into Issue40010

Merge branch 'main' into Issue40010

3526f99

wrong space

bd13612

Merge branch 'Issue40010' of https://github.com/PamelaBha/transformers …

4125588

…into Issue40010

lint issues

c881dc2

Lint issues

7bea595

PamelaBha force-pushed the Issue40010 branch from b3d158f to 49239b1 Compare September 10, 2025 22:48

PamelaBha closed this Sep 10, 2025

PamelaBha force-pushed the Issue40010 branch from 49239b1 to 98289c5 Compare September 10, 2025 22:51

gante reopened this Sep 12, 2025

PamelaBha force-pushed the Issue40010 branch from 779f006 to 7bea595 Compare September 13, 2025 22:46

PR comments addressed

9e0f895

		from transformers.generation.logits_process import LogitsProcessor, LogitProcessorRegistry
		import torch

Conversation

PamelaBha commented Aug 23, 2025

What does this PR do?

Before submitting

Who can review?

Uh oh!

Rocketknight1 commented Aug 25, 2025

Uh oh!

gante left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PamelaBha commented Aug 26, 2025

Uh oh!

PamelaBha commented Aug 29, 2025

Uh oh!

PamelaBha commented Sep 2, 2025

Uh oh!

gante commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PamelaBha commented Sep 10, 2025

Uh oh!

gante commented Sep 12, 2025

Uh oh!

PamelaBha commented Sep 15, 2025

Uh oh!

PamelaBha commented Sep 17, 2025

Uh oh!

PamelaBha commented Oct 9, 2025

Uh oh!

zucchini-nlp commented Nov 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gante commented Sep 9, 2025 •

edited

Loading