[Help Wanted] Tokenzier warning messages

[kylesayrs]

Many users see a warning about tokenizer parallelism. However, this warning is somewhat hard to reproduce. The scope of this ticket is to determine the source of this warning and prevent it from happening when running LLM Compressor.

2025-11-08T20:32:27.729606+0000 | reset | INFO - Compression lifecycle reset
2025-11-08T20:32:27.729954+0000 | from_modifiers | INFO - Creating recipe from modifiers
2025-11-08T20:32:27.730019+0000 | infer_mapping_from_model | INFO - Unrecognized model architecture Gemma3ForConditionalGeneration. Falling back to default mappings
2025-11-08T20:32:27.730046+0000 | infer_norm_mapping_from_model | INFO - Unrecognized model architecture Gemma3ForConditionalGeneration. Falling back to default mappings
2025-11-08T20:32:27.815196+0000 | initialize | INFO - Compression lifecycle initialized for 2 modifiers
2025-11-08T20:32:27.815337+0000 | moe_calibration_context | INFO - Entering MoE calibration context
2025-11-08T20:32:27.815840+0000 | IndependentPipeline | INFO - Inferred `SequentialPipeline` for `SpinQuantModifier`
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...

[Nsumithreddy]

Hi @kylesayrs , I'm a new contributor and would love to help with this 'good first issue'. Is it okay if I take it on and try to reproduce the warning?

[kylesayrs]

Hi @Nsumithreddy! I think it'd be great to know why this warning happens, and whether we should edit the LLM Compressor example scripts to avoid this, or if there's an edit to LLM Compressor that's required.

LLM Compressor itself doesn't use process parallelism at the moment, although I suspect that datasets' map function might, which might be the source of the warning and why it's hard to replicate (datasets often caches map results).

def tokenize(sample):
    return tokenizer(
        sample["text"],
        padding=False,
        max_length=MAX_SEQUENCE_LENGTH,
        truncation=True,
        add_special_tokens=False,
    )

If this function is the source of the issues, we should set TOKENIZERS_PARALLELISM=True in the example scripts, since this environment variable is specific to those usages of map in the examples.

[kylesayrs]

You can test without caching using

ds = ds.map(tokenize, remove_columns=ds.column_names, load_from_cache_file=False)