feat: Add weakref support to Tokenizer class

[mrkm4ntr]

Motivation

The tokenizers.Tokenizer class currently does not support Python weak references, which prevents its use with frameworks that rely on weakref for resource management.

Error without this change:

import weakref
from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("tokenizer.json")
weak_ref = weakref.ref(tokenizer)
# TypeError: cannot create weak reference to 'tokenizers.Tokenizer' object

Use cases that benefit:

• Apache Beam's shared.Shared for multi-threaded model sharing
• Dask, Ray, and other distributed computing frameworks
• Memory-efficient caching patterns using weakref.WeakValueDictionary
• Any Python code using weak references for resource lifecycle management

Solution

Add weakref parameter to the #[pyclass] macro in PyO3 bindings.

[ArthurZucker]

SGTM ty

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

can you run fmt + clippy!?