Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix edge case for tokenize (#36277) #36555

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix edge case for tokenize (#36277) #36555

wants to merge 1 commit into from
+17 −11

Conversation

wangzhen0518
Copy link

What does this PR do?

Fixes #36277

The dtype of the tensor returned by tokenize is inconsistent depending on whether the input text is empty or not, e.g.,

def show_return_dtype(x:str):
    return_type = ["tf", "pt", "jax", "mlx", "np"]
    for t in return_type:
        output = tokenizer(x, return_tensors=t)
        print(f"{t}: {output['input_ids'].dtype}")

x = ""
show_return_dtype(x)
# tf: <dtype: 'float32'>
# pt: torch.float32
# jax: float32
# mlx: mlx.core.float32
# np: float64

x = "abc"
show_return_dtype(x)
# tf: <dtype: 'int32'>
# pt: torch.int64
# jax: int32
# mlx: mlx.core.int32
# np: int64

After this PR is fixed:

def show_return_dtype(x:str):
    return_type = ["tf", "pt", "jax", "mlx", "np"]
    for t in return_type:
        output = tokenizer(x, return_tensors=t)
        print(f"{t}: {output['input_ids'].dtype}")

x = ""
show_return_dtype(x)
# tf: <dtype: 'int32'>
# pt: torch.int64
# jax: int32
# mlx: mlx.core.int32
# np: int64

x = "abc"
show_return_dtype(x)
# tf: <dtype: 'int32'>
# pt: torch.int64
# jax: int32
# mlx: mlx.core.int32
# np: int64

Before submitting

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@Rocketknight1 and @ArthurZucker

Sorry, something went wrong.

@github-actions github-actions bot marked this pull request as draft March 5, 2025 08:08
@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 5, 2025

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the Ready for review button (at the bottom of the PR page).

@wangzhen0518 wangzhen0518 marked this pull request as ready for review March 5, 2025 08:12
@Rocketknight1
Copy link
Member

cc @ArthurZucker @itazap for review. The full explanation is at #36277, but the tl;dr is:

  • Tokenizer tensor outputs are generated with torch.tensor()
  • torch.tensor() infers the dtype according to the input data and the following rule:
    • If every element is an int, dtype=torch.int64
    • If at least one element is a float, dtype=torch.float32
    • If the input list is empty, dtype=torch.float32

This creates an edge cases when our tokenizers return empty arrays - the dtype switches from torch.int64 to torch.float32, and this breaks ops like torch.concat. This PR always casts the dtype to torch.int64, but is there any case where tokenizers return float32 arrays? I can't think of one, but this will definitely break those cases if they exist!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

The output tensor's data type is not torch.long when the input text is empty.
2 participants
@wangzhen0518 @Rocketknight1