Skip to content
This repository was archived by the owner on Nov 26, 2025. It is now read-only.
This repository was archived by the owner on Nov 26, 2025. It is now read-only.

Use of padding index in tokenizer.tokenize + utils.tokenized_text_in_tokens function #43

@jpramil

Description

@jpramil

Might be an issue with setting a default value for the padding index in the utils.tokenized_text_in_tokens function.
Actually, more specifically, I think this function is only used in the tokenizer.tokenize function, which in turn is only used in the predict method of the FastTextModel class.
But it is called before the actual padding index is calculated (which depends on the input data).

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions