Skip to content

Commit

Permalink
Remove kwargs mapping on Tokenizer decode/decode_batch as their is on…
Browse files Browse the repository at this point in the history
…ly one possible arg.

This is suggested by the current issue #54 (comment).

kwargs cannot be called as positional argument, they have to be named one, replacing kwargs with the actual skip_special_tokens
allows both (named and positional) syntax.

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>
  • Loading branch information
mfuntowicz committed Jan 15, 2020
1 parent a779714 commit 4839154
Showing 1 changed file with 10 additions and 22 deletions.
32 changes: 10 additions & 22 deletions bindings/python/src/tokenizer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -189,34 +189,22 @@ impl Tokenizer {
.into()
}

#[args(kwargs = "**")]
fn decode(&self, ids: Vec<u32>, kwargs: Option<&PyDict>) -> PyResult<String> {
let mut skip_special_tokens = true;

if let Some(kwargs) = kwargs {
if let Some(skip) = kwargs.get_item("skip_special_tokens") {
skip_special_tokens = skip.extract()?;
}
}

ToPyResult(self.tokenizer.decode(ids, skip_special_tokens)).into()
fn decode(&self, ids: Vec<u32>, skip_special_tokens: Option<bool>) -> PyResult<String> {
ToPyResult(self.tokenizer.decode(
ids,
skip_special_tokens.unwrap_or(true),
)).into()
}

#[args(kwargs = "**")]
fn decode_batch(
&self,
sentences: Vec<Vec<u32>>,
kwargs: Option<&PyDict>,
skip_special_tokens: Option<bool>,
) -> PyResult<Vec<String>> {
let mut skip_special_tokens = true;

if let Some(kwargs) = kwargs {
if let Some(skip) = kwargs.get_item("skip_special_tokens") {
skip_special_tokens = skip.extract()?;
}
}

ToPyResult(self.tokenizer.decode_batch(sentences, skip_special_tokens)).into()
ToPyResult(self.tokenizer.decode_batch(
sentences,
skip_special_tokens.unwrap_or(true),
)).into()
}

fn token_to_id(&self, token: &str) -> Option<u32> {
Expand Down

0 comments on commit 4839154

Please sign in to comment.