Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tokenize arrays of strings #93

Closed
ghost opened this issue Sep 30, 2019 · 1 comment
Closed

Tokenize arrays of strings #93

ghost opened this issue Sep 30, 2019 · 1 comment
Labels
enhancement New feature or request

Comments

@ghost
Copy link

ghost commented Sep 30, 2019

Is your feature request related to a problem? Please describe.

The BaseSampler will handle arrays of tensors elegantly. However, these arrays only come to be if you manually construct them yourself.

Describe the solution you'd like

To be able to have an array-as-one-observation, and for the TextField to handle that gracefully (then pass it along to the BaseSampler).

Additional context

Models that embed a large sequence of context utterances, where simply joining those utterances into one string makes for one really long string.

@ghost ghost added the enhancement New feature or request label Sep 30, 2019
@ghost ghost changed the title Tokenize lists of strings Tokenize arrays of strings Sep 30, 2019
@jeremyasapp
Copy link
Contributor

Done in #255

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant