Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modified to be able to use HFTransformers NLP in Japanese #5745

Conversation

harada4atsushi
Copy link

Proposed changes:

Status (please check what you already did):

  • added some tests for the functionality
  • updated the documentation
  • updated the changelog (please check changelog for instructions)
  • reformat files using black (please check Readme for instructions)

@CLAassistant
Copy link

CLAassistant commented Apr 29, 2020

CLA assistant check
All committers have signed the CLA.

@sara-tagger
Copy link
Collaborator

Thanks for opening a draft pull request 🚀If you have any questions, you can direct them to @tabergma

@dakshvar22
Copy link
Contributor

@harada4atsushi Thanks for suggesting a fix. For better maintainability of the code, I would suggest creating a custom component at this point for yourself. The component can inherit from HFTransformersNLP component and just override _load_model method to load the corresponding japanese specific tokenizer and model.
Adding a conditional construct in Rasa just for one language could get out of hand very soon IMO. There might be a better approach of solving this by using AutoTokenizers but that would also require more amount of refactoring on the internal implementation of HFTransformersNLP

@harada4atsushi
Copy link
Author

@dakshvar22 Thank you for your comment and suggenstion. Certainly that's right, I attempt to implement custom component.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants