Skip to content

Navigation Menu

Explore
By size
By industry
By use case
Topics
Explore
- GitHub Sponsors
  Fund open source developers
- The ReadME Project
  GitHub community articles
Repositories
- Enterprise platform
  AI-powered developer platform
Available add-ons
Pricing

Search code, repositories, users, issues, pull requests...

Search

Clear

Search syntax tips

Provide feedback

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Saved searches

Use saved searches to filter your results more quickly

Name

Query

To see all available qualifiers, see our documentation.

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

This repository has been archived by the owner on Nov 22, 2022. It is now read-only.

facebookresearch / pytext Public archive

Notifications You must be signed in to change notification settings
Fork 801
Star 6.3k

Code
Issues 61
Pull requests 84
Actions
Projects 1
Security
Insights

Additional navigation options

Code
Issues
Pull requests
Actions
Projects
Security
Insights

Adding Google SentencePiece as a Tokenizer #1106

Closed

snisarg wants to merge 10 commits into master from sentencepiece

Closed

Adding Google SentencePiece as a Tokenizer #1106

snisarg wants to merge 10 commits into master from sentencepiece

Conversation 11 Commits 10 Checks 0 Files changed

Conversation

Copy link

Contributor

snisarg commented Nov 4, 2019

This diff adds SentencePiece as a pip requirement, and a tokenizer shell
for PyText

Motivation and Context

We need SentencePiece to support modern cross lingual models

How Has This Been Tested

A unit test has been added.

Types of changes

Docs change / refactoring / dependency upgrade
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have completed my CLA (see CONTRIBUTING)
I have added tests to cover my changes.
All new and existing tests passed.

Sorry, something went wrong.

All reactions

snisarg added 3 commits

November 4, 2019 14:10


          Adding Google SentencePiece as a Tokenizer

1b1276f

This diff adds SentencePiece as a pip requirement, and a tokenizer shell
for PyText


          Addressing lint

f17353e


          Forgot one more

04d3bc3

facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label

snisarg added 2 commits

November 4, 2019 14:26


          Doc requirements added.

49398ea


          Help me!

e529163

facebook-github-bot reviewed

View reviewed changes

Copy link

Contributor

facebook-github-bot left a comment

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@snisarg has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Sorry, something went wrong.

All reactions


          More lint stuff

a21a1c8

facebook-github-bot reviewed

View reviewed changes

Copy link

Contributor

facebook-github-bot left a comment

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@snisarg has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Sorry, something went wrong.

All reactions


          Changed model with a smaller one from sentencepiece's repo

935c064

Copy link

Contributor

facebook-github-bot commented Nov 6, 2019

@snisarg has updated the pull request. Re-import the pull request

All reactions

Sorry, something went wrong.

facebook-github-bot reviewed

View reviewed changes

Copy link

Contributor

facebook-github-bot left a comment

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@snisarg has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Sorry, something went wrong.

All reactions


          Committing new model

d253824

Copy link

Contributor

facebook-github-bot commented Nov 6, 2019

@snisarg has updated the pull request. Re-import the pull request

All reactions

Sorry, something went wrong.

facebook-github-bot reviewed

View reviewed changes

Copy link

Contributor

facebook-github-bot left a comment

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@snisarg has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Sorry, something went wrong.

All reactions


          Facebook's linter fix

82f6784

Copy link

Contributor

facebook-github-bot commented Nov 6, 2019

@snisarg has updated the pull request. Re-import the pull request

All reactions

Sorry, something went wrong.

facebook-github-bot reviewed

View reviewed changes

Copy link

Contributor

facebook-github-bot left a comment

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@snisarg has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Sorry, something went wrong.

All reactions


          FB linter, quotes change.

e022a08

Copy link

Contributor

facebook-github-bot commented Nov 6, 2019

@snisarg has updated the pull request. Re-import the pull request

All reactions

Sorry, something went wrong.

facebook-github-bot reviewed

View reviewed changes

Copy link

Contributor

facebook-github-bot left a comment

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@snisarg has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Sorry, something went wrong.

All reactions

facebook-github-bot closed this in

322fc47

Copy link

Contributor

facebook-github-bot commented Nov 8, 2019

@snisarg merged this pull request in 322fc47.

All reactions

Sorry, something went wrong.

facebook-github-bot added the Merged label

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

facebook-github-bot facebook-github-bot left review comments

Assignees

No one assigned

Labels

Do not delete this pull request or issue due to inactivity.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

None yet

2 participants

Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.

Footer

© 2024 GitHub, Inc.

Footer navigation

Terms
Privacy
Security
Status
Docs
Contact

You can’t perform that action at this time.