Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Noam/dee 521 unknown token check #2483

Merged
merged 10 commits into from May 2, 2023
Merged

Noam/dee 521 unknown token check #2483

merged 10 commits into from May 2, 2023

Conversation

noamzbr
Copy link
Collaborator

@noamzbr noamzbr commented May 1, 2023

What does this implement/fix? Explain your changes.

Implements the unknown tokens check.

Display:
image

The less common words / words with 1 appearne go into the "other":
image

TODO:
Write more tests
Write plot example?

@noamzbr noamzbr added feature Feature update or code change to the package nlp Affects deepchecks.nlp package labels May 1, 2023
@noamzbr noamzbr self-assigned this May 1, 2023
@noamzbr noamzbr requested a review from a team as a code owner May 1, 2023 09:04
@noamzbr noamzbr requested a review from ItayGabbay as a code owner May 1, 2023 14:19
Nadav-Barak
Nadav-Barak approved these changes May 1, 2023
View reviewed changes
Copy link
Collaborator

@Nadav-Barak Nadav-Barak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to add somewhere in the display description on what the percent represent

deepchecks/nlp/checks/data_integrity/unknown_tokens.py Show resolved Hide resolved
@noamzbr noamzbr enabled auto-merge (squash) May 1, 2023 16:30
@noamzbr noamzbr merged commit c503c94 into main May 2, 2023
22 checks passed
@delete-merged-branch delete-merged-branch bot deleted the noam/dee-521-unknown-token-check branch May 2, 2023 06:40
kishore-s-15 pushed a commit to kishore-s-15/deepchecks that referenced this pull request May 7, 2023
@noamzbr @kishore-s-15
Add unknown token check (deepchecks#2483)
c49075d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ItayGabbay ItayGabbay ItayGabbay approved these changes

@Nadav-Barak Nadav-Barak Nadav-Barak approved these changes

Assignees

@noamzbr noamzbr

Labels
feature Feature update or code change to the package nlp Affects deepchecks.nlp package
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@noamzbr @ItayGabbay @Nadav-Barak