Skip to content
This repository has been archived by the owner on Nov 22, 2022. It is now read-only.

Fewer out-of-vocab print messages, with some stats #697

Closed
wants to merge 1 commit into from

Conversation

mwu1993
Copy link
Contributor

@mwu1993 mwu1993 commented Jun 17, 2019

Summary: Printing every example with many OOV tokens can crowd the logs for large datasets (even if e.g. only 1% of examples are printed), and we can't tell just many such examples there are. Change the vocab to print one out of every 100 examples, capped at total 200 examples, along with some stats for how many OOVs there have been.

Differential Revision: D15832610

@facebook-github-bot facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Jun 17, 2019
mwu1993 pushed a commit to mwu1993/pytext-1 that referenced this pull request Jun 17, 2019
)

Summary:
Pull Request resolved: facebookresearch#697

Printing every example with many OOV tokens can crowd the logs for large datasets (even if e.g. only 1% of examples are printed), and we can't tell just many such examples there are. Change the vocab to print one out of every 100 examples, capped at total 200 examples, along with some stats for how many OOVs there have been.

Reviewed By: rutyrinott

Differential Revision: D15832610

fbshipit-source-id: b3f8dc9751d0d73e965b5dd0eca25d0d7b34407c
)

Summary:
Pull Request resolved: facebookresearch#697

Printing every example with many OOV tokens can crowd the logs for large datasets (even if e.g. only 1% of examples are printed), and we can't tell just many such examples there are. Change the vocab to print one out of every 100 examples, capped at total 200 examples, along with some stats for how many OOVs there have been.

Reviewed By: rutyrinott

Differential Revision: D15832610

fbshipit-source-id: ea8e33b9b8808a8821a5dfaefccd68e62a46c00b
@facebook-github-bot
Copy link
Contributor

This pull request has been merged in 5c94861.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
CLA Signed Do not delete this pull request or issue due to inactivity. Merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants