Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Batching MD step #10

Closed
mickvanhulst opened this issue Mar 21, 2020 · 1 comment
Closed

Batching MD step #10

mickvanhulst opened this issue Mar 21, 2020 · 1 comment
Assignees
@mickvanhulst

Comments

@mickvanhulst
Copy link
Collaborator

Documents have a varying number of sentences. To optimize this we may process all documents into sentences, where we store indexes to know which sentences belong to which documents. Then we may create equal-sized batches of these sentences, which can then be parsed for MD.
@mickvanhulst mickvanhulst changed the title Batching MD and ED step Batching MD step Mar 21, 2020
@mickvanhulst mickvanhulst self-assigned this Mar 23, 2020
@mickvanhulst
Copy link
Collaborator Author

mickvanhulst commented Apr 5, 2020 •
edited
Loading

Implemented and done.

@mickvanhulst mickvanhulst reopened this Apr 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mickvanhulst mickvanhulst

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mickvanhulst