Skip to content

let number of output directories be specified as an argument#201

Open
ahadsuleymanli wants to merge 1 commit into
WikiExtractor:masterfrom
ahadsuleymanli:feature-distribute_amongst_n_directories
Open

let number of output directories be specified as an argument#201
ahadsuleymanli wants to merge 1 commit into
WikiExtractor:masterfrom
ahadsuleymanli:feature-distribute_amongst_n_directories

Conversation

@ahadsuleymanli

Copy link
Copy Markdown

When the user specifies --dir_count the output text files will be evenly distributed among the specified amount of folder. This is useful when the text corpus will be further batch processed and the user needs to have one folder per CPU-thread.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant