Fix to issue #17 limits cmd_merge to be single-threaded

[kleinj]

Hi,

it looks like the fix for issue #17, which puts some limits on the number of threads in cmd_merge, is a bit too aggressive, resulting in only using a single thread even for big workloads:

deduplicate-text-datasets/src/main.rs

Lines 1020 to 1023 in ad86c7f

	// Make sure we have enough space to take strided offsets for multiple threads
	// This should be an over-approximation, and starts allowing new threads at 1k of data
	let num_threads = std::cmp::min(num_threads, std::cmp::max((texts.len() as i64 - 1024)/10, 1));
	println!("AA {}", num_threads);

texts.len() is equal to nn (the number of input parts), I think you want something like

    let num_threads = std::cmp::min(num_threads, std::cmp::max((texts_len.iter().sum::<usize>() as i64 - 1024)/10, 1));

instead.

[carlini]

Oops. Yeah that's bad. Let me fix this.

[TristanThrush]

I just added the great suggestion by @kleinj as a PR: https://github.com/google-research/deduplicate-text-datasets/pull/22/files.

I was running the code on a 10's of gigabyte dataset and this fix makes the code only run in about 2 hours with a lot of CPUs (it would've taken 100's of hours without this fix). Let me know what you think @carlini!

[carlini]

This should have been fixed.