Question about GoogleSuffixArrayDeduplicator

[TristanThrush]

Hi, this repo makes a lot of deduplication code very easy to use; I'm very grateful for it!

I just noticed that the suffix array is built with this command:

https://github.com/ChenghaoMou/text-dedup/blob/main/text_dedup/exact_dedup/suffix_array/suffix_array_google.py#L60

But right above that command, the suffix array is also built with this command:

https://github.com/ChenghaoMou/text-dedup/blob/main/text_dedup/exact_dedup/suffix_array/suffix_array_google.py#L57

I could be misunderstanding the code, but if lines 60 and 57 actually do the same thing, it could be good to remove line 57. This is because line 57 tries to load the whole suffix array into memory which fails for large datasets.

Thanks!

[ChenghaoMou]

You are right, that cargo command is redundant. You can either use the latest commit or pip install --upgrade text-dedup to 0.2.1 for the fix.

[ChenghaoMou]

And thanks for the issue. It's much appreciated!

[TristanThrush]

thanks; so fast!