You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I could be misunderstanding the code, but if lines 60 and 57 actually do the same thing, it could be good to remove line 57. This is because line 57 tries to load the whole suffix array into memory which fails for large datasets.
Thanks!
The text was updated successfully, but these errors were encountered:
Hi, this repo makes a lot of deduplication code very easy to use; I'm very grateful for it!
I just noticed that the suffix array is built with this command:
https://github.com/ChenghaoMou/text-dedup/blob/main/text_dedup/exact_dedup/suffix_array/suffix_array_google.py#L60
But right above that command, the suffix array is also built with this command:
https://github.com/ChenghaoMou/text-dedup/blob/main/text_dedup/exact_dedup/suffix_array/suffix_array_google.py#L57
I could be misunderstanding the code, but if lines 60 and 57 actually do the same thing, it could be good to remove line 57. This is because line 57 tries to load the whole suffix array into memory which fails for large datasets.
Thanks!
The text was updated successfully, but these errors were encountered: