Size limitations

[geekscrapy]

Has any testing currently been done as to the limitations of the db? Mainly regarding:

• Maximum number of files
• Maximum file size

[icedevml]

Hello,

so far we've got information about 2 TB of data (in total) succesfully indexed with mquery, while still keeping reasonable query time of 3-5 seconds.

When indexing a single file, the size shouldn't matter, because it will never have more than 2^24 of unique trigrams.

I don't think if we've got any more usage data so far.

[msm-code]

Just to expand on this before closing the issue: right now we have much more than 2TB indexed, and query time is still around 5 seconds.

As icedevml has said, there are no limitations for a single file size - but the bigger are your files the less sense it makes for mquery (due to the way it works - at some point it will contain every possible trigram). Hard to say when, but indexing 4GB files is definitely counter-productive, while 10MB files are definitely handled well.