Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Allow the user to skip verification of index integrity on startup #1485
While loading midpoints from PTable index files during start up, the whole PTable file is read to also verify its integrity using an MD5 checksum. This can take a long time for very large databases/indexes.
This change allows the user to skip this verification by specifying a command line argument:
Specifying that parameter will not read the whole file but will only read the midpoints by seeking directly to the midpoint locations in the file.
It might be worth pairing this change with a lazy midpoint load which would prevent the scan entirely (but the reads would be slower until warmed up). In my testing just disabling the check had little to no effect on the overall time to load.
To do a lazy midpoint load just initialize all the midpoints to -1 when searching if -1 then do the read and populate.