Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mbtiles: only create 1 unique index on tiles #603

Merged
merged 1 commit into from
Dec 22, 2023

Conversation

cldellow
Copy link
Contributor

@cldellow cldellow commented Dec 4, 2023

Before:

  • the table definition created a unique index
  • we created an explicit unique index on closeForWriting

After:

  • we create only the explicit index, at program startup

This reduces the mbtiles file by 0.5% and processing time by perhaps 0.3%, although it's quite noisy.

Creating the index at the start, rather than at the end, seemed to be fastest.

Before:
- the table definition created a unique index
- we created an explicit unique index on closeForWriting

After:
- we create only the explicit index, at program startup

This reduces the mbtiles file by 0.5% and processing time by
perhaps 0.3%, although it's quite noisy.

Creating the index at the start, rather than at the end, seemed
to be fastest.
@systemed systemed mentioned this pull request Dec 21, 2023
18 tasks
@systemed systemed merged commit a4fe312 into systemed:master Dec 22, 2023
5 checks passed
@systemed
Copy link
Owner

Good spot! Usual wisdom is to create the index at the end, but because we run all the inserts in one transaction, it should be fine to do it at the start.

@cldellow
Copy link
Contributor Author

Yeah, I think it depends what you want to optimize for. I would expect an index created at the end to have no fragmentation, to use less CPU time to build, and to be slightly smaller on disk.

But at the end, all of our cores are idle. So even though it takes less CPU time, it will take more wall clock time vs creating it at the start, and then amortizing the cost over the run of the program while other cores are still doing useful work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants