Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

buffer sqlite writes if needed #608

Merged
merged 1 commit into from
Dec 9, 2023
Merged

Conversation

cldellow
Copy link
Contributor

@cldellow cldellow commented Dec 7, 2023

Fixes #596, I think.

Time to process Antarctica drops from 9m5s to 3m51s

Fixes systemed#596, I think.

Time to process Antarctica drops from 9m5s to 3m51s
@systemed
Copy link
Owner

systemed commented Dec 8, 2023

This is great - CPU happily pegged at 16x100% throughout!

Before #607 I'm getting a consistent "Command terminated by signal 11" right at the end of tile generation when processing Antarctica (including landcover). I put a bit of logging into closeForWriting but it's dying before that's called; I'll add a few more earlier in the process and let you know if I find anything. It's fine with #607, and it doesn't happen with the smaller areas I've tried (Oxfordshire and Great Britain).

@cldellow
Copy link
Contributor Author

cldellow commented Dec 8, 2023

I think #595 introduced some out of bounds memory accesses when processing nodes that have invalid lat/lngs :(

There was one that I caught and fixed in #595 (066ca0a), and another that I fixed in #607 (47bcf4a).

It's possible that there are more lurking about, I can try running Antarctica under valgrind this weekend and see if it pops anything up

@systemed
Copy link
Owner

systemed commented Dec 8, 2023

Ah - that would make sense why it stopped after #607. I'd wondered about valgrind but the thought of it slowly crunching through the whole of Antarctica put me off!

This was referenced Dec 8, 2023
@systemed systemed merged commit 0d951ac into systemed:master Dec 9, 2023
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Low CPU usage when writing tiles
2 participants