New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactored loadshapefiles for speed #40

Merged
merged 1 commit into from May 3, 2013

Conversation

Projects
None yet
2 participants
@palewire
Contributor

palewire commented May 1, 2013

I was able to drastically increase the speed of the loadshapefiles management comment by changing a few small things.

  1. The len(layer) to set the count attribute of new BoundarySet objects takes a really long time, to my surprise. Just initializing the set with zero in that field and then letting the count be calculated via SQL after everything is in the database is way faster. Since the app is already doing that recalculation anyway, you don't have to add any code.
  2. A modification of the process that guarantees a unique slug that doesn't require pulling of every single slug out of the database and into memory with each save call. I replaced the existing process with SQL count queries that I suspect are faster.

The result is that time it takes my laptop to load a SHP file with ~13,000 features dropped from more than 8 hours to less than 3 minutes.

ryanmark added a commit that referenced this pull request May 3, 2013

Merge pull request #40 from datadesk/loadshapefiles-refactor
Refactored loadshapefiles for speed

@ryanmark ryanmark merged commit d1ec895 into newsapps:master May 3, 2013

@palewire

This comment has been minimized.

Show comment
Hide comment
@palewire

palewire May 11, 2013

Contributor

Thanks for dealing with these.

Contributor

palewire commented May 11, 2013

Thanks for dealing with these.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment