Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

spatial load command improvements #2261

Merged

Conversation

@noirbizarre
Copy link
Member

commented Jul 29, 2019

This PR improves the spatial load command to be more resilient:

  • extracts the geozones archive into its own tmp storage subdirectory
  • makes use of a temporary collection and then overwrite the existing one when using --drop
  • cleanup the temporary files on success
  • handles exceptions and keyboard interrupts and cleanup when it occurs

These changes:

  • prevent filling the tmp storage with undeleted files
  • prevent against partial loads in case of errors or interruptions
  • prevent frontend/API/admin errors on missing spatial zone during spatial load --drop
@noirbizarre noirbizarre requested a review from opendatateam/etalab Jul 29, 2019
noirbizarre added 5 commits Jul 23, 2019
…ial load
@noirbizarre noirbizarre force-pushed the noirbizarre:zero-downtime-spatial-load branch from 3972565 to ad21361 Jul 29, 2019
@bzg
bzg approved these changes Jul 29, 2019
@noirbizarre noirbizarre merged commit 2a97201 into opendatateam:master Jul 29, 2019
3 checks passed
3 checks passed
ci/circleci: assets Your tests passed on CircleCI!
Details
ci/circleci: dist Your tests passed on CircleCI!
Details
ci/circleci: python Your tests passed on CircleCI!
Details
@noirbizarre noirbizarre deleted the noirbizarre:zero-downtime-spatial-load branch Jul 29, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.