Speed and accuracy improvements surrounding import --replace operations #75
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
On rr's that have more than one entry, for example MX records, the order they are in the zone file may not match the order they are in the the AWS pull of the record. This leads to needless deletes and adds when doing an import --replace.
The original algorithm went O(n^2) during the comparison. This resulted in hilariously long runtimes for moderate sized zones. This crudely smashes a hash table (dict to be exact) into the process, dramatically reducing the comparison time.
Conflicts: scripts/cli53
Conflicts: scripts/cli53
|
Thanks for this, it's great to have it tested to this scale. Your code is almost there, I'll tweak it a bit and push the changes to master this week. Thanks again. |
|
FWIW, I've picked up your code and run it in my test environment. It's run time is within epsilon of my version, so I'm adopting your implementation. Thanks for hammering it into acceptable python, and integrating it into mainline. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I picked up cli53 to handle syncing from a legacy system to AWS Route53.
Attempts to sync my zone file, which the cut down minimal one is 22k RR's resulted in hilariously long run times (82 minutes of CPU), and needless replacement of some record types (MX records coming back in a different order than in the local zone file).
I'm not a python programmer. I've hit this code with a hammer to make it run acceptably in my environment.
Run time on my zone is down to 51 seconds wall clock, 18 seconds cpu.
Consider this, if not reasonable patches, at least suggestions on how to make it run faster. Dig around in my branch history if you want the bits for the problems as isolated chunks.
Thanks.