ci: Issue 509 script for replaying all dynamodb records into elastics…

[Zambonilli]

…earch

Issue #, if available:

Description of changes:
Script that can be ran to synchronize all dynamodb data with elasticsearch. Useful for DR or cluster migration scenarios.

Checklist:

• Have you successfully deployed to an AWS account with your changes?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[ssvegaraju]

Looks good to me! Quick question: Do you have a rough estimate of how many items would constitute a "small to medium size" implementation for this script? It would be nice to include in the comments, if possible.

[Zambonilli]

With a SEGMENT_COUNT of 15 and a SCAN_PAGE_LIMIT of 300, we saw ~1m dynamodb documents synchronized per hour with no errors in a dev FWoA deployment. The throughput here is a direct reflection of the ElasticSearch cluster write capacity. I can update this script to reflect these numbers and performance profile.

[ssvegaraju]

With a SEGMENT_COUNT of 15 and a SCAN_PAGE_LIMIT of 300, we saw ~1m dynamodb documents synchronized per hour with no errors in a dev FWoA deployment. The throughput here is a direct reflection of the ElasticSearch cluster write capacity. I can update this script to reflect these numbers and performance profile.

That would be great!

[codecov-commenter]

Codecov Report

Merging #681 (67c9a81) into develop (2170107) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop     #681   +/-   ##
========================================
  Coverage    95.65%   95.65%           
========================================
  Files            4        4           
  Lines          207      207           
  Branches        30       30           
========================================
  Hits           198      198           
  Misses           9        9           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d31cbc3...67c9a81. Read the comment docs.