Redis cache deletion

[base0010]

Description

Type of change

• Docs change / dependency upgrade
• Configuration / tooling changes
• Refactoring
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Requires changes in customer code

High-level change(s) description - from the user's perspective

Related Issue(s)

Fixes

Related pull request(s)

[base0010]

Update to use redis hgetall via streams for non-blocking operation. We think, that since the hook runs frequently there shouldn't ever be too many "Completed" transactions and thus traversing the list shouldn't be too expensive.

[base0010]

Looking to understand the impact of first run (deleting large amount of completed txns) via some simple manual testing/tools like redis-rdb-tools locally on testnet data, perhaps this could also be fleshed out in to better tests against live data instead of mocks in the future.

[alexwhte]

@base0010 Double checking is this waiting on review?

[rhlsthrm]

@base0010 this looks good now. We need to make sure this does not break the setup so we will need to monitor and test when this gets into staging.

[base0010]

Yeah, this looks to work fine with mocks. I haven't been able to get mocha to test my local db(non-mock) instance. Also the redis cache backup size didn't seem to grow that fast over the week (comparing rdb dumps) so I don't think we're gonna hit any limits. But monitor indeed.