Speed up redis IP pool name listing on high latency links #1744

Open
arr2036 opened this Issue Sep 18, 2016 · 0 comments

Projects

None yet

1 participant

@arr2036
Member
arr2036 commented Sep 18, 2016 edited

Issue type

See here for debugging instructions.

Defect/Feature description

The redis ippool tool can list all pools currently present on the server. It does this by doing a key scan, which is horrifically slow if there's any latency on the network between the tool and the Redis server.

We should probably do this server side and increase the number of iterations per call.

https://blog.al4.co.nz/2014/08/safely-running-bulk-operations-on-redis-with-lua-scripts/

describes the problem well...

Doing it server side and passing through the cursor, allows us to pre-filter the results to at least we're not transferring large numbers of keys back.

The alternative is keeping metadata in redis... I'm against this. The primary purpose of the ippool code is to allocate IP addresses, and keeping pool lists, and other things that can get out of sync with the data is annoying, and contrary to the design philosophy. There's no referential integrity in Redis, so it can get all crunchy...

The fact that the code went into production with relatively few bug fixes, and has not yet caused any major service outages is a good indicator of robustness. Interestingly the only major service outage related to redis ippool code, was when someone wrote an import script in perl and inserted corrupted (partial) pool keys...

If admins really want a quick way of knowing which pools are present, then they can maintain their own list 👍

@arr2036 arr2036 added this to the 4.0.0 release milestone Sep 18, 2016
@arr2036 arr2036 self-assigned this Sep 18, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment