Pipeline in cluster mode

[ruofeilyu]

I have a task that need to chain 5000 commands in pipeline and I have to use the pipeline because the large number of commands, it would perform way better than sending command one by one. So the solution is to form different pipeline in client side based on the key. So i need to get the slot-node table in client.
Question 1
Where can i get the cached slot-node table variable in ioredis?
Question 2
Does ioredis only check the first command in pipeline and decide which server to go?

Much thanks. A suggestion for ioredis is to have more information about pipeling in cluster mode because a serious application using redis must also use cluster. Thanks.

[luin]

Sorry for the late response. I do not recommend to use pipeline in this case since the slot-node table changes and accessing slot-node on the application side breaks the slot redirection (ask, moved or node failovers).

For the question 2: ioredis checks the keys of all the commands to ensure that all these commands are able to sent to the same node.

For your case, just send the 5000 commands without pipeline should not cause any performance problems because Node.js by default follows pipeline mode (that sends the next command without waiting for the receiving of the previous command) when sending commands.

[AVVS]

I've actually had to hack my way through this not so long ago. In my experience often cluster is used with pre-hashed keys, and I would want the pipeline to be formed for my requests to do some batching. I know that the requests would go to the same machine, hence for that case it would be reasonable to enable pipelining. @luin what do you think?

Example code with batching:

/**
 * Queues more requests to pipeline
 */
function addToPipeline(key) {
  // fetchData is a lua script
  this.pipeline.fetchData(1, key, this.omitFields);
}

module.exports.batch = function fetchDataBatch(keys, omitFields = []) {
  const timer = perf('fetchData:batch');
  const redis = this.redis;

  // this must include {}
  const prefix = redis.options.keyPrefix;
  const slot = calcSlot(prefix);
  // they will all refer to the same slot, because we prefix with {}
  // this has possibility of throwing, but not likely to since previous operations
  // would've been rejected already, in a promise this will result in a rejection
  const nodeKeys = redis.slots[slot];
  const masters = redis.connectionPool.nodes.master;
  const masterNode = nodeKeys.reduce((node, key) => node || masters[key], null);

  // uses internal implementation details
  if (is.fn(masterNode.fetchData) !== true) {
    masterNode.options.keyPrefix = prefix;
    masterNode.defineCommand('fetchData', { lua: fetchDataScript });
    masterNode.options.keyPrefix = null;
  }

  const pipeline = masterNode.pipeline();
  keys.forEach(addToPipeline, { pipeline, omitFields });

  return pipeline
    .exec()
    .tap(timer('pipeline'))
    .map(deserializePipeline)
    .finally(timer);
};

[luin]

@AVVS Why not just call Cluster#pipeline() directly instead of Redis#pipeline() given all your keys belong to the same slot since your prefix them with the common {} flag.

[AVVS]

it throws for lua scripts https://github.com/luin/ioredis/blob/master/lib/pipeline.js#L240-L243 <https://github.com/luin/ioredis/blob/master/lib/pipeline.js#L240-L243>

…

On Jan 25, 2017, at 8:59 PM, Zihua 子骅 ***@***.***> wrote: @AVVS <https://github.com/AVVS> Why not just call Cluster#pipeline() directly instead of Redis#pipeline() given all your keys belong to the same slot since your prefix them with the common {} flag. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#416 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABol0Zepds23VgyfGBULkI5KTO0gWzYYks5rWCg0gaJpZM4LT4hL>.

[luin]

@AVVS Aha! Then your code looks good to me. Why don't you just update your lua script to support multiple keys in one command?

[ruofeilyu]

@luin
Because Node.js by default follows pipeline mode (that sends the next command without waiting for the receiving of the previous command) when sending commands.

Regarding this sentence, I think the advantage of pipeline is to save network bandwidth, hence reducing the time this task need. The node.js server performance is surely same in both mode(pipeline and send one by one) since it is asynchronous.

http://bbs.redis.cn/forum.php?mod=viewthread&tid=826
In the link above, they(youku) do the same thing(forming different pipeline to different node) and then merge together(they also check the slot-node table, I think that ask, moved is a rare thing, performance should be the first), otherwise cluster for caching is not an ideal option for sizeable application. I would hack it and pull requests later.