-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pipeline in cluster mode #416
Comments
Sorry for the late response. I do not recommend to use pipeline in this case since the slot-node table changes and accessing slot-node on the application side breaks the slot redirection ( For the question 2: ioredis checks the keys of all the commands to ensure that all these commands are able to sent to the same node. For your case, just send the 5000 commands without pipeline should not cause any performance problems because Node.js by default follows pipeline mode (that sends the next command without waiting for the receiving of the previous command) when sending commands. |
I've actually had to hack my way through this not so long ago. In my experience often cluster is used with pre-hashed keys, and I would want the pipeline to be formed for my requests to do some batching. I know that the requests would go to the same machine, hence for that case it would be reasonable to enable pipelining. @luin what do you think? Example code with batching: /**
* Queues more requests to pipeline
*/
function addToPipeline(key) {
// fetchData is a lua script
this.pipeline.fetchData(1, key, this.omitFields);
}
module.exports.batch = function fetchDataBatch(keys, omitFields = []) {
const timer = perf('fetchData:batch');
const redis = this.redis;
// this must include {}
const prefix = redis.options.keyPrefix;
const slot = calcSlot(prefix);
// they will all refer to the same slot, because we prefix with {}
// this has possibility of throwing, but not likely to since previous operations
// would've been rejected already, in a promise this will result in a rejection
const nodeKeys = redis.slots[slot];
const masters = redis.connectionPool.nodes.master;
const masterNode = nodeKeys.reduce((node, key) => node || masters[key], null);
// uses internal implementation details
if (is.fn(masterNode.fetchData) !== true) {
masterNode.options.keyPrefix = prefix;
masterNode.defineCommand('fetchData', { lua: fetchDataScript });
masterNode.options.keyPrefix = null;
}
const pipeline = masterNode.pipeline();
keys.forEach(addToPipeline, { pipeline, omitFields });
return pipeline
.exec()
.tap(timer('pipeline'))
.map(deserializePipeline)
.finally(timer);
}; |
@AVVS Why not just call |
it throws for lua scripts
https://github.com/luin/ioredis/blob/master/lib/pipeline.js#L240-L243 <https://github.com/luin/ioredis/blob/master/lib/pipeline.js#L240-L243>
… On Jan 25, 2017, at 8:59 PM, Zihua 子骅 ***@***.***> wrote:
@AVVS <https://github.com/AVVS> Why not just call Cluster#pipeline() directly instead of Redis#pipeline() given all your keys belong to the same slot since your prefix them with the common {} flag.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#416 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABol0Zepds23VgyfGBULkI5KTO0gWzYYks5rWCg0gaJpZM4LT4hL>.
|
@AVVS Aha! Then your code looks good to me. Why don't you just update your lua script to support multiple keys in one command? |
@luin Regarding this sentence, I think the advantage of pipeline is to save network bandwidth, hence reducing the time this task need. The node.js server performance is surely same in both mode(pipeline and send one by one) since it is asynchronous. http://bbs.redis.cn/forum.php?mod=viewthread&tid=826 |
I have a task that need to chain 5000 commands in pipeline and I have to use the pipeline because the large number of commands, it would perform way better than sending command one by one. So the solution is to form different pipeline in client side based on the key. So i need to get the slot-node table in client.
Question 1
Where can i get the cached slot-node table variable in ioredis?
Question 2
Does ioredis only check the first command in pipeline and decide which server to go?
Much thanks. A suggestion for ioredis is to have more information about pipeling in cluster mode because a serious application using redis must also use cluster. Thanks.
The text was updated successfully, but these errors were encountered: