Chord Callback Called Multiple Time

[RichardKnop]

Chord callback is being called N times where N is number of tasks in the group.

This affects Redis and Memcache backends. AMQP backend is not affected.

Workaround until this is fixed is to make sure chord callback is idempotent.

[vamsu]

FYI, this affects mongo backend as well. Broker: amqp.

This happens in the default example. It happens intermittently, there seems to be race condition somewhere.

[machinery: amqp.go:159: Received new message: {"UUID":"task_fe149098-78be-4a7a-8707-ed9618cb8a12","Name":"add","RoutingKey":"","GroupUUID":"group_9af5e908-3306-4387-be9c-0d9b49650afe","GroupTaskCount":3,"Args":[{"Type":"int64","Value":2},{"Type":"int64","Value":2}],"Headers":null,"Immutable":false,"OnSuccess":null,"OnError":null,"ChordCallback":{"UUID":"chord_a604eb54-dcce-4872-9325-fa223d021b88","Name":"multiply","RoutingKey":"","GroupUUID":"","GroupTaskCount":0,"Args":null,"Headers":null,"Immutable":false,"OnSuccess":null,"OnError":null,"ChordCallback":null}}
INVOKED ADD: 5machinery: worker.go:110: Processed task_fe149098-78be-4a7a-8707-ed9618cb8a12. Result = 4
machinery: amqp.go:159: Received new message: {"UUID":"task_ab92c0e2-a4c6-4d65-9a76-7cfd15094abe","Name":"add","RoutingKey":"","GroupUUID":"group_9af5e908-3306-4387-be9c-0d9b49650afe","GroupTaskCount":3,"Args":[{"Type":"int64","Value":1},{"Type":"int64","Value":1}],"Headers":null,"Immutable":false,"OnSuccess":null,"OnError":null,"ChordCallback":{"UUID":"chord_a604eb54-dcce-4872-9325-fa223d021b88","Name":"multiply","RoutingKey":"","GroupUUID":"","GroupTaskCount":0,"Args":null,"Headers":null,"Immutable":false,"OnSuccess":null,"OnError":null,"ChordCallback":null}}
machinery: amqp.go:159: Received new message: {"UUID":"task_4820d0cd-e6a8-4501-a8c1-427aa5170319","Name":"add","RoutingKey":"","GroupUUID":"group_9af5e908-3306-4387-be9c-0d9b49650afe","GroupTaskCount":3,"Args":[{"Type":"int64","Value":5},{"Type":"int64","Value":6}],"Headers":null,"Immutable":false,"OnSuccess":null,"OnError":null,"ChordCallback":{"UUID":"chord_a604eb54-dcce-4872-9325-fa223d021b88","Name":"multiply","RoutingKey":"","GroupUUID":"","GroupTaskCount":0,"Args":null,"Headers":null,"Immutable":false,"OnSuccess":null,"OnError":null,"ChordCallback":null}}
INVOKED ADD: 6INVOKED ADD: 7machinery: worker.go:110: Processed task_ab92c0e2-a4c6-4d65-9a76-7cfd15094abe. Result = 2
machinery: worker.go:110: Processed task_4820d0cd-e6a8-4501-a8c1-427aa5170319. Result = 11
machinery: amqp.go:159: Received new message: {"UUID":"chord_a604eb54-dcce-4872-9325-fa223d021b88","Name":"multiply","RoutingKey":"","GroupUUID":"","GroupTaskCount":0,"Args":[{"Type":"int64","Value":2},{"Type":"int64","Value":4},{"Type":"int64","Value":11}],"Headers":null,"Immutable":false,"OnSuccess":null,"OnError":null,"ChordCallback":null}
INVOKED MULTIPLY: 1machinery: worker.go:110: Processed chord_a604eb54-dcce-4872-9325-fa223d021b88. Result = 88
machinery: amqp.go:159: Received new message: {"UUID":"chord_a604eb54-dcce-4872-9325-fa223d021b88","Name":"multiply","RoutingKey":"","GroupUUID":"","GroupTaskCount":0,"Args":[{"Type":"int64","Value":2},{"Type":"int64","Value":4},{"Type":"int64","Value":11}],"Headers":null,"Immutable":false,"OnSuccess":null,"OnError":null,"ChordCallback":null}
INVOKED MULTIPLY: 2machinery: worker.go:110: Processed chord_a604eb54-dcce-4872-9325-fa223d021b88. Result = 88]

[RichardKnop]

@vamsu Update. This has been fixed for Redis backend now. The way to fix it is using distributed locks.

For MongoDB backend, you could fix this bug by implementing this method: https://github.com/RichardKnop/machinery/blob/master/v1/backends/mongodb.go#L100

See Redis implementation for reference: https://github.com/RichardKnop/machinery/blob/master/v1/backends/redis.go#L98