chord not creating queue messages for all the tasks

[MugdhaHardikar-GSLab]

Hi Team,

I am using celery chord for my workflow.
Bellow is the setup:
I have ~1,30,000 tasks which gets added in the chord.
I have one worker executing these task with 16 concurrency.
I have 16core 126GB machine.
using MongoDB as result backend and rabbitmq as a broker.

Behaviour:
celery.chord_unlock keeps on going infinitely, without proceeding to the link of the chord.
multiple tasks( sometimes 1400, 300, 400, 1400, 3500 for same set of data) remains without processing.
all tasks in celery.celry_taskmeta are in "SUCCESS" status.
There are exactly same number of entries in the celery collection as that of completed task No trace of missing tasks

Observation:
Message count in rabbitmq is less compared to the number of tasks added to the chord.
Somehow chord_unlock knows that there are some tasks which are complete and it waits but as there are no messages in the queue, no worker processes them.
If I use subset of data ( ~1,00,000) files then it goes fine.
but for other sub set of 28,000 files it will have those missing tasks.
I tried running them separately but sometimes it goes tho' sometimes it doesn't.

In celery_taskmeta group task has correct number of children entries but documents with "parent_id" are less.

Expectations:
Though right now I have 1,30,000 tasks in a chord I am hoping to increase it to 5,00,000 ( offcource with better machine or more workers)

Please help me debug the issue.
I am interested in knowing how celery.chord_unlock knows how many tasks are there? Dont see it stored anywhere in the MongoDB. Also don't have those many entries in celery_taskmeata collection.

Versions:
rabbitmq: 3.12.7
Celery : 5.1.2
Mongo: 5.0.21