Chunk Redis Backend Results when larger than 512MB

[ErikBZ]

Checklist

• I have checked the issues list
  for similar or identical feature requests.
• I have checked the pull requests list
  for existing proposed implementations of this feature.
• I have checked the commit log
  to find out if the same feature was already implemented in the
  master branch.
• I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

Related Issues and Possible Duplicates

Related Issues

• Celery 7911

Possible Duplicates

• None

Brief Summary

Redis has a limit of 512MB for String values. While generating tasks, some may get larger than that limit resulting in a BackendStoreError #6533. In applications where a size larger than 512MB might be expected, we'd like to chunk the task down to manageable pieces and then coalesce the data when it's needed.

This is related to the other issue I posted, #7911, since we would be able to send the IDs of the chunked data and then merge the results later on when a worker has picked up the task.

Design

During a SET we'd use the same key as we normally would celery-task-meta-561cb2b3-c43e-4944-9531-e1b37e84c33d but the data saved to this key would instead be a list of sub-keys, prefixed by REDIS_CHUNK_KEYS, followed by the ids of the chunks delimited by a ,. For example REDIS_CHUNK_KEYS,key1,key2.

These subkeys would be where the chunks of the actual data are stored.

During a GET, we'd check if the string value is prefixed by REDIS_CHUNK_KEYS. If it is then we'd parse the value, and then make another request for each of data chunks pointed to by the subkeys.

Architectural Considerations

None

Proposed Behavior

Proposed UI/UX

Diagrams

N/A

Alternatives

None

[open-collective-bot]

Hey @ErikBZ 👋,
Thank you for opening an issue. We will get back to you as soon as we can.
Also, check out our Open Collective and consider backing us - every little helps!

We also offer priority support for our sponsors.
If you require immediate assistance please consider sponsoring us.

[auvipy]

do you have implementation detail in mind? also how does #7801 (comment) fits in this case?

[ErikBZ]

maxmemory sets the max usage for the service, but not the max size of a single value. A key's value hitting the 512MB limit is what we're hitting.
As for the implementation, during a SET we'd use the same key as we normally would celery-task-meta-561cb2b3-c43e-4944-9531-e1b37e84c33d but the data saved to this key would instead be a list of sub-keys, prefixed by REDIS_CHUNK_KEYS, followed by the ids of the chunks delimited by a ,. For example REDIS_CHUNK_KEYS,key1,key2.

These subkeys would be where the chunks of the actual data are stored.

During a GET, we'd check if the string value is prefixed by REDIS_CHUNK_KEYS. If it is then we'd parse the value, and then make another request for each of data chunks pointed to by the subkeys.

[auvipy]

OK, so you are up for the implementation?

[ErikBZ]

Yup, sure am

[auvipy]

Ping me on the pr for review

[denyszhak]

Are there any updates on this? It's being here for a while now

[auvipy]

we are still waiting for contributions