Skip to content

Add Redis list-based queue support with async hook#63937

Open
tardunge wants to merge 2 commits intoapache:mainfrom
tardunge:feature/redis-list-queue-support
Open

Add Redis list-based queue support with async hook#63937
tardunge wants to merge 2 commits intoapache:mainfrom
tardunge:feature/redis-list-queue-support

Conversation

@tardunge
Copy link
Copy Markdown

@tardunge tardunge commented Mar 19, 2026

Adds durable list-based messaging to the Redis provider using LPUSH/BRPOP, complementing the existing pub/sub support.

Motivation: Redis pub/sub is fire-and-forget — messages are lost if no subscriber is listening. List-based queues (LPUSH + BRPOP) provide message persistence until consumed, exactly-once delivery via atomic BRPOP, and priority queue patterns via ordered multi-list BRPOP.

New components:

  • RedisLPushOperator — push messages to Redis lists (FIFO queue producer)
  • AwaitMessageFromListTrigger — async trigger using BRPOP with priority queue support (multiple lists checked in order)
  • RedisListMessageQueueProvider — common-messaging integration with redis+list:// scheme
  • RedisHook.get_async_conn() — async context manager using redis.asyncio for truly non-blocking trigger operations (the existing hook is sync-only, which requires sync_to_async wrapping in triggers)

No new dependenciesredis.asyncio is part of redis>=4.2.0, and the provider already requires redis>=4.5.2.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.6)

Generated-by: Claude Code (Opus 4.6) following the guidelines

@potiuk
Copy link
Copy Markdown
Member

potiuk commented Mar 20, 2026

@tardunge This PR has a few issues that need to be addressed before it can be reviewed — please see our Pull Request quality criteria.

Issues found:

  • mypy (type checking): Failing: CI image checks / MyPy checks (mypy-providers). Run prek --stage manual mypy-providers --all-files locally to reproduce. You need breeze ci-image build --python 3.10 for Docker-based mypy. See mypy (type checking) docs.

What to do next:

  • The comment informs you what you need to do.
  • Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
  • There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates.
  • Maintainers will then proceed with a normal review.

There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the Airflow Slack.

TARDUNGE and others added 2 commits March 21, 2026 03:07
Adds durable list-based messaging to the Redis provider using LPUSH/BRPOP,
complementing the existing pub/sub support. List-based queues provide
message persistence until consumed and exactly-once delivery semantics
via atomic BRPOP, which pub/sub cannot guarantee.

New components:
- RedisLPushOperator: push messages to Redis lists
- AwaitMessageFromListTrigger: async trigger using BRPOP with
  priority queue support (multiple lists checked in order)
- RedisListMessageQueueProvider: common-messaging integration
  with redis+list:// scheme
- RedisHook.get_async_conn(): async context manager using
  redis.asyncio for truly non-blocking trigger operations
- Provide defaults for host/port/db to satisfy AsyncRedis constructor
  type requirements (str, int, int instead of Optional types)
- Use close() instead of aclose() for redis-py compatibility
@tardunge tardunge force-pushed the feature/redis-list-queue-support branch from e476eb6 to aeaa9c0 Compare March 20, 2026 19:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants