[Redis] Prioritized tasks are always enqueued as highest priority

[flbraun]

Checklist

• I have verified that the issue exists against the main branch of Celery.
• This has already been asked to the discussions forum first.
• I have read the relevant section in the
  contribution guide
  on reporting bugs.
• I have checked the issues list
  for similar or identical bug reports.
• I have checked the pull requests list
  for existing proposed fixes.
• I have checked the commit log
  to find out if the bug was already fixed in the main branch.
• I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

• I have included the output of celery -A proj report in the issue.
  (if you are not able to do this, then at least specify the Celery
  version affected).
• I have verified that the issue exists against the main branch of Celery.
• I have included the contents of pip freeze in the issue.
• I have included all the versions of all the external dependencies required
  to reproduce this bug.

Optional Debugging Information

• I have tried reproducing the issue on more than one Python version
  and/or implementation.
• I have tried reproducing the issue on more than one message broker and/or
  result backend.
• I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
• I have tried reproducing the issue on more than one operating system.
• I have tried reproducing the issue on more than one workers pool.
• I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
• I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

• None

Possible Duplicates

• None

Environment & Settings

Celery version: 5.3.6

celery report Output:

software -> celery:5.3.6 (emerald-rush) kombu:5.3.5 py:3.11.7
            billiard:4.2.0 redis:5.0.3
platform -> system:Linux arch:64bit
            kernel version:6.5.0-27-generic imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:disabled

broker_url: 'redis://redis.local.tonie.cloud:6379/8'
deprecated_settings: None
imports: ['tasks']
broker_transport_options: 
    'priority_steps': [0, 25, 50, 75, 100],
    'queue_order_strategy': 'priority',
    'sep': ':'}
task_default_priority: 50

Steps to Reproduce

Required Dependencies

• Minimal Python Version: N/A or Unknown
• Minimal Celery Version: N/A or Unknown
• Minimal Kombu Version: N/A or Unknown
• Minimal Broker Version: N/A or Unknown
• Minimal Result Backend Version: N/A or Unknown
• Minimal OS and/or Kernel Version: N/A or Unknown
• Minimal Broker Client Version: N/A or Unknown
• Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

Other Dependencies

N/A

Minimally Reproducible Test Case

#
# app.py
#
import os

from celery import Celery


app = Celery('tasks', broker=f"redis://{os.environ['REDIS_HOST']}:{os.environ['REDIS_PORT']}/8")
app.conf.imports = ['tasks']
app.conf.broker_transport_options = {
    'priority_steps': [0, 25, 50, 75, 100],
    'sep': ':',
    'queue_order_strategy': 'priority',
}
app.conf.task_default_priority = 50

#
# tasks.py
#
import time

from prio import app


@app.task
def default():
    print('default')
    time.sleep(1)


@app.task(priority=75)
def high():
    print('high')
    time.sleep(1)


if __name__ == '__main__':
    for i in range(10):
        default.delay()
    high.delay()

Run python tasks.py first to enqueue some tasks, then run celery -A prio:app worker --concurrency 1 to consume them.

Expected Behavior

Tasks are enqueued in the Redis key matching the tasks' priority.

Actual Behavior

When spawning a couple of tasks (see Minimally Reproducible Test Case) with a default priority of 50 and one task with a higher priority of 75, all tasks are enqueued into the highest priority key (in this case celery, which I assume is an alias for celery:0) instead of celery:50 and celery:75, respectively.
You can see this behavior using redis-cli MONITOR (task body truncated):

1713192234.524321 [8 172.20.0.22:44290] "LPUSH" "celery" "{... \"priority\": 50, ...}"
1713192234.525032 [8 172.20.0.22:44290] "LPUSH" "celery" "{... \"priority\": 50, ...}"
1713192234.525566 [8 172.20.0.22:44290] "LPUSH" "celery" "{... \"priority\": 50, ...}"
1713192234.526095 [8 172.20.0.22:44290] "LPUSH" "celery" "{... \"priority\": 50, ...}"
1713192234.526651 [8 172.20.0.22:44290] "LPUSH" "celery" "{... \"priority\": 50, ...}"
1713192234.527272 [8 172.20.0.22:44290] "LPUSH" "celery" "{... \"priority\": 50, ...}"
1713192234.527900 [8 172.20.0.22:44290] "LPUSH" "celery" "{... \"priority\": 50, ...}"
1713192234.528452 [8 172.20.0.22:44290] "LPUSH" "celery" "{... \"priority\": 50, ...}"
1713192234.528958 [8 172.20.0.22:44290] "LPUSH" "celery" "{... \"priority\": 50, ...}"
1713192234.529502 [8 172.20.0.22:44290] "LPUSH" "celery" "{... \"priority\": 50, ...}"
1713192234.530302 [8 172.20.0.22:44290] "LPUSH" "celery" "{... \"priority\": 75, ...}"

The worker polls tasks using the correct Redis command:

1713192822.408074 [8 172.20.0.22:59782] "BRPOP" "celery" "celery:25" "celery:50" "celery:75" "celery:100" "1"

But since all tasks ended up in the same key, all queued tasks are effectively worked on in the order they were enqueued, NOT by their priority.

I also tested this using 'priority_steps': [1, 25, 50, 75, 100] because I was wary of seeing the Redis key celery instead of the expected celery:0, but it yields the exact same results. All tasks are pushed into celery:1, just as they were pushed into celery before.