Worker/huey ignores REDIS_HOST and connects to localhost when OTEL_ENABLED=true (import-order race in settings)

[mgradalska]

What's happening

With OTEL_ENABLED=true and an OTLP endpoint configured, the huey worker (and the task-enqueue path in the API server) connect to localhost:6379 instead of the configured REDIS_HOST. The worker then fails in a tight retry loop:

huey/consumer.py", line 177, in loop
    task_list = self.huey.read_schedule(now)
...
redis.exceptions.ConnectionError: Error 99 connecting to localhost:6379. Cannot assign requested address.

The worker pegs CPU at 100% retrying. karrio-server logs the same error (it enqueues via the same broken huey instance). Background tasks never run.

This does not happen when OTel is disabled.

Root cause

In huey.contrib.djhuey, the module reads settings.HUEY once at import time. If that attribute isn't set yet, it falls back to constructing a default RedisHuey() - which connects to localhost:6379 - and that fallback instance is cached at module level for the lifetime of the process.

Karrio's settings package builds settings.HUEY in apps/api/karrio/server/settings/workers.py, and the settings __init__.py imports modules in this order:

from karrio.server.settings.base import *
from karrio.server.settings.apm import *      # <-- runs BEFORE workers
...
from karrio.server.settings.workers import *  # <-- sets settings.HUEY here

When OTEL_ENABLED=true and an OTLP endpoint is set, apm.py eagerly imports the OpenTelemetry instrumentation graph and calls RedisInstrumentor().instrument() at settings-load time. That import graph pulls in huey.contrib.djhuey - which runs getattr(settings, 'HUEY', None) before workers.py has executed. So djhuey sees None, builds its own default RedisHuey() pointed at localhost, and caches it forever.

workers.py then sets settings.HUEY correctly, but it's too late - djhuey already has its broken singleton.

When OTel is disabled, apm.py doesn't import the instrumentation graph, djhuey is imported later (after workers.py has run), and it correctly captures the configured instance.

Reproduction

Using karrio/server:2026.1.31 with Redis/Valkey reachable at host valkey (adjust to your setup):

Broken - OTel enabled:

OTEL_ENABLED=true \
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \
REDIS_HOST=valkey REDIS_PORT=6379 \
karrio shell -c "
from django.conf import settings
from huey.contrib.djhuey import HUEY as DJ
print('settings.HUEY host:', settings.HUEY.storage.conn.connection_pool.connection_kwargs.get('host'))
print('djhuey is settings.HUEY:', DJ is settings.HUEY)
print('djhuey host:', DJ.storage.conn.connection_pool.connection_kwargs.get('host'))
"

Output:

settings.HUEY host: valkey
djhuey is settings.HUEY: False
djhuey host: None              # -> connects to localhost:6379

Working - OTel disabled (same command without the two OTEL_* vars):

settings.HUEY host: valkey
djhuey is settings.HUEY: True
djhuey host: valkey

Impact

• Any deployment with OTel enabled (OTEL_ENABLED=true + OTLP endpoint) has a non-functional worker. Background jobs (tracking pulses, webhooks, data archiving) silently never run, and the worker burns a CPU core in a retry loop.
• The misleading symptom (localhost:6379) sends operators chasing Redis/networking config when REDIS_HOST is in fact set correctly. settings.HUEY looks right under inspection; only the djhuey singleton is wrong.

Notes on fixing

The fix needs to guarantee settings.HUEY is populated before huey.contrib.djhuey is first imported. A few directions (not validated here - leaving the choice to maintainers):

• Reorder the settings imports so workers.py runs before apm.py.
• Defer RedisInstrumentor().instrument() (and the OTel instrumentation imports) out of settings-load time.
• Avoid assigning a pre-built Huey instance to settings.HUEY in a way that's sensitive to import order.

Workaround for affected operators

Disable OTel on the server and worker (OTEL_ENABLED=false). djhuey then imports after workers.py runs and captures the correct broker connection. Costs you Karrio's OpenTelemetry tracing until a fix lands.