Cannot get flower to work with Redis in Docker

[parisk]

Hi everyone. I am trying to setup Flower in Docker to monitor my Celery workers and I can't seem to be able to get things working right.

What I am trying to do is run Flower in its own container, without direct access to my application's code and Celery configuration and.

I am able to run Flower successfully (without exceptions) and point it to the appropriate broker, using the following Docker Compose configuration:

flower:
    image: mher/flower
    command: ["--app sl.celery ", "--broker=redis://redis:6379/2"]

The problem is that although Flower connects successfully to the broker, it does not print out the actual tasks and workers registered for my Celery application:

[I 181108 17:56:27 command:139] Visit me at http://localhost:5555
[I 181108 17:56:27 command:144] Broker: redis://redis:6379/2
[I 181108 17:56:27 command:147] Registered tasks:
    ['celery.accumulate',
     'celery.backend_cleanup',
     'celery.chain',
     'celery.chord',
     'celery.chord_unlock',
     'celery.chunks',
     'celery.group',
     'celery.map',
     'celery.starmap']
[I 181108 17:56:27 mixins:224] Connected to redis://redis:6379/2

Despite that, when I connect to my Redis server, I can see the workers registered in the Broker:

# redis-cli
127.0.0.1:6379> select 2
OK
127.0.0.1:6379[2]> keys *
1) "_kombu.binding.default"
2) "_kombu.binding.celery.pidbox"
3) "_kombu.binding.celery"
4) "unacked_mutex"
5) "_kombu.binding.processes"
6) "_kombu.binding.celeryev"

Can you please help me set up Flower in its own isolated container, or let me know if that is not possible and I have to bundle it with my applications code?

Thanks a lot for your time 🙏.

[rahulkrish13]

This configuration works for me:

redis:
image: "redis:alpine"
expose:
- 6379
ports:
- "6379:6379"

flower:
image: mher/flower
command: ["flower", "--broker=redis://redis", "--port=5555"]
ports:
- 5555:5555
expose:
- 5555
depends_on:
- redis

[hbksagar]

@parisk Any luck. I am facing the same issue.

[parisk]

No luck. Registered tasks do not get displayed, unless the .py files of the Celery tasks are included.

[BartlomiejSkwira]

@parisk are you sure your celery broker is at redis:6379/2? I would double-check and grep this url.

I have it configured like so (.env):

CELERY_BACKEND_URL=redis://redis:6379/2
CELERY_BROKER_URL=redis://redis:6379/3

Notice the 3. So my docker-compose.yml

  flower:
    image: mher/flower
    ports:
      - "5555:5555"
    environment:
      - CELERY_BROKER_API=redis://redis:6379/2
      - CELERY_BROKER_URL=redis://redis:6379/3

[denisvolokh]

Hi,

I am experiencing same problem with not getting registered tasks and workers at Flower.

Here is my setup for Celery:

and Flower:

Both Celery and Flower are using broker at the same address.

  redis:
    image: 'redis:4-alpine'
    ports:
      - 6379:6379

  celery:
    build:
      dockerfile: celery.dockerfile
      context: ./
    volumes:
      - ./:/code
    environment:
      - C_FORCE_ROOT=true
    command:
      - bash
      - -c
      - |
        rm celerybeat.pid
        python manage.py celery worker --loglevel=DEBUG -B -E
    depends_on:
      - redis
    links:
      - redis

  flower:  
    image: mher/flower
    environment:
      - CELERY_BROKER_URL=redis://redis/2
      - CELERY_BACKEND_URL=redis://redis/1
      - FLOWER_PORT=8888
    command: ["flower", "--broker=redis://redis/2", "--port=8888"]
    ports:  
      - 8888:8888

I would appreciate if someone could point at the right setup for Celery and Flower.

Thanks.

[parisk]

Wow, I hadn't taken a look at my notifications for this issue for quite a while and there is considerable activity.

So, I have solved the issue and have been running Flower in Docker Swarm successfully now. There are two things to have in mind:

1. You need to provide BOTH Celery and Flower configuration
2. Although Flower might NOT print the registered tasks at first, they should appear in the dashboard within seconds

So, I will move forward and close this issue, while providing my setup.

Setup

docker-compose.yml

version: "3.7"

services:
  flower:
    image: docker.pkg.github.com/parisk/flower/web:latest
    build: .
    secrets:
      - broker_url
      - auth
      - oauth2_key
      - oauth2_secret
      - oauth2_redirect_uri

secrets:
  broker_url:
    external: true
    name: flower-broker-url-v1
  auth:
    external: true
    name: flower-auth-v1
  oauth2_key:
    external: true
    name: flower-oauth2-key-v1
  oauth2_secret:
    external: true
    name: flower-oauth2-secret-v1
  oauth2_redirect_uri:
    external: true
    name: flower-oauth2-redirect-uri-v1

Dockerfile

FROM python:3.6

RUN pip install --upgrade pip==19.2.3
RUN pip install --upgrade pipenv==2018.11.26

WORKDIR /usr/src/app

ADD Pipfile Pipfile.lock ./
RUN pipenv install --system --deploy

ADD flowerconfig.py celeryconfig.py ./

ENTRYPOINT ["flower"]

Pipfile

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
flower = "*"
sec = "*"
redis = "*"

[requires]
python_version = "3.6"

celeryconfig.py

import sec


broker_url = sec.load("broker_url")

flowerconfig.py

import sec


address = "0.0.0.0"

auth = sec.load("auth")
oauth2_key = sec.load("oauth2_key")
oauth2_secret = sec.load("oauth2_secret")
oauth2_redirect_uri = sec.load("oauth2_redirect_uri")

[denisvolokh]

Hi @parisk

Thank you for your response and example but it did not work for me.

I have created a new flower image based on the latest flower code (version 0.9.3)

flower.dockerfile

FROM python:alpine

# Get latest root certificates
RUN apk add --no-cache ca-certificates && update-ca-certificates

# Install the required packages
RUN pip install --no-cache-dir redis flower

# PYTHONUNBUFFERED: Force stdin, stdout and stderr to be totally unbuffered. (equivalent to `python -u`)
# PYTHONHASHSEED: Enable hash randomization (equivalent to `python -R`)
# PYTHONDONTWRITEBYTECODE: Do not write byte files to disk, since we maintain it as readonly. (equivalent to `python -B`)
ENV PYTHONUNBUFFERED=1 PYTHONHASHSEED=random PYTHONDONTWRITEBYTECODE=1

# Default port
EXPOSE 5555

# Run as a non-root user by default, run as user with least privileges.
# USER nobody

ENTRYPOINT ["flower"]

and start it as services in docker compose:

celery:
    build:
      dockerfile: celery.dockerfile
      context: ./
    volumes:
      - ./:/code
    environment:
      - C_FORCE_ROOT=true
    command:
      - bash
      - -c
      - |
        rm celerybeat.pid
        python manage.py celery worker --loglevel=DEBUG -B -E
    depends_on:
      - redis
    links:
      - redis

  flower:  
    build:
      dockerfile: flower.dockerfile
      context: ./
    environment:
      - broker_url=redis://redis/2
      - CELERY_BROKER_URL=redis://redis/2
      - CELERY_BACKEND_URL=redis://redis/1
      - FLOWER_PORT=8888
    command: ["flower", "--broker=redis://redis/2", "--port=8888"]
    ports:  
      - 8888:8888
    links:
      - redis

Flower still does not show any workers or tasks.

[sudhirbatchu9383]

+1
Same problem

[denisvolokh]

I found what was the problem in my case.
If you start celery under Django make sure you start flower under Django too.

So the fix would be:

python manage.py celery flower --broker=redis://redis/2 --port=8888