on_error task running twice when the task's timeout is exceeded

[MrEarle]

I've been having an issue with error handling in chains that timeout. Not sure if I'm doing something wrong or if it is a bug with Celery.

What I want to do

I want to run a callback in case the task fails or times out.

What I'm doing

# src/celery.py
from celery import Celery

app = Celery(
    "tasks",
    broker="redis://localhost",
    backend="redis://localhost",
    include=["src.tasks"],
)

if __name__ == "__main__":
    app.start()

# src/tasks.py
@app.task(time_limit=2)
def add(req_id: str, x: int, y: int) -> int:
    sleep(10)
    return x + y

@app.task
def log(*args):
    print("Error here", args[-1])

# main.py
from src.tasks import add, log

if __name__ == "__main__":
    s = add.s("myreq", 1, 2).on_error(log.s("myreq"))
    task = s.delay()

Running the worker: celery -A src worker -l INFO

Running the task: python main.py

Expected output

The log task is run once, and logs Error here myreq.

Real output

The log task ran twice and logs that message twice

[2024-02-14 12:40:47,295: INFO/MainProcess] Task src.tasks.add[443584d6-7cf4-43a8-a1b5-af772a8b9bc1] received
[2024-02-14 12:40:49,305: WARNING/MainProcess] Error here
[2024-02-14 12:40:49,305: WARNING/MainProcess]  
[2024-02-14 12:40:49,306: WARNING/MainProcess] myreq
[2024-02-14 12:40:49,306: ERROR/MainProcess] Task handler raised error: TimeLimitExceeded(2)
Traceback (most recent call last):
  File "/Users/benjaminearle/Documents/test-celery/.venv/lib/python3.9/site-packages/billiard/pool.py", line 683, in on_hard_timeout
    raise TimeLimitExceeded(job._timeout)
billiard.einfo.ExceptionWithTraceback: 
"""
Traceback (most recent call last):
  File "/Users/benjaminearle/Documents/test-celery/.venv/lib/python3.9/site-packages/billiard/pool.py", line 683, in on_hard_timeout
    raise TimeLimitExceeded(job._timeout)
billiard.exceptions.TimeLimitExceeded: TimeLimitExceeded(2,)
"""
[2024-02-14 12:40:49,306: ERROR/MainProcess] Hard time limit (2s) exceeded for src.tasks.add[443584d6-7cf4-43a8-a1b5-af772a8b9bc1]
[2024-02-14 12:40:49,308: WARNING/MainProcess] Error here
[2024-02-14 12:40:49,308: WARNING/MainProcess]  
[2024-02-14 12:40:49,308: WARNING/MainProcess] myreq
[2024-02-14 12:40:49,410: ERROR/MainProcess] Process 'ForkPoolWorker-8' pid:29151 exited with 'signal 9 (SIGKILL)'

---

Is there any way of running this error callback only once on timeout? It works as expected when an error is raised from within the task.