Celery multi 4.4.6 not working due to /var/run/celery mkdir #6199

Andrew-Chen-Wang · 2020-06-30T01:07:54Z

Checklist

I have verified that the issue exists against the master branch of Celery.
This has already been asked to the discussion group first.
I have read the relevant section in the
contribution guide
on reporting bugs.
I have checked the issues list
for similar or identical bug reports.
I have checked the pull requests list
for existing proposed fixes.
I have checked the commit log
to find out if the bug was already fixed in the master branch.
I have included all related issues and possible duplicate issues
in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.
(if you are not able to do this, then at least specify the Celery
version affected).
I have verified that the issue exists against the master branch of Celery.
I have included the contents of pip freeze in the issue.
I have included all the versions of all the external dependencies required
to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version
and/or implementation.
I have tried reproducing the issue on more than one message broker and/or
result backend.
I have tried reproducing the issue on more than one version of the message
broker and/or result backend.
I have tried reproducing the issue on more than one operating system.
I have tried reproducing the issue on more than one workers pool.
I have tried reproducing the issue with autoscaling, retries,
ETA/Countdown & rate limits disabled.
I have tried reproducing the issue after downgrading
and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version:

celery report Output:

celery multi start worker beat -A config.celery_app --pool=solo
Traceback

celery report
software -> celery:4.4.6 (cliffs) kombu:4.6.11 py:3.8.0
564            billiard:3.6.3.0 redis:3.5.0
565platform -> system:Linux arch:64bit, ELF
566            kernel version:4.15.0-1077-gcp imp:CPython
567loader   -> celery.loaders.default.Loader
568settings -> transport:redis results:disabled

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown
Minimal Celery Version: 4.4.5 (not working) vs 4.4.2 (working)
Minimal Kombu Version: N/A or Unknown
Minimal Broker Version: N/A or Unknown
Minimal Result Backend Version: N/A or Unknown
Minimal OS and/or Kernel Version: N/A or Unknown
Minimal Broker Client Version: N/A or Unknown
Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

Other Dependencies

You can find them in the linked repository below

Minimally Reproducible Test Case

Expected Behavior

The celery task should work

Actual Behavior

It fails. Please visit my test repository here: https://travis-ci.com/github/Andrew-Chen-Wang/test-cookiecutter-django-cache/jobs/355630502

The job that succeeds use celery v4.4.2. On a package, we upgraded to 4.4.5, but it stopped working, saying PermissionError: [Errno 13] Permission denied: '/var/run/celery'

The text was updated successfully, but these errors were encountered:

auvipy · 2020-06-30T04:22:22Z

please check the changelog and commits.

Andrew-Chen-Wang · 2020-06-30T04:26:42Z

@auvipy Still a problem on 4.4.6 and 4.4.5, even though fixed on 4.4.4. https://travis-ci.com/github/Andrew-Chen-Wang/test-cookiecutter-django-cache/jobs/355630502

auvipy · 2020-06-30T04:27:41Z

it was a buggy feature initially. currently, it has been fixed

Andrew-Chen-Wang · 2020-06-30T04:32:01Z

The changelog says it was fixed in 4.4.4, right? The linked Travis job used 4.4.6. Unless you're asking me to look into the master branch.

Edit: the hyperlink over 4.4.6.

Andrew-Chen-Wang · 2020-06-30T04:55:30Z

@auvipy It's still a problem on 4.4.6. Another user commented on the changelog's linked PR 6 days ago saying it still didn't work on 4.4.5. In the Travis config, you can see that it doesn't work when setting the PID in line 490.

I checked the PR, and there was a test for the PID default to be in /var/run/celery which is where the error occurred in the first place.

Going down the rabbit hole, we land in this issue saying that some file locations were incorrectly located in that they shouldn't be in the working directory. Note, the user was on 4.4.2, but 4.4.2 works correctly and has been for a long time for me. I also linked to a Travis job for 4.4.2 for the same configuration and repo and it succeeded.

thedrow · 2020-07-27T08:45:39Z

Could you actually run celery report and edit the description?

thedrow · 2020-07-27T08:48:19Z

I'm confused. You're saying #6136 didn't fix the original issue?

Andrew-Chen-Wang · 2020-07-27T14:41:25Z

Hi @thedrow thanks for commenting. No, it didn't. I managed to at least get a checkmark in Travis, but celery won't run (this job includes celery report, but there's nothing special in there:

software -> celery:4.4.6 (cliffs) kombu:4.6.11 py:3.8.0
564            billiard:3.6.3.0 redis:3.5.0
565platform -> system:Linux arch:64bit, ELF
566            kernel version:4.15.0-1077-gcp imp:CPython
567loader   -> celery.loaders.default.Loader
568settings -> transport:redis results:disabled

unless that's all there is).

The Travis builds in that link is from me testing as many configurations as possible. The one that worked was making the directories for /var/run/celery, but I've never had to do that before with 4.4.2.

thedrow · 2020-07-28T16:01:13Z

@auvipy Why does this issue need verification?
We've added this to Travis ourselves too.

Andrew-Chen-Wang · 2020-07-28T21:57:55Z

It seems like when I tried to use the chown command myself, it didn't work. But using your Travis configuration made it work, so thank you for that!

I'm still not sure why 4.4.3 to 4.4.6 forced us to perform these commands. The last version in which I didn't need to create new directories and chown was 4.4.2. Perhaps it's a bug in Travis? Some kind of permission basis that celery had to circumvent before?

raiderrobert · 2020-07-29T13:26:49Z

We have a similar issue that emerged yesterday when upgrading to 4.4.6. The crux of the problem is that Celery was ignoring our config that specified where to put the logs. See the traceback below.

celery init v10.0.
Using config script: /etc/default/celeryd.drip_emails
celery multi v4.4.6 (cliffs)
Traceback (most recent call last):
  File "/usr/local/bin/celery", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/celery/__main__.py", line 16, in main
    _main()
  File "/usr/local/lib/python3.6/dist-packages/celery/bin/celery.py", line 322, in main
    cmd.execute_from_commandline(argv)
  File "/usr/local/lib/python3.6/dist-packages/celery/bin/celery.py", line 495, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/usr/local/lib/python3.6/dist-packages/celery/bin/base.py", line 305, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/usr/local/lib/python3.6/dist-packages/celery/bin/celery.py", line 487, in handle_argv
    return self.execute(command, argv)
  File "/usr/local/lib/python3.6/dist-packages/celery/bin/celery.py", line 419, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/usr/local/lib/python3.6/dist-packages/celery/bin/celery.py", line 335, in run_from_argv
    return cmd.execute_from_commandline([command] + argv)
  File "/usr/local/lib/python3.6/dist-packages/celery/bin/multi.py", line 272, in execute_from_commandline
    return self.call_command(argv[0], argv[1:])
  File "/usr/local/lib/python3.6/dist-packages/celery/bin/multi.py", line 279, in call_command
    return self.commands[command](*argv) or EX_OK
  File "/usr/local/lib/python3.6/dist-packages/celery/bin/multi.py", line 149, in _inner
    return fun(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/celery/bin/multi.py", line 165, in _inner
    p, cluster = self._cluster_from_argv(argv)
  File "/usr/local/lib/python3.6/dist-packages/celery/bin/multi.py", line 372, in _cluster_from_argv
    return p, self.Cluster(list(nodes), cmd=cmd)
  File "/usr/local/lib/python3.6/dist-packages/celery/apps/multi.py", line 304, in <genexpr>
    for name in names
  File "/usr/local/lib/python3.6/dist-packages/celery/apps/multi.py", line 312, in _node_from_options
    p.optmerge(namespace, options), p.passthrough)
  File "/usr/local/lib/python3.6/dist-packages/celery/apps/multi.py", line 136, in __init__
    options or OrderedDict())
  File "/usr/local/lib/python3.6/dist-packages/celery/apps/multi.py", line 144, in _annotate_with_default_opts
    self._setdefaultopt(options, ['--logfile', '-f'], '/var/log/celery/%n%I.log')
  File "/usr/local/lib/python3.6/dist-packages/celery/apps/multi.py", line 157, in _setdefaultopt
    os.makedirs(dir_path)
  File "/usr/lib/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/var/log/celery'

Basically, we have a configuration file that specifies to use a location other than /var/log/celery, which appears to be the default fallback, and that default location doesn't exist on our system--nor do we want it to. We want it to use our config. It seems that for whatever reason after Celery version 4.4.2 for the initial start-up and running it generally uses the config file, but on shut down (or restart), it's attempting to log out to default location and ignoring the passed in config. Since it gives that error, we could not get the workers to respond to commands to shutdown, and we're forced to kill the workers from the OS layer.

Our solution in the near term was to downgrade to Celery 4.4.2, as that version seems to have the expected behavior.

thedrow · 2020-07-30T09:33:58Z

@auvipy Please investigate and provide a fix.

Andrew-Chen-Wang · 2020-09-11T17:11:37Z

@raiderrobert Does the problem persist? I'm not using a configuration like yours, and my Travis works fine when I include the chown command. However, I haven't tried specifying a new location which seems to be the issue for two other issues and a PR that tried to resolve it.

I also haven't tried using Celery without the chown command. I believe a new version of Celery came out, so you can try your configuration again. (Version 4.4.7 is the latest as of today)

daveisagit · 2020-11-09T10:59:19Z

I am trying to switch from worker to multi. Worker is fine but I'm getting this same permissions issue with multi in docker.
Using 4.4.7 with a non root user in the image. I tried chown prior to call celery multi but to no avail.

chown: cannot access '/var/log/celery/': No such file or directory

chown: cannot access '/var/run/celery/': No such file or directory

celery multi v4.4.7 (cliffs)

Traceback (most recent call last):

File "/usr/local/bin/celery", line 8, in <module>

sys.exit(main())

File "/usr/local/lib/python3.8/site-packages/celery/__main__.py", line 16, in main

_main()

File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 322, in main

cmd.execute_from_commandline(argv)

File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 499, in execute_from_commandline

super(CeleryCommand, self).execute_from_commandline(argv)))

File "/usr/local/lib/python3.8/site-packages/celery/bin/base.py", line 305, in execute_from_commandline

return self.handle_argv(self.prog_name, argv[1:])

File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 491, in handle_argv

return self.execute(command, argv)

File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 415, in execute

return cls(

File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 335, in run_from_argv

return cmd.execute_from_commandline([command] + argv)

File "/usr/local/lib/python3.8/site-packages/celery/bin/multi.py", line 272, in execute_from_commandline

return self.call_command(argv[0], argv[1:])

File "/usr/local/lib/python3.8/site-packages/celery/bin/multi.py", line 279, in call_command

return self.commands[command](*argv) or EX_OK

File "/usr/local/lib/python3.8/site-packages/celery/bin/multi.py", line 149, in _inner

return fun(self, *args, **kwargs)

File "/usr/local/lib/python3.8/site-packages/celery/bin/multi.py", line 157, in _inner

return fun(self, self.cluster_from_argv(argv), **kwargs)

File "/usr/local/lib/python3.8/site-packages/celery/bin/multi.py", line 367, in cluster_from_argv

_, cluster = self._cluster_from_argv(argv, cmd=cmd)

File "/usr/local/lib/python3.8/site-packages/celery/bin/multi.py", line 372, in _cluster_from_argv

return p, self.Cluster(list(nodes), cmd=cmd)

File "/usr/local/lib/python3.8/site-packages/celery/apps/multi.py", line 302, in <genexpr>

self._node_from_options(

File "/usr/local/lib/python3.8/site-packages/celery/apps/multi.py", line 311, in _node_from_options

return Node(nodename, cmd, append,

File "/usr/local/lib/python3.8/site-packages/celery/apps/multi.py", line 135, in __init__

self.options = self._annotate_with_default_opts(

File "/usr/local/lib/python3.8/site-packages/celery/apps/multi.py", line 143, in _annotate_with_default_opts

self._setdefaultopt(options, ['--pidfile', '-p'], '/var/run/celery/%n.pid')

File "/usr/local/lib/python3.8/site-packages/celery/apps/multi.py", line 157, in _setdefaultopt

os.makedirs(dir_path)

File "/usr/local/lib/python3.8/os.py", line 223, in makedirs

mkdir(name, mode)

PermissionError: [Errno 13] Permission denied: '/var/run/celery'

celery multi v4.4.7 (cliffs)

Traceback (most recent call last):

File "/usr/local/bin/celery", line 8, in <module>

sys.exit(main())

File "/usr/local/lib/python3.8/site-packages/celery/__main__.py", line 16, in main

_main()

File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 322, in main

cmd.execute_from_commandline(argv)

File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 499, in execute_from_commandline

super(CeleryCommand, self).execute_from_commandline(argv)))

File "/usr/local/lib/python3.8/site-packages/celery/bin/base.py", line 305, in execute_from_commandline

return self.handle_argv(self.prog_name, argv[1:])

File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 491, in handle_argv

return self.execute(command, argv)

File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 415, in execute

return cls(

File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 335, in run_from_argv

return cmd.execute_from_commandline([command] + argv)

File "/usr/local/lib/python3.8/site-packages/celery/bin/multi.py", line 272, in execute_from_commandline

return self.call_command(argv[0], argv[1:])

File "/usr/local/lib/python3.8/site-packages/celery/bin/multi.py", line 279, in call_command

return self.commands[command](*argv) or EX_OK

File "/usr/local/lib/python3.8/site-packages/celery/bin/multi.py", line 149, in _inner

return fun(self, *args, **kwargs)

File "/usr/local/lib/python3.8/site-packages/celery/bin/multi.py", line 157, in _inner

return fun(self, self.cluster_from_argv(argv), **kwargs)

File "/usr/local/lib/python3.8/site-packages/celery/bin/multi.py", line 367, in cluster_from_argv

_, cluster = self._cluster_from_argv(argv, cmd=cmd)

File "/usr/local/lib/python3.8/site-packages/celery/bin/multi.py", line 372, in _cluster_from_argv

return p, self.Cluster(list(nodes), cmd=cmd)

File "/usr/local/lib/python3.8/site-packages/celery/apps/multi.py", line 302, in <genexpr>

self._node_from_options(

File "/usr/local/lib/python3.8/site-packages/celery/apps/multi.py", line 311, in _node_from_options

return Node(nodename, cmd, append,

File "/usr/local/lib/python3.8/site-packages/celery/apps/multi.py", line 135, in __init__

self.options = self._annotate_with_default_opts(

File "/usr/local/lib/python3.8/site-packages/celery/apps/multi.py", line 143, in _annotate_with_default_opts

self._setdefaultopt(options, ['--pidfile', '-p'], '/var/run/celery/%n.pid')

File "/usr/local/lib/python3.8/site-packages/celery/apps/multi.py", line 157, in _setdefaultopt

os.makedirs(dir_path)

File "/usr/local/lib/python3.8/os.py", line 223, in makedirs

mkdir(name, mode)

PermissionError: [Errno 13] Permission denied: '/var/run/celery'

thedrow · 2020-11-09T11:04:14Z

Can't you change the path of the PID files?

daveisagit · 2020-11-09T11:14:14Z

Thanks @thedrow, yes using --pidfile=./my.pid --logfile=./my.log works. I using celery-beat and structured logging to stdout/stderr for kubernetes target. Is it the case that you still need the pid / log regardless ?

daveisagit · 2020-11-09T17:02:07Z

The docs https://docs.celeryproject.org/en/stable/reference/celery.bin.multi.html say

$ # Pidfiles and logfiles are stored in the current directory
$ # by default.  Use --pidfile and --logfile argument to change
$ # this.  The abbreviation %n will be expanded to the current
$ # node name.

So why is it trying to create them in /var/run and /var/log?

thedrow · 2020-12-07T10:01:48Z

@auvipy Which PR did you merge that caused this?

auvipy · 2020-12-13T16:19:44Z

probably 1561cad one

thedrow · 2020-12-13T16:30:09Z

No. It's this.
14c1baa
Why did we change the default paths?

auvipy · 2020-12-13T16:45:52Z

this was change as per #6017 as far as I can recall

thedrow · 2020-12-14T10:47:30Z

Again, I'm asking Why the change was made.

auvipy · 2020-12-14T12:30:29Z

Again, I'm asking Why the change was made.

apparently, the issue was a false-positive and all the upcoming 2-3 PR was merged based on that false-positive report I failed to verify/reason properly! @kwist-sgr @hanchau @mchataigner can you recheck, please?

auvipy · 2020-12-14T12:35:41Z

The docs https://docs.celeryproject.org/en/stable/reference/celery.bin.multi.html say
$ # Pidfiles and logfiles are stored in the current directory
$ # by default.  Use --pidfile and --logfile argument to change
$ # this.  The abbreviation %n will be expanded to the current
$ # node name.
So why is it trying to create them in /var/run and /var/log?

can you please check https://docs.celeryproject.org/en/latest/userguide/daemonizing.html and let us know we should undo the change or update the docs?

tpickle-py · 2021-08-17T20:02:26Z

I am running into issues with demonizing this and specifying the log file location as well as the run pid. Any and all help is welcomed here. Thanks in advance.
I am overwriting this in the service options, Permission denied: '/var/log/celery' , and I am running v5.1.2

Systemctl output of service status

Aug 17 14:25:30 SERVER sh[18099]: celery multi v5.1.2 (sun-harmonics)
Aug 17 14:25:30 SERVER sh[18099]: > Starting nodes...
Aug 17 14:25:30 SERVER sh[18099]: > w2@SERVER: OK
Aug 17 14:25:30 SERVER systemd[1]: Started Celery Service for App.
Aug 17 14:25:35 SERVER systemd[1]: celery.service: main process exited, code=exited, status=1/FAILURE
Aug 17 14:25:36 SERVER sh[18116]: celery multi v5.1.2 (sun-harmonics)
Aug 17 14:25:36 SERVER sh[18116]: Traceback (most recent call last):
Aug 17 14:25:36 SERVER sh[18116]: File "/app/.venv/bin/celery", line 8, in <module>
Aug 17 14:25:36 SERVER sh[18116]: sys.exit(main())
Aug 17 14:25:36 SERVER sh[18116]: File "/app/.venv/lib/python3.7/site-packages/celery/__main__.py", line 15, in main
Aug 17 14:25:36 SERVER sh[18116]: sys.exit(_main())
Aug 17 14:25:36 SERVER sh[18116]: File "/app/.venv/lib/python3.7/site-packages/celery/bin/celery.py", line 213, in main
Aug 17 14:25:36 SERVER sh[18116]: return celery(auto_envvar_prefix="CELERY")
Aug 17 14:25:36 SERVER sh[18116]: File "/app/.venv/lib/python3.7/site-packages/click/core.py", line 829, in __call__
Aug 17 14:25:36 SERVER sh[18116]: return self.main(*args, **kwargs)
Aug 17 14:25:36 SERVER sh[18116]: File "/app/.venv/lib/python3.7/site-packages/click/core.py", line 782, in main
Aug 17 14:25:36 SERVER sh[18116]: rv = self.invoke(ctx)
Aug 17 14:25:36 SERVER sh[18116]: File "/app/.venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
Aug 17 14:25:36 SERVER sh[18116]: return _process_result(sub_ctx.command.invoke(sub_ctx))
Aug 17 14:25:36 SERVER sh[18116]: File "/app/.venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
Aug 17 14:25:36 SERVER sh[18116]: return ctx.invoke(self.callback, **ctx.params)
Aug 17 14:25:36 SERVER sh[18116]: File "/app/.venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
Aug 17 14:25:36 SERVER sh[18116]: return callback(*args, **kwargs)
Aug 17 14:25:36 SERVER sh[18116]: File "/app/.venv/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
Aug 17 14:25:36 SERVER sh[18116]: return f(get_current_context(), *args, **kwargs)
Aug 17 14:25:36 SERVER sh[18116]: File "/app/.venv/lib/python3.7/site-packages/celery/bin/base.py", line 133, in caller
Aug 17 14:25:36 SERVER sh[18116]: return f(ctx, *args, **kwargs)
Aug 17 14:25:36 SERVER sh[18116]: File "/app/.venv/lib/python3.7/site-packages/celery/bin/multi.py", line 480, in multi
Aug 17 14:25:36 SERVER sh[18116]: return cmd.execute_from_commandline(args)
Aug 17 14:25:36 SERVER sh[18116]: File "/app/.venv/lib/python3.7/site-packages/celery/bin/multi.py", line 271, in execute_from_commandl
Aug 17 14:25:36 SERVER sh[18116]: return self.call_command(argv[0], argv[1:])
Aug 17 14:25:36 SERVER sh[18116]: File "/app/.venv/lib/python3.7/site-packages/celery/bin/multi.py", line 278, in call_command
Aug 17 14:25:36 SERVER sh[18116]: return self.commands[command](*argv) or EX_OK
Aug 17 14:25:36 SERVER sh[18116]: File "/app/.venv/lib/python3.7/site-packages/celery/bin/multi.py", line 148, in _inner
Aug 17 14:25:36 SERVER sh[18116]: return fun(self, *args, **kwargs)
Aug 17 14:25:36 SERVER sh[18116]: File "/app/.venv/lib/python3.7/site-packages/celery/bin/multi.py", line 164, in _inner
Aug 17 14:25:36 SERVER sh[18116]: p, cluster = self._cluster_from_argv(argv)
Aug 17 14:25:36 SERVER sh[18116]: File "/app/.venv/lib/python3.7/site-packages/celery/bin/multi.py", line 371, in _cluster_from_argv
Aug 17 14:25:36 SERVER sh[18116]: return p, self.Cluster(list(nodes), cmd=cmd)
Aug 17 14:25:36 SERVER sh[18116]: File "/app/.venv/lib/python3.7/site-packages/celery/apps/multi.py", line 321, in <genexpr>
Aug 17 14:25:36 SERVER sh[18116]: for name in names
Aug 17 14:25:36 SERVER sh[18116]: File "/app/.venv/lib/python3.7/site-packages/celery/apps/multi.py", line 329, in _node_from_options
Aug 17 14:25:36 SERVER sh[18116]: p.optmerge(namespace, options), p.passthrough)
Aug 17 14:25:36 SERVER sh[18116]: File "/app/.venv/lib/python3.7/site-packages/celery/apps/multi.py", line 133, in __init__
Aug 17 14:25:36 SERVER sh[18116]: options or OrderedDict())
Aug 17 14:25:36 SERVER sh[18116]: File "/app/.venv/lib/python3.7/site-packages/celery/apps/multi.py", line 141, in _annotate_with_defau
Aug 17 14:25:36 SERVER sh[18116]: self._setdefaultopt(options, ['--logfile', '-f'], '/var/log/celery/%n%I.log')
Aug 17 14:25:36 SERVER sh[18116]: File "/app/.venv/lib/python3.7/site-packages/celery/apps/multi.py", line 154, in _setdefaultopt
Aug 17 14:25:36 SERVER sh[18116]: os.makedirs(dir_path)
Aug 17 14:25:36 SERVER sh[18116]: File "/usr/local/lib/python3.7/os.py", line 223, in makedirs
Aug 17 14:25:36 SERVER sh[18116]: mkdir(name, mode)
Aug 17 14:25:36 SERVER sh[18116]: PermissionError: [Errno 13] Permission denied: '/var/log/celery'

Service File

[Unit]
Description=Celery Service for app
After=network.target

[Service]
Type=forking
User=app
Group=app_b
EnvironmentFile=/app/.env
WorkingDirectory=/app
ExecStart=/bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi start $CELERYD_NODES --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} --loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait $CELERYD_NODES --pidfile=${CELERYD_PID_FILE} --loglevel="${CELERYD_LOG_LEVEL}"'
ExecReload=/bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi restart $CELERYD_NODES --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} --loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS'
Restart=always
StandardOutput=journal
StandardError=journal

[Install]

VARIABLE File

#CELERY_APP
CELERYD_NODES="w2"
CELERY_BIN="app/.venv/bin/celery"
# App instance to use
# comment out this line if you don't use an app
CELERY_APP="firewallmgr"
CELERYD_MULTI="multi"
CELERYD_OPTS="--time-limit=300 --concurrency=4"
CELERYD_PID_FILE="/app/a01/run/celery/%n.pid"
CELERYD_LOG_FILE="/app/a01/log/celery/%n%I.log"
CELERYD_LOG_LEVEL="INFO"
CELERYBEAT_PID_FILE="/app/a01/run/celery/beat.pid"
CELERYBEAT_LOG_FILE="/app/a01/log/celery/beat.log"

kwist-sgr · 2021-08-23T16:21:12Z

I can't reproduce this issue with celery 5.1.2 (sun-harmonics)

celery/celery#6199 Signed-off-by: David Galloway <dgallowa@redhat.com>

maxmalysh · 2023-05-18T23:58:08Z

The problem is still present in celery==5.2.7.

We launch Celery via systemd. Our log path is defined this way:

CELERYD_LOG_FILE="/home/example/log/celery/%n%I.log"

This works and we see logs in this directory. However, during worker startup we see this exception in Sentry:

PermissionError
[Errno 13] Permission denied: '/var/log/celery'

This is the place where multi tries to create the default log directory (/var/log/celery) even if we don't want this:

# celery/apps/multi.py in _setdefaultopt at line 154

151    value = d.setdefault(alt[0], os.path.normpath(value))
152    dir_path = os.path.dirname(value)
153    if dir_path and not os.path.exists(dir_path):
154        os.makedirs(dir_path)
155    return value

gpmarques · 2023-11-16T21:30:10Z

I'm having the same issue as @maxmalysh, is there a fix for this?

auvipy · 2023-11-19T13:09:18Z

so 1561cad didn't fix the issue, right?

Andrew-Chen-Wang added the Issue Type: Bug Report label Jun 30, 2020

Andrew-Chen-Wang changed the title ~~Celery not working on Travis~~ Celery 4.4.5 not working on Travis Jun 30, 2020

Andrew-Chen-Wang changed the title ~~Celery 4.4.5 not working on Travis~~ Celery 4.4.6 not working on Travis Jun 30, 2020

Andrew-Chen-Wang mentioned this issue Jun 30, 2020

Add PostgreSQL 12 support cookiecutter/cookiecutter-django#2593

Closed

5 tasks

auvipy closed this as completed Jun 30, 2020

Andrew-Chen-Wang changed the title ~~Celery 4.4.6 not working on Travis~~ Celery multi 4.4.6 not working due to /var/run/celery mkdir Jun 30, 2020

thedrow mentioned this issue Jul 27, 2020

Celery 4.4.3 always trying create /var/run/celery directory, even if it's not needed. #6136

Closed

thedrow reopened this Jul 27, 2020

auvipy added the Status: Needs Verification ✘ label Jul 28, 2020

thedrow assigned auvipy Jul 28, 2020

auvipy mentioned this issue Jul 29, 2020

fix lint violations and CI errors: use rabbitmq via docker and only for integration-rabbitmq tests #6253

Merged

auvipy added this to the 4.4.x milestone Jul 30, 2020

auvipy modified the milestones: 4.4.x, 4.5 Feb 18, 2021

auvipy modified the milestones: 4.5, 5.3 Sep 16, 2021

djgalloway pushed a commit to ceph/mita that referenced this issue Jan 5, 2022

deploy: Create a PID file for celery

c007ceb

celery/celery#6199 Signed-off-by: David Galloway <dgallowa@redhat.com>

auvipy modified the milestones: 5.3, 5.3.x Dec 2, 2022

auvipy modified the milestones: 5.3.x, 5.4.x Nov 19, 2023

auvipy removed their assignment Nov 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Celery multi 4.4.6 not working due to /var/run/celery mkdir #6199

Celery multi 4.4.6 not working due to /var/run/celery mkdir #6199

Andrew-Chen-Wang commented Jun 30, 2020 •

edited by sync-by-unito bot

auvipy commented Jun 30, 2020

Andrew-Chen-Wang commented Jun 30, 2020 •

edited

auvipy commented Jun 30, 2020

Andrew-Chen-Wang commented Jun 30, 2020 •

edited

Andrew-Chen-Wang commented Jun 30, 2020 •

edited

thedrow commented Jul 27, 2020

thedrow commented Jul 27, 2020

Andrew-Chen-Wang commented Jul 27, 2020

thedrow commented Jul 28, 2020

Andrew-Chen-Wang commented Jul 28, 2020

raiderrobert commented Jul 29, 2020 •

edited

thedrow commented Jul 30, 2020

Andrew-Chen-Wang commented Sep 11, 2020 •

edited

daveisagit commented Nov 9, 2020

thedrow commented Nov 9, 2020

daveisagit commented Nov 9, 2020

daveisagit commented Nov 9, 2020

thedrow commented Dec 7, 2020

auvipy commented Dec 13, 2020

thedrow commented Dec 13, 2020

auvipy commented Dec 13, 2020

thedrow commented Dec 14, 2020

auvipy commented Dec 14, 2020

auvipy commented Dec 14, 2020

tpickle-py commented Aug 17, 2021

kwist-sgr commented Aug 23, 2021

maxmalysh commented May 18, 2023

gpmarques commented Nov 16, 2023

auvipy commented Nov 19, 2023

Celery multi 4.4.6 not working due to /var/run/celery mkdir #6199

Celery multi 4.4.6 not working due to /var/run/celery mkdir #6199

Comments

Andrew-Chen-Wang commented Jun 30, 2020 • edited by sync-by-unito bot

Checklist

Mandatory Debugging Information

Optional Debugging Information

Related Issues and Possible Duplicates

Related Issues

Possible Duplicates

Environment & Settings

Steps to Reproduce

Required Dependencies

Python Packages

Other Dependencies

Minimally Reproducible Test Case

Expected Behavior

Actual Behavior

auvipy commented Jun 30, 2020

Andrew-Chen-Wang commented Jun 30, 2020 • edited

auvipy commented Jun 30, 2020

Andrew-Chen-Wang commented Jun 30, 2020 • edited

Andrew-Chen-Wang commented Jun 30, 2020 • edited

thedrow commented Jul 27, 2020

thedrow commented Jul 27, 2020

Andrew-Chen-Wang commented Jul 27, 2020

thedrow commented Jul 28, 2020

Andrew-Chen-Wang commented Jul 28, 2020

raiderrobert commented Jul 29, 2020 • edited

thedrow commented Jul 30, 2020

Andrew-Chen-Wang commented Sep 11, 2020 • edited

daveisagit commented Nov 9, 2020

thedrow commented Nov 9, 2020

daveisagit commented Nov 9, 2020

daveisagit commented Nov 9, 2020

thedrow commented Dec 7, 2020

auvipy commented Dec 13, 2020

thedrow commented Dec 13, 2020

auvipy commented Dec 13, 2020

thedrow commented Dec 14, 2020

auvipy commented Dec 14, 2020

auvipy commented Dec 14, 2020

tpickle-py commented Aug 17, 2021

kwist-sgr commented Aug 23, 2021

maxmalysh commented May 18, 2023

gpmarques commented Nov 16, 2023

auvipy commented Nov 19, 2023

Andrew-Chen-Wang commented Jun 30, 2020 •

edited by sync-by-unito bot

Andrew-Chen-Wang commented Jun 30, 2020 •

edited

Andrew-Chen-Wang commented Jun 30, 2020 •

edited

Andrew-Chen-Wang commented Jun 30, 2020 •

edited

raiderrobert commented Jul 29, 2020 •

edited

Andrew-Chen-Wang commented Sep 11, 2020 •

edited