Python 3.8 support

[auvipy]

No description provided.

[rajwitt]

When running celery 4.4.0rc3 with python 3.8 and trying to deamonize during a docker startup script we get:

2019-10-24T16:47:22.613375800Z celery multi v4.4.0rc3 (cliffs)
2019-10-24T16:47:22.617521400Z > Starting nodes...
2019-10-24T16:47:24.365411800Z Traceback (most recent call last):
2019-10-24T16:47:24.365445300Z   File "/usr/local/lib/python3.8/runpy.py", line 192, in _run_module_as_main
2019-10-24T16:47:24.365597000Z     return _run_code(code, main_globals, None,
2019-10-24T16:47:24.365609800Z   File "/usr/local/lib/python3.8/runpy.py", line 85, in _run_code
2019-10-24T16:47:24.365614400Z     exec(code, run_globals)
2019-10-24T16:47:24.365618300Z   File "/usr/local/lib/python3.8/site-packages/celery/__main__.py", line 20, in <module>
2019-10-24T16:47:24.365623000Z     main()
2019-10-24T16:47:24.365626800Z   File "/usr/local/lib/python3.8/site-packages/celery/__main__.py", line 16, in main
2019-10-24T16:47:24.365630700Z     _main()
2019-10-24T16:47:24.365634400Z   File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 322, in main
2019-10-24T16:47:24.366199600Z     cmd.execute_from_commandline(argv)
2019-10-24T16:47:24.366218500Z   File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline
2019-10-24T16:47:24.366238900Z     super(CeleryCommand, self).execute_from_commandline(argv)))
2019-10-24T16:47:24.366243800Z   File "/usr/local/lib/python3.8/site-packages/celery/bin/base.py", line 305, in execute_from_commandline
2019-10-24T16:47:24.366312000Z     return self.handle_argv(self.prog_name, argv[1:])
2019-10-24T16:47:24.366319800Z   File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 488, in handle_argv
2019-10-24T16:47:24.366385800Z     return self.execute(command, argv)
2019-10-24T16:47:24.366392200Z   File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 416, in execute
2019-10-24T16:47:24.368839600Z     return cls(
2019-10-24T16:47:24.368859300Z   File "/usr/local/lib/python3.8/site-packages/celery/bin/worker.py", line 221, in run_from_argv
2019-10-24T16:47:24.368933200Z     *self.parse_options(prog_name, argv, command))
2019-10-24T16:47:24.368940900Z   File "/usr/local/lib/python3.8/site-packages/celery/bin/base.py", line 428, in parse_options
2019-10-24T16:47:24.369117400Z     self.parser = self.create_parser(prog_name, command)
2019-10-24T16:47:24.369125500Z   File "/usr/local/lib/python3.8/site-packages/celery/bin/base.py", line 440, in create_parser
2019-10-24T16:47:24.369284100Z     description=self._format_description(self.description),
2019-10-24T16:47:24.369297400Z   File "/usr/local/lib/python3.8/site-packages/celery/bin/base.py", line 462, in _format_description
2019-10-24T16:47:24.369420400Z     text.fill_paragraphs(text.dedent(description), width))
2019-10-24T16:47:24.369426800Z   File "/usr/local/lib/python3.8/site-packages/celery/utils/text.py", line 58, in fill_paragraphs
2019-10-24T16:47:24.369491100Z     return sep.join(fill(p, width) for p in s.split(sep))
2019-10-24T16:47:24.369505200Z   File "/usr/local/lib/python3.8/site-packages/celery/utils/text.py", line 58, in <genexpr>
2019-10-24T16:47:24.369561300Z     return sep.join(fill(p, width) for p in s.split(sep))
2019-10-24T16:47:24.369567500Z   File "/usr/local/lib/python3.8/textwrap.py", line 391, in fill
2019-10-24T16:47:24.369643800Z     return w.fill(text)
2019-10-24T16:47:24.369650100Z   File "/usr/local/lib/python3.8/textwrap.py", line 363, in fill
2019-10-24T16:47:24.369732400Z     return "\n".join(self.wrap(text))
2019-10-24T16:47:24.369738700Z   File "/usr/local/lib/python3.8/textwrap.py", line 354, in wrap
2019-10-24T16:47:24.369820100Z     return self._wrap_chunks(chunks)
2019-10-24T16:47:24.369826300Z   File "/usr/local/lib/python3.8/textwrap.py", line 248, in _wrap_chunks
2019-10-24T16:47:24.369903100Z     raise ValueError("invalid width %r (must be > 0)" % self.width)
2019-10-24T16:47:24.369956900Z ValueError: invalid width -2 (must be > 0)
2019-10-24T16:47:24.562519000Z  > celery@ac1135a07cab: * Child terminated with exit code 1
2019-10-24T16:47:24.564876400Z FAILED
2019-10-24T16:47:24.679649500Z celery init v10.1.
2019-10-24T16:47:24.813327900Z Using configuration: /etc/default/celeryd
2019-10-24T16:47:24.830932800Z Starting celerybeat...
2019-10-24T16:47:27.307874200Z Traceback (most recent call last):
2019-10-24T16:47:27.309529300Z   File "/usr/local/bin/celery", line 8, in <module>
2019-10-24T16:47:27.312288700Z     sys.exit(main())
2019-10-24T16:47:27.314089900Z   File "/usr/local/lib/python3.8/site-packages/celery/__main__.py", line 16, in main
2019-10-24T16:47:27.314430800Z     _main()
2019-10-24T16:47:27.314558400Z   File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 322, in main
2019-10-24T16:47:27.314883800Z     cmd.execute_from_commandline(argv)
2019-10-24T16:47:27.315091300Z   File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline
2019-10-24T16:47:27.315681600Z     super(CeleryCommand, self).execute_from_commandline(argv)))
2019-10-24T16:47:27.315801200Z   File "/usr/local/lib/python3.8/site-packages/celery/bin/base.py", line 305, in execute_from_commandline
2019-10-24T16:47:27.316227500Z     return self.handle_argv(self.prog_name, argv[1:])
2019-10-24T16:47:27.316340600Z   File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 488, in handle_argv
2019-10-24T16:47:27.319986500Z     return self.execute(command, argv)
2019-10-24T16:47:27.320399700Z   File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 416, in execute
2019-10-24T16:47:27.320838000Z     return cls(
2019-10-24T16:47:27.321093200Z   File "/usr/local/lib/python3.8/site-packages/celery/bin/base.py", line 308, in run_from_argv
2019-10-24T16:47:27.321484200Z     return self.handle_argv(prog_name,
2019-10-24T16:47:27.321992200Z   File "/usr/local/lib/python3.8/site-packages/celery/bin/base.py", line 392, in handle_argv
2019-10-24T16:47:27.324326500Z     *self.parse_options(prog_name, argv, command))
2019-10-24T16:47:27.324560200Z   File "/usr/local/lib/python3.8/site-packages/celery/bin/base.py", line 428, in parse_options
2019-10-24T16:47:27.325119400Z     self.parser = self.create_parser(prog_name, command)
2019-10-24T16:47:27.325502300Z   File "/usr/local/lib/python3.8/site-packages/celery/bin/base.py", line 440, in create_parser
2019-10-24T16:47:27.326254300Z     description=self._format_description(self.description),
2019-10-24T16:47:27.326431700Z   File "/usr/local/lib/python3.8/site-packages/celery/bin/base.py", line 462, in _format_description
2019-10-24T16:47:27.328868100Z     text.fill_paragraphs(text.dedent(description), width))
2019-10-24T16:47:27.329160900Z   File "/usr/local/lib/python3.8/site-packages/celery/utils/text.py", line 58, in fill_paragraphs
2019-10-24T16:47:27.331312600Z     return sep.join(fill(p, width) for p in s.split(sep))
2019-10-24T16:47:27.331597300Z   File "/usr/local/lib/python3.8/site-packages/celery/utils/text.py", line 58, in <genexpr>
2019-10-24T16:47:27.332248000Z     return sep.join(fill(p, width) for p in s.split(sep))
2019-10-24T16:47:27.332645000Z   File "/usr/local/lib/python3.8/textwrap.py", line 391, in fill
2019-10-24T16:47:27.333713700Z     return w.fill(text)
2019-10-24T16:47:27.334444800Z   File "/usr/local/lib/python3.8/textwrap.py", line 363, in fill
2019-10-24T16:47:27.336448400Z     return "\n".join(self.wrap(text))
2019-10-24T16:47:27.336668800Z   File "/usr/local/lib/python3.8/textwrap.py", line 354, in wrap
2019-10-24T16:47:27.338252700Z     return self._wrap_chunks(chunks)
2019-10-24T16:47:27.338467300Z   File "/usr/local/lib/python3.8/textwrap.py", line 248, in _wrap_chunks
2019-10-24T16:47:27.340207400Z     raise ValueError("invalid width %r (must be > 0)" % self.width)

However once you remote into the docker and start it manually it works fine.

[thedrow]

@rajwitt Can you please provide us a test case to reproduce this problem?

[rajwitt]

so I actually solved this - just to let you know:

If you run a docker wiht FROM python:3.8-buster

and then you want to start the celeryd or celerybeat via a bash script in the start up process
then the terminal has no width and will crash with the above - i got this to work by exporting a columns variable:

export COLUMNS=80

but maybe make the python processes have a default of 80 if it cannot read the terminal width for output formatting?

[dalberto]

I tracked down the Docker / Python 3.8 issue:

• Celery 4.3.x (the version I'm using, though this seems to apply to later versions too) uses argparse to determine the terminal width.
• In Python 3.8, argparse was refactored to use shutil to determine the terminal width. Previously it used the COLUMNS environment variable directly and defaulted to 80 when it wasn't set or didn't exist.
• shutil.get_terminal_size defaults to 80 for columns but it seems like that default is not being used.

I'm unsure why shutil.get_terminal_size returns 0 for columns in this environment however.

[georgepsarakis]

@dalberto @rajwitt thanks for sharing the details of the root cause. I found this StackOverflow post that might be relevant. As far as I understand, the only reliable way to resolve this is by using the environment variable.

[rcjsuen]

We are being hit by this also. There are comments indicating that this may be an issue with Docker (although there are similarly also reports of people saying it is still an issue). Unfortunately, we are using an ancient version of Docker here (docker://17.3.2 reported by Kubernetes) so I can't really confirm/deny on that front. :(

[adongy]

If you are getting this error, it may be because you're allocating a pseudo-tty from docker.

Here are some commands:

No tty:

$ docker run --rm python:3.8-buster python -c 'print(__import__("shutil").get_terminal_size().columns)'
80

With tty:

$ docker run -t --rm python:3.8-buster python -c 'print(__import__("shutil").get_terminal_size().columns)'
0

In case you do need a tty, setting COLUMNS does solve the issue by shortcutting get_terminal_size.

The tty size being 0 0 is most likely related to docker/cli#1529 docker/for-linux#314

[forgoty]

Have the same issue on celery 4.4.2, docker image - python:3.8.1-slim. Docker - 19.03.8. Docker-compose - 1.25.5. Launched on Arch linux 5.6.11 kernel.

Setting this solves the problem

environment:
  - COLUMNS=80