Cannot run ddtrace-run celery worker attempts to execute celery with python

[jcummins54]

ddtrace-run celery runs the celery application, but if anything is added after "celery" ddtrace-run attempts to run it with python:

$ ddtrace-run celery worker --queues default --loglevel=INFO --concurrency=4 --maxtasksperchild=100 -n default.%%n

/home/vagrant/venv/bin/python2.7 /home/vagrant/venv/bin/celery worker --queues default --loglevel=INFO --concurrency=4 --maxtasksperchild=100 -n default.%%n: can't open file '/home/vagrant/venv/bin/celery worker --queues default --loglevel=INFO --concurrency=4 --maxtasksperchild=100 -n default.%%n': [Errno 2] No such file or directory

Update: This is only an issue with ddtrace 0.12.1, verified that ddtrace 0.12.0 runs celery. However, we aren't seeing APM metrics in the dashboard either way.

[palazzem]

Hello @jcummins54 and thanks for the feedback! I'm checking this issue with the ddtrace-run script that requires more work in this 0.12.x release. Once I've reproduced the problem, I'll provide you a PR to test so that we're sure the script properly wraps any kind of binary.

[mgu]

Same error for me when lauching gunicorn.
If I revert https://github.com/DataDog/dd-trace-py/pull/469/files the error disappears.

Looks like sys.argv is not filled for a good reason https://bugs.python.org/issue2972

[mgu]

After some tests, the error with gunicorn appears with ddtrace 0.12.1 AND setproctitle installed.
If I uninstall setproctitle then I can launch gunicorn without error

[palazzem]

Thanks @mgu for the investigation! We're preparing an alternative to make it work when celery is imported while sys.argv is not filled. We'll try to fix the problem with ddtrace-run as soon as possible.

Thank you!

[palazzem]

Sorry for the delay, but we're checking if the new way we're instrumenting Celery address properly this issue, so that the included patch can be reverted to avoid issues (as explained in https://bugs.python.org/issue2972). This is the last part we're missing, but we'll be sure to find the solution quickly and ship the new release next week.

[mgu]

I can test your fix before the release if it can help

[Kyle-Verhoog]

Hi @jcummins54, @mgu

I've been looking into this issue, can you provide some more detail about your setups? A regression test or sample app would be of great help!

[mgu]

I will try to setup a test or a small app in order to reproduce this issue, I'll keep you updated

[mgu]

Harder to reproduce than expected (on a dedicated app), I'll try something else tomorrow :)

[Kyle-Verhoog]

Hey @mgu thanks so much for the effort! Haha, I had absolutely no success replicating it either.

Don't worry about trying to replicate further. We've changed our patching mechanism for celery with #534. It is also based off of #530 which should include our improved celery integration.

If you could try using the kyle-verhoog/celery-hook branch and report back if you have any problems that'd be awesome!

Thanks 😄

[mgu]

Tested (with a fix to avoid undefined patched variable in monkey.py) :

• [Errno 2] No such file or directory is gone
• it seems to work without patch(celery=True)

Thanks a lot ! 😁

[palazzem]

Great @mgu ! ok we'll connect all the dots to prepare our release. We had also other positive feedbacks so we can proceed. I'll keep the thread updated.

Thank you very much, especially for your patience!