Unicode error when running ./setup.sh run-scan-worker

[MM294]

issue is unneeded unicode conversion in
env/lib/python2.7/site-packages/celery-3.0.19-py2.7.egg/celery/apps/worker.py
causes
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 93: ordinal not in range(128)

removing unicode brackets from line 230 fixes

[st3fan]

Line 230 of which file?

[MM294]

env/lib/python2.7/site-packages/celery-3.0.19-py2.7.egg/celery/apps/worker.py is the file

there seem to be further issues with the copy of celery brought down by setup.sh too, as even after fixing this bug minion will not run

./setup.sh run-plugin-worker

now errors out as

Usage: celery [options]
celery: error: no such option: --loglevel

attempting to run celery directly from bin with

./celery worker -A minion.backend.tasks --loglevel=INFO --concurrency=16 -Q scan -n scan

results in the same result, indicating that the arguments arent formed correctly, i dont seem to be able to reform them to work however.

[MM294]

minion-backend/scripts/minion-scan-worker is where the celery command came from

[st3fan]

We cannot make changes in dependencies.

I think this has to do with your environment or the site that you are scanning. Can you tell me if you use a non-english system or if you are scanning a site that possibly has unicode in the url or content?

[MM294]

im on an english system running fedora 19, i havent tryed to scan a site yet as the 3 worker scripts all error out in the same way

./setup.sh run-scan-worker
./setup.sh run-state-worker
./setup.sh run-plugin-worker

it appears to be a problem in how the minion backend is calling celery

[st3fan]

Can you post the full stack trace that you get?

[MM294]

the full output of the original unicode problem is

./setup.sh run-scan-worker
Traceback (most recent call last):
  File "/home/H55-UD3H/michael.morley/Documents/minion/env/bin/celery", line 8, in <module>
    load_entry_point('celery==3.0.19', 'console_scripts', 'celery')()
  File "/home/H55-UD3H/michael.morley/Documents/minion/env/lib/python2.7/site-packages/celery-3.0.19-py2.7.egg/celery/__main__.py", line 14, in main
    main()
  File "/home/H55-UD3H/michael.morley/Documents/minion/env/lib/python2.7/site-packages/celery-3.0.19-py2.7.egg/celery/bin/celery.py", line 957, in main
    cmd.execute_from_commandline(argv)
  File "/home/H55-UD3H/michael.morley/Documents/minion/env/lib/python2.7/site-packages/celery-3.0.19-py2.7.egg/celery/bin/celery.py", line 901, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/home/H55-UD3H/michael.morley/Documents/minion/env/lib/python2.7/site-packages/celery-3.0.19-py2.7.egg/celery/bin/base.py", line 187, in execute_from_commandline
    return self.handle_argv(prog_name, argv[1:])
  File "/home/H55-UD3H/michael.morley/Documents/minion/env/lib/python2.7/site-packages/celery-3.0.19-py2.7.egg/celery/bin/celery.py", line 893, in handle_argv
    return self.execute(command, argv)
  File "/home/H55-UD3H/michael.morley/Documents/minion/env/lib/python2.7/site-packages/celery-3.0.19-py2.7.egg/celery/bin/celery.py", line 868, in execute
    return cls(app=self.app).run_from_argv(self.prog_name, argv)
  File "/home/H55-UD3H/michael.morley/Documents/minion/env/lib/python2.7/site-packages/celery-3.0.19-py2.7.egg/celery/bin/celery.py", line 148, in run_from_argv
    return self(*args, **options)
  File "/home/H55-UD3H/michael.morley/Documents/minion/env/lib/python2.7/site-packages/celery-3.0.19-py2.7.egg/celery/bin/celery.py", line 118, in __call__
    ret = self.run(*args, **kwargs)
  File "/home/H55-UD3H/michael.morley/Documents/minion/env/lib/python2.7/site-packages/celery-3.0.19-py2.7.egg/celery/bin/celery.py", line 220, in run
    return self.target.run(*args, **kwargs)
  File "/home/H55-UD3H/michael.morley/Documents/minion/env/lib/python2.7/site-packages/celery-3.0.19-py2.7.egg/celery/bin/celeryd.py", line 153, in run
    return self.app.Worker(**kwargs).run()
  File "/home/H55-UD3H/michael.morley/Documents/minion/env/lib/python2.7/site-packages/celery-3.0.19-py2.7.egg/celery/apps/worker.py", line 178, in run
    print(str(self.colored.cyan(' \n', self.startup_info())) +
  File "/home/H55-UD3H/michael.morley/Documents/minion/env/lib/python2.7/site-packages/celery-3.0.19-py2.7.egg/celery/apps/worker.py", line 256, in startup_info
    'queues': app.amqp.queues.format(indent=0, indent_first=False),
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 93: ordinal not in range(128)

after fixing the unicode issue as before

$ ./setup.sh run-scan-worker
Usage: celery <command> [options] 
celery: error: no such option: --loglevel

i didnt think you could stacktrace bash/python scripts though?

[st3fan]

I'll install Fedora 19 in a VM here today to find out what is going on. This should just work.

[MM294]

many thanks :) , my coding is very rudimentary, apologies for not being more help

[st3fan]

No worries. We'll sort it out. One more question: do you SSH into that machine or do you run the commands from a Terminal on the machine?

[MM294]

running it on localhosts terminal at the moment

[st3fan]

The problem is that /etc/system-release on Fedora 19 contains unicode:

Fedora release 19 (Schrödinger’s Cat)

It seems Celery reads that file when it prints a banner when a queue worker starts and then it trips over the Fedora release name.

We cannot fix this in Minion so the best way to proceed is to edit /etc/system-release and make it all-ASCII.

[MM294]

ah thank you, this solved the problem