Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Redis exceptions on processing worker command #1937

Closed
sonstephendo opened this issue Oct 10, 2019 · 6 comments
Closed

Redis exceptions on processing worker command #1937

sonstephendo opened this issue Oct 10, 2019 · 6 comments

Comments

@sonstephendo
Copy link

Hi, I got problem when try install the server - python3-version. (i have not tried version 2 yet)
I have already setup the redis server and sentinal by following the guide:

redis-server contrib/sentinel.conf --sentinel
Ouput

tcp        0      0 0.0.0.0:6379            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:26379           0.0.0.0:*               LISTEN      9810/redis-server *

and rqscheduler:

rqscheduler --host 127.0.0.1 --port 6379 --interval 60 --db 0
01:42:30 Registering birth

But got the exception with the worker when try:
python app_context_rqworker.py scheduled_jobs super high medium low email maintenance

So please help me to figure out the problem

01:43:33 super: Job OK (5b9fba89-c289-49f4-8a03-89c790f69153)
01:43:33 Result is kept for 500 seconds
01:43:33 super: pybossa.leaderboard.jobs.leaderboard() (237d6c21-f427-418e-a275-f4aca2437d28)
01:43:33 super: Job OK (237d6c21-f427-418e-a275-f4aca2437d28)
01:43:33 Result is kept for 500 seconds
01:43:33 maintenance: pybossa.jobs.check_failed() (07cccfb4-d60e-4498-bac0-da46ddf42f02)
01:43:33 maintenance: Job OK (07cccfb4-d60e-4498-bac0-da46ddf42f02)
01:43:33 Result is kept for 500 seconds
01:43:33 maintenance: pybossa.jobs.check_failed() (4db2f275-66a2-4d8f-8ecf-c9b7e3f5cfb7)
01:43:33 maintenance: Job OK (4db2f275-66a2-4d8f-8ecf-c9b7e3f5cfb7)
01:43:33 Result is kept for 500 seconds
Traceback (most recent call last):
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/redis/connection.py", line 181, in _read_from_socket
    data = recv(self._sock, socket_read_size)
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/redis/_compat.py", line 71, in recv
    return sock.recv(*args, **kwargs)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/redis/client.py", line 755, in execute_command
    return self.parse_response(connection, command_name, **options)
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/redis/client.py", line 768, in parse_response
    response = connection.read_response()
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/redis/sentinel.py", line 56, in read_response
    return super(SentinelManagedConnection, self).read_response()
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/redis/connection.py", line 636, in read_response
    raise e
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/redis/connection.py", line 633, in read_response
    response = self._parser.read_response()
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/redis/connection.py", line 291, in read_response
    response = self._buffer.readline()
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/redis/connection.py", line 223, in readline
    self._read_from_socket()
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/redis/connection.py", line 194, in _read_from_socket
    raise TimeoutError("Timeout reading from socket")
redis.exceptions.TimeoutError: Timeout reading from socket

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/redis/connection.py", line 181, in _read_from_socket
    data = recv(self._sock, socket_read_size)
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/redis/_compat.py", line 71, in recv
    return sock.recv(*args, **kwargs)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "app_context_rqworker.py", line 34, in <module>
    w.work()
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/rq/worker.py", line 487, in work
    result = self.dequeue_job_and_maintain_ttl(timeout)
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/rq/worker.py", line 520, in dequeue_job_and_maintain_ttl
    job_class=self.job_class)
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/rq/queue.py", line 470, in dequeue_any
    result = cls.lpop(queue_keys, timeout, connection=connection)
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/rq/queue.py", line 420, in lpop
    result = connection.blpop(queue_keys, timeout)
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/redis/client.py", line 1493, in blpop
    return self.execute_command('BLPOP', *keys)
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/redis/client.py", line 761, in execute_command
    return self.parse_response(connection, command_name, **options)
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/redis/client.py", line 768, in parse_response
    response = connection.read_response()
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/redis/sentinel.py", line 56, in read_response
    return super(SentinelManagedConnection, self).read_response()
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/redis/connection.py", line 636, in read_response
    raise e
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/redis/connection.py", line 633, in read_response
    response = self._parser.read_response()
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/redis/connection.py", line 291, in read_response
    response = self._buffer.readline()
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/redis/connection.py", line 223, in readline
    self._read_from_socket()
  File "/home/sondh/.pyenv/versions/3.6.7/lib/python3.6/site-packages/redis/connection.py", line 194, in _read_from_socket
    raise TimeoutError("Timeout reading from socket")
redis.exceptions.TimeoutError: Timeout reading from socket
@sonstephendo sonstephendo changed the title Redis exceptions on process worker command Redis exceptions on processing worker command Oct 10, 2019
@teleyinex
Copy link
Member

I don't know what could be causing this problem. In principle, it should work without problems, because for the tests we run sentinel as well.

Can you check that you can connect to sentinel with the redis client?

@cravindra
Copy link
Contributor

cravindra commented Jan 8, 2020 •
edited

@sonstephendo Did you solve this issue?

@teleyinex I tried running this using the python2.7 branch and it seemed to work fine. I'm seeing this only on the master (previously migrate-python-3) branch.

Since it's working with the python2.7 version, and the file app_context_rqworker.py is identical to those in the python3 version, is it something to do with the dependencies?

I assume redis is working correctly because the 2.7 version works with the same instance.
Redis is also reachable using:
redis-cli -h 127.0.0.1 -p 6379 PING
and
redis-cli -h 127.0.0.1 -p 26379 PING

Detail Output Dump for 3.7:

# python --version
Python 3.7.2
# rqscheduler --host 127.0.0.1
20:02:47 Registering birth
# python --version
Python 3.7.2

# python app_context_rqworker.py scheduled_jobs super high medium low email maintenance

Slave binds are misssing, adding Master as slave too.
/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/flask_sqlalchemy/__init__.py:794: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future.  Set it to True or False to suppress this warning.
  'SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and '
<class 'KeyError'>
('TWITTER_CONSUMER_KEY',)
'TWITTER_CONSUMER_KEY'
Twitter signin disabled
<class 'KeyError'>
('FACEBOOK_APP_ID',)
'FACEBOOK_APP_ID'
Facebook signin disabled
<class 'KeyError'>
('GOOGLE_CLIENT_ID',)
'GOOGLE_CLIENT_ID'
Google signin disabled
<class 'KeyError'>
('FLICKR_API_KEY',)
'FLICKR_API_KEY'
Flickr importer not available
<class 'KeyError'>
('DROPBOX_APP_KEY',)
'DROPBOX_APP_KEY'
Dropbox importer not available
<class 'KeyError'>
('TWITTER_CONSUMER_KEY',)
'TWITTER_CONSUMER_KEY'
Twitter importer not available
<class 'KeyError'>
('YOUTUBE_API_SERVER_KEY',)
'YOUTUBE_API_SERVER_KEY'
Youtube importer not available
20:05:16 RQ worker 'rq:worker:Chirags-MacBook-Pro.60766' started, version 0.13.0
20:05:16 *** Listening on scheduled_jobs, super, high, medium, low, email, maintenance...
20:05:16 Cleaning registries for queue: scheduled_jobs
20:05:16 Cleaning registries for queue: super
20:05:16 Cleaning registries for queue: high
20:05:16 Cleaning registries for queue: medium
20:05:16 Cleaning registries for queue: low
20:05:16 Cleaning registries for queue: email
20:05:16 Cleaning registries for queue: maintenance
Traceback (most recent call last):
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/redis/connection.py", line 181, in _read_from_socket
    data = recv(self._sock, socket_read_size)
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/redis/_compat.py", line 71, in recv
    return sock.recv(*args, **kwargs)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/redis/client.py", line 755, in execute_command
    return self.parse_response(connection, command_name, **options)
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/redis/client.py", line 768, in parse_response
    response = connection.read_response()
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/redis/sentinel.py", line 56, in read_response
    return super(SentinelManagedConnection, self).read_response()
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/redis/connection.py", line 636, in read_response
    raise e
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/redis/connection.py", line 633, in read_response
    response = self._parser.read_response()
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/redis/connection.py", line 291, in read_response
    response = self._buffer.readline()
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/redis/connection.py", line 223, in readline
    self._read_from_socket()
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/redis/connection.py", line 194, in _read_from_socket
    raise TimeoutError("Timeout reading from socket")
redis.exceptions.TimeoutError: Timeout reading from socket

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/redis/connection.py", line 181, in _read_from_socket
    data = recv(self._sock, socket_read_size)
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/redis/_compat.py", line 71, in recv
    return sock.recv(*args, **kwargs)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "app_context_rqworker.py", line 34, in <module>
    w.work()
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/rq/worker.py", line 487, in work
    result = self.dequeue_job_and_maintain_ttl(timeout)
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/rq/worker.py", line 520, in dequeue_job_and_maintain_ttl
    job_class=self.job_class)
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/rq/queue.py", line 470, in dequeue_any
    result = cls.lpop(queue_keys, timeout, connection=connection)
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/rq/queue.py", line 420, in lpop
    result = connection.blpop(queue_keys, timeout)
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/redis/client.py", line 1493, in blpop
    return self.execute_command('BLPOP', *keys)
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/redis/client.py", line 761, in execute_command
    return self.parse_response(connection, command_name, **options)
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/redis/client.py", line 768, in parse_response
    response = connection.read_response()
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/redis/sentinel.py", line 56, in read_response
    return super(SentinelManagedConnection, self).read_response()
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/redis/connection.py", line 636, in read_response
    raise e
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/redis/connection.py", line 633, in read_response
    response = self._parser.read_response()
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/redis/connection.py", line 291, in read_response
    response = self._buffer.readline()
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/redis/connection.py", line 223, in readline
    self._read_from_socket()
  File "/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa/.venv/lib/python3.7/site-packages/redis/connection.py", line 194, in _read_from_socket
    raise TimeoutError("Timeout reading from socket")

Detail Output Dump for 2.7:

# python --version
Python 2.7.16

# rqscheduler --host 127.0.0.1
20:09:50 Running RQ scheduler...
20:09:50 Checking for scheduled jobs...
# python --version
Python 2.7.16

# python app_context_rqworker.py scheduled_jobs super high medium low email maintenance
Slave binds are misssing, adding Master as slave too.
/Users/chiragravindra/Documents/drishti/github/dt-das/pybossa2.7/.venv/lib/python2.7/site-packages/flask_sqlalchemy/__init__.py:794: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future.  Set it to True or False to suppress this warning.
  'SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and '
<type 'exceptions.KeyError'>
('TWITTER_CONSUMER_KEY',)
'TWITTER_CONSUMER_KEY'
Twitter signin disabled
<type 'exceptions.KeyError'>
('FACEBOOK_APP_ID',)
'FACEBOOK_APP_ID'
Facebook signin disabled
<type 'exceptions.KeyError'>
('GOOGLE_CLIENT_ID',)
'GOOGLE_CLIENT_ID'
Google signin disabled
<type 'exceptions.KeyError'>
('FLICKR_API_KEY',)
'FLICKR_API_KEY'
Flickr importer not available
<type 'exceptions.KeyError'>
('DROPBOX_APP_KEY',)
'DROPBOX_APP_KEY'
Dropbox importer not available
<type 'exceptions.KeyError'>
('TWITTER_CONSUMER_KEY',)
'TWITTER_CONSUMER_KEY'
Twitter importer not available
<type 'exceptions.KeyError'>
('YOUTUBE_API_SERVER_KEY',)
'YOUTUBE_API_SERVER_KEY'
Youtube importer not available
20:12:17 RQ worker started, version 0.4.6
20:12:17 
20:12:17 *** Listening on scheduled_jobs, super, high, medium, low, email, maintenance...

@teleyinex
Copy link
Member

Hi,

It looks like there's an issue with the version as you have said @cravindra. Please check this issue.

I've created a new PR (see #1956) with a bump in the redis client version. Can you try it?

@teleyinex teleyinex reopened this Jan 11, 2020
@teleyinex
Copy link
Member

Quick follow up. I've it running now without problems. The issue was the library version plus the default SOCKET_TIMEOUT. By default, it's 0.1 but recent versions of redis-py need to set it to None (also documented in the rq documentation.

Double-check, but this should do the trick!

@cravindra
Copy link
Contributor

cravindra commented Jan 13, 2020 •
edited

@teleyinex Thanks for the response and fix - it's working now 🎉

PS: It may help if the timeout value is fixed in the template file as well https://github.com/Scifabric/pybossa/blob/master/settings_local.py.tmpl for people just following instructions on the docs

PPS: I have created a docker + kubernetes + helm deployment for PyBossa. I'd be happy to share that with the community (I had trouble finding the docker images and making them work with kubernetes myself). I would love to hear any tips you have on where to raise the PR to add those files (if it's not already there somewhere)

@teleyinex
Copy link
Member

@cravindra thanks a lot for the feedback.

Regarding settings local: checkout this PR #1958

Regarding: kubernetes + docker + helm: yes, please, send them :-) It would be awesome to have it here, so feel fre to send those PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@teleyinex @cravindra @sonstephendo