Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rfid_post causing 500 errors #50

Open
ssfrr opened this issue Feb 13, 2015 · 0 comments
Open

rfid_post causing 500 errors #50

ssfrr opened this issue Feb 13, 2015 · 0 comments

Comments

@ssfrr
Copy link
Member

ssfrr commented Feb 13, 2015

This is a weird one.

There were 4 errors thrown at 6:37AM yesterday. Seems like the first error is the trigger that caused the next 3. This link seems to imply that this error ("terminating connection due to administrator command") can happen if the process gets killed, possibly because it's exceeding it's ulimit and opening too many files. I'm wondering if this is also related to the issue with websocketd crapping out with "too many open files"

This is the output from the medialab_rfid_post script:

2015-02-12 06:37:25 chain-api __main__[20494] INFO Updating considerate
Traceback (most recent call last):
  File "/usr/local/bin/rfidpost", line 7, in <module>
    execfile(__file__)
  File "/home/sfr/chain-api/collectors/rfidpost", line 254, in <module>
    main()
  File "/home/sfr/chain-api/collectors/rfidpost", line 82, in main
    update_rfid_sensor(site, poller, sensor)
  File "/home/sfr/chain-api/collectors/rfidpost", line 145, in update_rfid_sensor
    sanitized_tags)))
  File "/home/sfr/chain-api/collectors/rfidpost", line 144, in <lambda>
    map(lambda tag: get_person_link_from_tags(tag, site),
  File "/home/sfr/chain-api/collectors/rfidpost", line 184, in get_person_link_from_tags
    person = get_person_from_tags(tags, site)
  File "/home/sfr/chain-api/collectors/rfidpost", line 194, in get_person_from_tags
    site['_links']['ch:people']['href'])
  File "/home/sfr/chain-api/collectors/rfidpost", line 223, in get_person_by_rfid
    list_response = chainclient.get(people_url_filtered, auth=COLLECTOR_AUTH)
  File "/home/sfr/chainclient.py/chainclient.py", line 47, in get
    response = _request_with_error('GET', href, auth=auth).json()
  File "/home/sfr/chainclient.py/chainclient.py", line 39, in _request_with_error
    raise ChainException(response.content)
chainclient.ChainException: {"status": 500, "message": "I'm sorry, you broke the server. Those responsible have been sacked."}
Traceback (most recent call last):
  File "/usr/local/bin/rfidpost", line 7, in <module>
    execfile(__file__)
  File "/home/sfr/chain-api/collectors/rfidpost", line 254, in <module>
    main()
  File "/home/sfr/chain-api/collectors/rfidpost", line 76, in main
    site = chainclient.get(site_url, auth=COLLECTOR_AUTH)
  File "/home/sfr/chainclient.py/chainclient.py", line 47, in get
    response = _request_with_error('GET', href, auth=auth).json()
  File "/home/sfr/chainclient.py/chainclient.py", line 39, in _request_with_error
    raise ChainException(response.content)
chainclient.ChainException: <html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1>Internal Server Error</h1>

  </body>
</html>

Traceback (most recent call last):
  File "/usr/local/bin/rfidpost", line 7, in <module>
    execfile(__file__)
  File "/home/sfr/chain-api/collectors/rfidpost", line 254, in <module>
    main()
  File "/home/sfr/chain-api/collectors/rfidpost", line 76, in main
    site = chainclient.get(site_url, auth=COLLECTOR_AUTH)
  File "/home/sfr/chainclient.py/chainclient.py", line 47, in get
    response = _request_with_error('GET', href, auth=auth).json()
  File "/home/sfr/chainclient.py/chainclient.py", line 39, in _request_with_error
    raise ChainException(response.content)
chainclient.ChainException: <html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1>Internal Server Error</h1>

  </body>
</html>

Traceback (most recent call last):
  File "/usr/local/bin/rfidpost", line 7, in <module>
    execfile(__file__)
  File "/home/sfr/chain-api/collectors/rfidpost", line 254, in <module>
    main()
  File "/home/sfr/chain-api/collectors/rfidpost", line 76, in main
    site = chainclient.get(site_url, auth=COLLECTOR_AUTH)
  File "/home/sfr/chainclient.py/chainclient.py", line 47, in get
    response = _request_with_error('GET', href, auth=auth).json()
  File "/home/sfr/chainclient.py/chainclient.py", line 39, in _request_with_error
    raise ChainException(response.content)
chainclient.ChainException: <html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1>Internal Server Error</h1>

  </body>
</html>

Traceback (most recent call last):
  File "/usr/local/bin/rfidpost", line 7, in <module>
    execfile(__file__)
  File "/home/sfr/chain-api/collectors/rfidpost", line 254, in <module>
    main()
  File "/home/sfr/chain-api/collectors/rfidpost", line 76, in main
    site = chainclient.get(site_url, auth=COLLECTOR_AUTH)
  File "/home/sfr/chainclient.py/chainclient.py", line 47, in get
    response = _request_with_error('GET', href, auth=auth).json()
  File "/home/sfr/chainclient.py/chainclient.py", line 39, in _request_with_error
    raise ChainException(response.content)
chainclient.ChainException: <html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1>Internal Server Error</h1>

  </body>
</html>

Traceback (most recent call last):
  File "/usr/local/bin/rfidpost", line 7, in <module>
    execfile(__file__)
  File "/home/sfr/chain-api/collectors/rfidpost", line 254, in <module>
    main()
  File "/home/sfr/chain-api/collectors/rfidpost", line 76, in main
    site = chainclient.get(site_url, auth=COLLECTOR_AUTH)
  File "/home/sfr/chainclient.py/chainclient.py", line 47, in get
    response = _request_with_error('GET', href, auth=auth).json()
  File "/home/sfr/chainclient.py/chainclient.py", line 39, in _request_with_error
    raise ChainException(response.content)
chainclient.ChainException: <html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1>Internal Server Error</h1>

  </body>
</html>

And these are the errors from the server log:

Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 114, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)

  File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/csrf.py", line 57, in wrapped_view
    return view_func(*args, **kwargs)

  File "/home/sfr/chain-api/chain/core/api.py", line 621, in list_view
    limit=limit).serialize()

  File "/home/sfr/chain-api/chain/core/api.py", line 420, in serialize
    *args, **kwargs)

  File "/home/sfr/chain-api/chain/core/api.py", line 396, in serialize_list
    'totalCount': self.get_total_count()

  File "/home/sfr/chain-api/chain/core/api.py", line 262, in get_total_count
    self._total_count = qs.count()

  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 291, in count
    return self.query.get_count(using=self.db)

  File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 390, in get_count
    number = obj.get_aggregation(using=using)[None]

  File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 356, in get_aggregation
    result = query.get_compiler(using).execute_sql(SINGLE)

  File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 782, in execute_sql
    cursor.execute(sql, params)

  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)

  File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)

  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)

OperationalError: terminating connection due to administrator command
SSL connection has been closed unexpectedly



<WSGIRequest
path:/people/,
GET:<QueryDict: {u'rfid': [u'4D656469614C616243017174'], u'site_id': [u'5']}>,
POST:<QueryDict: {}>,
COOKIES:{},
META:{u'CSRF_COOKIE': u'dRsgp8LPYLRF9yKMBa8UcUGlWd1vjGuz',
 'HTTP_ACCEPT': '*/*',
 'HTTP_ACCEPT_ENCODING': 'gzip, deflate',
 'HTTP_AUTHORIZATION': 'Basic Y2hhaW5jb2xsZWN0b3JzOlY4VGlPRTZ5VndwaklpbTEzeWJoakE5Tw==',
 'HTTP_CONNECTION': 'close',
 'HTTP_HOST': 'chain-api.media.mit.edu',
 'HTTP_USER_AGENT': 'python-requests/2.3.0 CPython/2.7.3 Linux/3.11.0-26-generic',
 'HTTP_X_FORWARDED_FOR': '18.85.58.252',
 'PATH_INFO': u'/people/',
 'QUERY_STRING': 'site_id=5&rfid=4D656469614C616243017174',
 'RAW_URI': '/people/?site_id=5&rfid=4D656469614C616243017174',
 'REMOTE_ADDR': '18.85.58.252',
 'REMOTE_PORT': '80',
 'REQUEST_METHOD': 'GET',
 'SCRIPT_NAME': u'',
 'SERVER_NAME': 'chain-api.media.mit.edu',
 'SERVER_PORT': '80',
 'SERVER_PROTOCOL': 'HTTP/1.0',
 'SERVER_SOFTWARE': 'gunicorn/18.0',
 'gunicorn.socket': <socket._socketobject object at 0x3f82050>,
 'wsgi.errors': <open file '<stderr>', mode 'w' at 0x7f7535202270>,
 'wsgi.file_wrapper': <class gunicorn.http.wsgi.FileWrapper at 0x2cf4668>,
 'wsgi.input': <gunicorn.http.body.Body object at 0x3eba810>,
 'wsgi.multiprocess': True,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 0)}>

Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 114, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)

  File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/csrf.py", line 57, in wrapped_view
    return view_func(*args, **kwargs)

  File "/home/sfr/chain-api/chain/core/api.py", line 744, in create_view
    return cls.create_single(data, request)

  File "/home/sfr/chain-api/chain/core/api.py", line 751, in create_single
    new_resource.save()

  File "/home/sfr/chain-api/chain/core/api.py", line 544, in save
    self._obj.save()

  File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 545, in save
    force_update=force_update, update_fields=update_fields)

  File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 573, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)

  File "/usr/local/lib/python2.7/dist-packages/django/db/transaction.py", line 330, in __exit__
    connection.set_autocommit(True)

  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 333, in set_autocommit
    self._set_autocommit(autocommit)

  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 173, in _set_autocommit
    self.connection.autocommit = autocommit

InterfaceError: connection already closed


<WSGIRequest
path:/sensordata/create,
GET:<QueryDict: {u'sensor_id': [u'1069']}>,
POST:<QueryDict: {}>,
COOKIES:{},
META:{'CONTENT_LENGTH': '59',
 u'CSRF_COOKIE': u'k32g9IdorvZMWxuigBY6sDJuoCSqLwDz',
 'HTTP_ACCEPT': '*/*',
 'HTTP_ACCEPT_ENCODING': 'gzip, deflate',
 'HTTP_AUTHORIZATION': 'Basic Y2hhaW5jb2xsZWN0b3JzOlY4VGlPRTZ5VndwaklpbTEzeWJoakE5Tw==',
 'HTTP_CONNECTION': 'close',
 'HTTP_HOST': 'chain-api.media.mit.edu',
 'HTTP_USER_AGENT': 'python-requests/2.3.0 CPython/2.7.3 Linux/3.2.0-4-amd64',
 'HTTP_X_FORWARDED_FOR': '18.85.58.245',
 'PATH_INFO': u'/sensordata/create',
 'QUERY_STRING': 'sensor_id=1069',
 'RAW_URI': '/sensordata/create?sensor_id=1069',
 'REMOTE_ADDR': '18.85.58.245',
 'REMOTE_PORT': '80',
 'REQUEST_METHOD': 'POST',
 'SCRIPT_NAME': u'',
 'SERVER_NAME': 'chain-api.media.mit.edu',
 'SERVER_PORT': '80',
 'SERVER_PROTOCOL': 'HTTP/1.0',
 'SERVER_SOFTWARE': 'gunicorn/18.0',
 'gunicorn.socket': <socket._socketobject object at 0x3e93de0>,
 'wsgi.errors': <open file '<stderr>', mode 'w' at 0x7f7535202270>,
 'wsgi.file_wrapper': <class gunicorn.http.wsgi.FileWrapper at 0x2cf4668>,
 'wsgi.input': <gunicorn.http.body.Body object at 0x3ebbe50>,
 'wsgi.multiprocess': True,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 0)}>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ssfrr