Make all datasource/feature/dependencies-generally and their value JSONable

[halfak]

Right now, many datasources return values that cannot be encoded in JSON.

This is a bummer because it would be better if we could use the JSON serializer within ORES's celery.

This is the error we get when trying to use the JSON serializer within ORES for non-JSON serializable datasources:

3784623 HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/halfak/projects/ores/ores/wsgi/routes/scores.py", line 102, in score_revisions
    precache=precache)
  File "/home/halfak/projects/ores/ores/score_processors/score_processor.py", line 25, in score
    scores = self._score(context, model, rev_ids, caches=caches)
  File "/home/halfak/projects/ores/ores/score_processors/celery.py", line 146, in _score
    caches=caches))
  File "/home/halfak/projects/ores/ores/score_processors/celery.py", line 97, in _score_in_celery
    task_id=id_string
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/celery/app/task.py", line 559, in apply_async
    **dict(self._get_exec_options(), **options)
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/celery/app/base.py", line 353, in send_task
    reply_to=reply_to or self.oid, **options
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/celery/app/amqp.py", line 305, in publish_task
    **kwargs
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/kombu/messaging.py", line 165, in publish
    compression, headers)
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/kombu/messaging.py", line 241, in _prepare
    body) = dumps(body, serializer=serializer)
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/kombu/serialization.py", line 164, in dumps
    payload = encoder(data)
  File "/usr/lib/python3.4/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/kombu/serialization.py", line 59, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/kombu/five.py", line 132, in reraise
    raise value.with_traceback(tb)
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/kombu/serialization.py", line 55, in _reraise_errors
    yield
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/kombu/serialization.py", line 164, in dumps
    payload = encoder(data)
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/anyjson/__init__.py", line 141, in dumps
    return implementation.dumps(value)
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/anyjson/__init__.py", line 89, in dumps
    raise TypeError(TypeError(*exc.args)).with_traceback(sys.exc_info()[2])
  File "/home/halfak/env/3.4/lib/python3.4/site-packages/anyjson/__init__.py", line 87, in dumps
    return self._encode(data)
  File "/usr/lib/python3.4/json/__init__.py", line 230, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.4/json/encoder.py", line 192, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.4/json/encoder.py", line 250, in iterencode
    return _iterencode(o, 0)
kombu.exceptions.EncodeError: keys must be a string

[halfak]

We'll also need to make dependencies themselves trivially serializable because they are keys in the cache map.

[Ladsgroup]

Hey, Can you give me exact steps to reproduce?

[halfak]

Looks like this just isn't going to be possible. We're going to want complex datatypes that allow for non-json-ability.