tests: conflict between CliRunner and Celery in memory broker

[nharraud]

When I run the tests I get the exception:

tests/test_utils.py:105:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
invenio_indexer/api.py:195: in process_bulk_queue
    request_timeout=req_timeout,
../../lib/python3.5/site-packages/elasticsearch/helpers/__init__.py:190: in bulk
    for ok, item in streaming_bulk(client, actions, **kwargs):
../../lib/python3.5/site-packages/elasticsearch/helpers/__init__.py:162: in streaming_bulk
    for result in _process_bulk_chunk(client, bulk_actions, raise_on_exception, raise_on_error, **kwargs):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

client = <LocalProxy unbound>, bulk_actions = ['{"delete": {"_index": "records-default-v1.0.0", "_type": "default-v1.0.0", "_id": "7133d6e5-a332-455d-b0c8-973ca420f...ete": {"_index": "records-default-v1.0.0", "_type": "default-v1.0.0", "_id": "7133d6e5-a332-455d-b0c8-973ca420f686"}}'], raise_on_exception = True, raise_on_error = True
kwargs = {'request_timeout': 10}, errors = [{'delete': {'_id': '7133d6e5-a332-455d-b0c8-973ca420f686', '_index': 'records-default-v1.0.0', '_shards': {'failed': 0, 'successful': 1, 'total': 2}, '_type': 'default-v1.0.0', ...}}], resp = {'errors': False, 'items': [{}, {}], 'took': 7}

    def _process_bulk_chunk(client, bulk_actions, raise_on_exception=True, raise_on_error=True, **kwargs):

[...TRUNCATED FOR BREVITY...]

        if errors:
>           raise BulkIndexError('%i document(s) failed to index.' % len(errors), errors)
E           elasticsearch.helpers.BulkIndexError: ('1 document(s) failed to index.', [{'delete': {'found': False, '_version': 2, '_shards': {'successful': 1, 'total': 2, 'failed': 0}, '_id': '7133d6e5-a332-455d-b0c8-973ca420f686', 'status': 404, '_type': 'default-v1.0.0', '_index': 'records-default-v1.0.0'}}])

../../lib/python3.5/site-packages/elasticsearch/helpers/__init__.py:134: BulkIndexError

After some more digging it looks like two delete messages are queued when one is published here

This only happens when tests/test_cli.py::test_reindex is ran before tests/test_utils.py::test_record_indexing like this:

py.test tests/test_cli.py::test_reindex tests/test_utils.py::test_record_indexing

If tests/test_utils.py::test_record_indexing alone is ran then the issue does not appear.

I tracked down the issue to the BROCKER_URL config variable. The problem arises only with BROKER_URL=os.environ.get('BROKER_URL', 'memory://') but not with rabbitmq or redis.