inconsistent behavior of annotation upload

[mromanello]

I've been trying to upload some annotations via pycaprio to an existing project, but the behavior is somehow inconsistent.

I have the following minimum example

import zipfile
import io
import os
from pycaprio import Pycaprio
from pycaprio.mappings import InceptionFormat, AnnotationState
from pycaprio.core.exceptions import InceptionBadResponse

client = Pycaprio()

from pycaprio.mappings import InceptionFormat, AnnotationState
with open("/home/romanell/Downloads/clef-annotations/data/fr/GDL-1968-05-14-a-i0132.xmi") as annotation_file:
    try:
        new_annotation = client.api.create_annotation(
            2,
            1772,
            'mromanello',
            annotation_file,
            annotation_format=InceptionFormat.XMI,
        )
    except InceptionBadResponse as e:
        print(e.bad_response.content)
        raise e

which fails with the following exception

b'{"messages":[{"level":"ERROR","message":"Internal server error: org.xml.sax.SAXParseException; Premature end of file."}]}'
---------------------------------------------------------------------------
RetryableException                        Traceback (most recent call last)
~/.local/share/virtualenvs/CLEF-HIPE-2020-internal-xEF6Qd5X/lib/python3.6/site-packages/tenacity/__init__.py in call(self, fn, *args, **kwargs)
    361                 try:
--> 362                     result = fn(*args, **kwargs)
    363                 except BaseException:

~/.local/share/virtualenvs/CLEF-HIPE-2020-internal-xEF6Qd5X/lib/python3.6/site-packages/pycaprio/core/clients/retryable_client.py in _request(self, method, url, allowed_statuses, **kwargs)
     65         if status_code_is_retryable and status_code_is_not_allowed:
---> 66             raise RetryableException(response)
     67         elif status_code_is_not_allowed:

RetryableException: 

The above exception was the direct cause of the following exception:

RetryError                                Traceback (most recent call last)
~/.local/share/virtualenvs/CLEF-HIPE-2020-internal-xEF6Qd5X/lib/python3.6/site-packages/pycaprio/core/clients/retryable_client.py in request(self, method, url, allowed_statuses, **kwargs)
     47         try:
---> 48             return self._request(method, url, allowed_statuses, **kwargs)
     49         except RetryError as retry_error:

~/.local/share/virtualenvs/CLEF-HIPE-2020-internal-xEF6Qd5X/lib/python3.6/site-packages/tenacity/__init__.py in wrapped_f(*args, **kw)
    292         def wrapped_f(*args, **kw):
--> 293             return self.call(f, *args, **kw)
    294 

~/.local/share/virtualenvs/CLEF-HIPE-2020-internal-xEF6Qd5X/lib/python3.6/site-packages/tenacity/__init__.py in call(self, fn, *args, **kwargs)
    358         while True:
--> 359             do = self.iter(retry_state=retry_state)
    360             if isinstance(do, DoAttempt):

~/.local/share/virtualenvs/CLEF-HIPE-2020-internal-xEF6Qd5X/lib/python3.6/site-packages/tenacity/__init__.py in iter(self, retry_state)
    332                 raise retry_exc.reraise()
--> 333             six.raise_from(retry_exc, fut.exception())
    334 

~/.local/share/virtualenvs/CLEF-HIPE-2020-internal-xEF6Qd5X/lib/python3.6/site-packages/six.py in raise_from(value, from_value)

RetryError: RetryError[<Future at 0x7f0b57bd8b00 state=finished raised RetryableException>]

During handling of the above exception, another exception occurred:

InceptionBadResponse                      Traceback (most recent call last)
<ipython-input-17-44207ea1883a> in <module>
     20     except InceptionBadResponse as e:
     21         print(e.bad_response.content)
---> 22         raise e
     23 

<ipython-input-17-44207ea1883a> in <module>
     16             'mromanello',
     17             annotation_file,
---> 18             annotation_format=InceptionFormat.XMI,
     19         )
     20     except InceptionBadResponse as e:

~/.local/share/virtualenvs/CLEF-HIPE-2020-internal-xEF6Qd5X/lib/python3.6/site-packages/pycaprio/core/adapters/http_adapter.py in create_annotation(self, project_id, document_id, user_name, content, annotation_format, annotation_state)
     85                                     form_data={'format': annotation_format, 'state': annotation_state},
     86                                     files={"content": ('test/path', content)},
---> 87                                     allowed_statuses=(201, 200))
     88         annotation = AnnotationSchema().load(response.json()['body'], many=False)
     89         annotation.project_id = project_id

~/.local/share/virtualenvs/CLEF-HIPE-2020-internal-xEF6Qd5X/lib/python3.6/site-packages/pycaprio/core/clients/retryable_client.py in post(self, url, data, form_data, json, files, allowed_statuses)
     38             for k, v in form_data.items():
     39                 files[k] = (None, v)
---> 40         return self.request('post', url, allowed_statuses, data=data, json=json, files=files)
     41 
     42     def delete(self, url: str, allowed_statuses: Optional[status_list_type] = None) -> requests.Response:

~/.local/share/virtualenvs/CLEF-HIPE-2020-internal-xEF6Qd5X/lib/python3.6/site-packages/pycaprio/core/clients/retryable_client.py in request(self, method, url, allowed_statuses, **kwargs)
     48             return self._request(method, url, allowed_statuses, **kwargs)
     49         except RetryError as retry_error:
---> 50             raise InceptionBadResponse(retry_error.last_attempt._exception.bad_response)
     51 
     52     @retry(retry=retry_if_exception_type(RetryableException), stop=stop_after_attempt(MAX_RETRY_ATTEMPTS))

InceptionBadResponse: 

Despite the failed call, the annotations are created in inception, but the annotation status is not set to complete.

Trying to do the same via inception's swagger UI works without any issues.

NB: I realize this may be hard to reproduce on your end, so I'll be happy to share project exports (privately) if helpful.

[JavierLuna]

Could you surround the part that is failing like this?

from pycaprio.core.exceptions import InceptionBadResponse

try:
    your uploading code
except InceptionBadResponse as e:
 print(e.bad_response.content)
 raise e 

And yes, If you'd be so kind to send some files so I can reproduce it, send them over to jluna@savanamed.com

[JavierLuna]

Somewhat related, I'll try to implement a broader set of excepitons with a better error description.
#24

[mromanello]

I've also noted a strangetest/pathoccurring a few times in core/adapter/http_adapter.py:

files={"content": ('test/path', content)},

perhaps is related to this issue?

[JavierLuna]

It seems weird but I don't think this is the cause of the bug, this is a "trick" to allow pycaprio's functions to accept io streams as arguments and upload them as if they were files.

Nevertheless, I'll look into it when debugging this ticket.

[JavierLuna]

Fixed on version 0.0.3.

Thank you for the bug report and for the help debugging :)