Tests fails in Debian with python 3.8

[elMor3no]

im trying to packages python-jsonrpc-server for Debian but i get this issue on the test with python3.8
Im try to use the patch from https://git.archlinux.org/svntogit/community.git/tree/trunk/python-3.8.patch?h=packages/python-jsonrpc-server but i get this issue:

I: pybuild base:217: cd /pkg-python-jsonrpc-server/python-jsonrpc-server/.pybuild/cpython3_3.8/build; python3.8 -m pytest test
============================= test session starts ==============================
platform linux -- Python 3.8.4rc1, pytest-4.6.11, py-1.8.1, pluggy-0.13.0
rootdir: /pkg-python-jsonrpc-server/python-jsonrpc-server, inifile: setup.cfg, testpaths: test
plugins: cov-2.8.1
collected 27 items

test/test_endpoint.py ......................                             [ 81%]
test/test_streams.py ...FF                                               [100%]

=================================== FAILURES ===================================
_________________________________ test_writer __________________________________

wfile = <_io.BytesIO object at 0x7f61fd7cf1d0>
writer = <pyls_jsonrpc.streams.JsonRpcStreamWriter object at 0x7f61fd7ccdf0>

    def test_writer(wfile, writer):
        writer.write({
            'id': 'hello',
            'method': 'method',
            'params': {}
        })
    
        if os.name == 'nt':
            assert wfile.getvalue() == (
                b'Content-Length: 49\r\n'
                b'Content-Type: application/vscode-jsonrpc; charset=utf8\r\n'
                b'\r\n'
                b'{"id": "hello", "method": "method", "params": {}}'
            )
        else:
>           assert wfile.getvalue() == (
                b'Content-Length: 44\r\n'
                b'Content-Type: application/vscode-jsonrpc; charset=utf8\r\n'
                b'\r\n'
                b'{"id":"hello","method":"method","params":{}}'
            )
E           assert b'Content-Len..."params": {}}' == b'Content-Leng...,"params":{}}'
E             At index 17 diff: 57 != 52
E             Left contains 5 more items, first extra item: 58
E             Use -v to get the full diff

test/test_streams.py:90: AssertionError
___________________________ test_writer_bad_message ____________________________

wfile = <_io.BytesIO object at 0x7f61fd7b9db0>
writer = <pyls_jsonrpc.streams.JsonRpcStreamWriter object at 0x7f61fe0e23d0>

    def test_writer_bad_message(wfile, writer):
        # A datetime isn't serializable(or poorly serializable),
        # ensure the write method doesn't throw
        import datetime
        writer.write(datetime.datetime(
            year=2019,
            month=1,
            day=1,
            hour=1,
            minute=1,
            second=1,
        ))
    
        if os.name == 'nt':
            assert wfile.getvalue() == b''
        else:
>           assert wfile.getvalue() == (
                b'Content-Length: 10\r\n'
                b'Content-Type: application/vscode-jsonrpc; charset=utf8\r\n'
                b'\r\n'
                b'1546304461'
            )
E           AssertionError: assert b'' == b'Content-Length: 10\r\nCon...rset=utf8\r\n\r\n1546304461'
E             Right contains 88 more items, first extra item: 67
E             Use -v to get the full diff

test/test_streams.py:114: AssertionError
------------------------------ Captured log call -------------------------------
ERROR    pyls_jsonrpc.streams:streams.py:112 Failed to write message to output file 2019-01-01 01:01:01
Traceback (most recent call last):
  File "/pkg-python-jsonrpc-server/python-jsonrpc-server/.pybuild/cpython3_3.8/build/pyls_jsonrpc/streams.py", line 98, in write
    body = json.dumps(message, **self._json_dumps_args)
  File "/usr/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type datetime is not JSON serializable
- generated xml file: /pkg-python-jsonrpc-server/python-jsonrpc-server/.pybuild/cpython3_3.8/build/pytest.xml -

--------- coverage: platform linux, python 3.8.4-candidate-1 ---------
Name                    Stmts   Miss  Cover
-------------------------------------------
test/__init__.py            0      0   100%
test/test_endpoint.py     133      4    97%
test/test_streams.py       46      2    96%
-------------------------------------------
TOTAL                     179      6    97%
Coverage HTML written to dir htmlcov

===================== 2 failed, 25 passed in 0.40 seconds ======================
Coverage.py warning: Module pyls was never imported. (module-not-imported)
E: pybuild pybuild:352: test: plugin distutils failed with: exit code=1: cd /pkg-python-jsonrpc-server/python-jsonrpc-server/.pybuild/cpython3_3.8/build; python3.8 -m pytest test

Any idea?

[elMor3no]

Hi @maximbaz

I saw you write a patch for the issue with python3.8 for #33 Im using your path for debian but im getting this errors.
Any clue?

[maximbaz]

Hi, uh, sorry, I don't have any obvious idea :(

[elMor3no]

For more information I run the command in verbose mode and I get more information

# python3.8 -m pytest test -vv 
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.8.5, pytest-4.6.11, py-1.8.1, pluggy-0.13.0 -- /usr/bin/python3.8
cachedir: .pytest_cache
rootdir: /pkg-python-jsonrpc-server/python-jsonrpc-server, inifile: setup.cfg, testpaths: test
plugins: cov-2.8.1
collected 27 items                                                                                                                                                                                                

test/test_endpoint.py::test_bad_message PASSED                                                                                                                                                              [  3%]
test/test_endpoint.py::test_notify PASSED                                                                                                                                                                   [  7%]
test/test_endpoint.py::test_notify_none_params PASSED                                                                                                                                                       [ 11%]
test/test_endpoint.py::test_request PASSED                                                                                                                                                                  [ 14%]
test/test_endpoint.py::test_request_error FAILED                                                                                                                                                            [ 18%]
test/test_endpoint.py::test_request_cancel FAILED                                                                                                                                                           [ 22%]
test/test_endpoint.py::test_consume_notification PASSED                                                                                                                                                     [ 25%]
test/test_endpoint.py::test_consume_notification_error PASSED                                                                                                                                               [ 29%]
test/test_endpoint.py::test_consume_notification_method_not_found PASSED                                                                                                                                    [ 33%]
test/test_endpoint.py::test_consume_async_notification_error PASSED                                                                                                                                         [ 37%]
test/test_endpoint.py::test_consume_request PASSED                                                                                                                                                          [ 40%]
test/test_endpoint.py::test_consume_future_request PASSED                                                                                                                                                   [ 44%]
test/test_endpoint.py::test_consume_async_request PASSED                                                                                                                                                    [ 48%]
test/test_endpoint.py::test_consume_async_request_error[ValueError-error0] PASSED                                                                                                                           [ 51%]
test/test_endpoint.py::test_consume_async_request_error[KeyError-error1] PASSED                                                                                                                             [ 55%]
test/test_endpoint.py::test_consume_async_request_error[JsonRpcMethodNotFound-error2] PASSED                                                                                                                [ 59%]
test/test_endpoint.py::test_consume_request_method_not_found PASSED                                                                                                                                         [ 62%]
test/test_endpoint.py::test_consume_request_error[ValueError-error0] PASSED                                                                                                                                 [ 66%]
test/test_endpoint.py::test_consume_request_error[KeyError-error1] PASSED                                                                                                                                   [ 70%]
test/test_endpoint.py::test_consume_request_error[JsonRpcMethodNotFound-error2] PASSED                                                                                                                      [ 74%]
test/test_endpoint.py::test_consume_request_cancel PASSED                                                                                                                                                   [ 77%]
test/test_endpoint.py::test_consume_request_cancel_unknown PASSED                                                                                                                                           [ 81%]
test/test_streams.py::test_reader PASSED                                                                                                                                                                    [ 85%]
test/test_streams.py::test_reader_bad_message PASSED                                                                                                                                                        [ 88%]
test/test_streams.py::test_reader_bad_json PASSED                                                                                                                                                           [ 92%]
test/test_streams.py::test_writer FAILED                                                                                                                                                                    [ 96%]
test/test_streams.py::test_writer_bad_message FAILED                                                                                                                                                        [100%]Coverage.py warning: Module pyls was never imported. (module-not-imported)


==================================================================================================== FAILURES =====================================================================================================
_______________________________________________________________________________________________ test_request_error ________________________________________________________________________________________________

endpoint = <pyls_jsonrpc.endpoint.Endpoint object at 0x7feb3d18a7f0>, consumer = <MagicMock id='140648319068096'>

    def test_request_error(endpoint, consumer):
        future = endpoint.request('methodName', {'key': 'value'})
        assert not future.done()
    
        consumer.assert_called_once_with({
            'jsonrpc': '2.0',
            'id': MSG_ID,
            'method': 'methodName',
            'params': {'key': 'value'}
        })
    
        # Send an error back from the client
        error = exceptions.JsonRpcInvalidRequest(data=1234)
>       endpoint.consume({
            'jsonrpc': '2.0',
            'id': MSG_ID,
            'error': error.to_dict()
        })

test/test_endpoint.py:86: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pyls_jsonrpc/endpoint.py:109: in consume
    self._handle_response(message['id'], message.get('result'), message.get('error'))
pyls_jsonrpc/endpoint.py:241: in _handle_response
    request_future.set_result(result)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Future at 0x7feb3d18a6d0 state=finished raised JsonRpcInvalidRequest>, result = None

    def set_result(self, result):
        """Sets the return value of work associated with the future.
    
        Should only be used by Executor implementations and unit tests.
        """
        with self._condition:
            if self._state in {CANCELLED, CANCELLED_AND_NOTIFIED, FINISHED}:
>               raise InvalidStateError('{}: {!r}'.format(self._state, self))
E               concurrent.futures._base.InvalidStateError: FINISHED: <Future at 0x7feb3d18a6d0 state=finished raised JsonRpcInvalidRequest>

/usr/lib/python3.8/concurrent/futures/_base.py:524: InvalidStateError
_______________________________________________________________________________________________ test_request_cancel _______________________________________________________________________________________________

endpoint = <pyls_jsonrpc.endpoint.Endpoint object at 0x7feb3d063d00>, consumer = <MagicMock id='140648317861216'>

    def test_request_cancel(endpoint, consumer):
        future = endpoint.request('methodName', {'key': 'value'})
        assert not future.done()
    
        consumer.assert_called_once_with({
            'jsonrpc': '2.0',
            'id': MSG_ID,
            'method': 'methodName',
            'params': {'key': 'value'}
        })
    
        # Cancel the request
        future.cancel()
        consumer.assert_any_call({
            'jsonrpc': '2.0',
            'method': '$/cancelRequest',
            'params': {'id': MSG_ID}
        })
    
        with pytest.raises(exceptions.JsonRpcException) as exc_info:
>           assert future.result(timeout=2)

test/test_endpoint.py:119: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Future at 0x7feb3d151f70 state=cancelled>, timeout = 2

    def result(self, timeout=None):
        """Return the result of the call that the future represents.
    
        Args:
            timeout: The number of seconds to wait for the result if the future
                isn't done. If None, then there is no limit on the wait time.
    
        Returns:
            The result of the call that the future represents.
    
        Raises:
            CancelledError: If the future was cancelled.
            TimeoutError: If the future didn't finish executing before the given
                timeout.
            Exception: If the call raised then that exception will be raised.
        """
        with self._condition:
            if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:
>               raise CancelledError()
E               concurrent.futures._base.CancelledError

/usr/lib/python3.8/concurrent/futures/_base.py:430: CancelledError
------------------------------------------------------------------------------------------------ Captured log call ------------------------------------------------------------------------------------------------
ERROR    concurrent.futures:_base.py:330 exception calling callback for <Future at 0x7feb3d151f70 state=cancelled>
Traceback (most recent call last):
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 328, in _invoke_callbacks
    callback(self)
  File "/pkg-python-jsonrpc-server/python-jsonrpc-server/pyls_jsonrpc/endpoint.py", line 91, in callback
    future.set_exception(JsonRpcRequestCancelled())
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 539, in set_exception
    raise InvalidStateError('{}: {!r}'.format(self._state, self))
concurrent.futures._base.InvalidStateError: CANCELLED: <Future at 0x7feb3d151f70 state=cancelled>
___________________________________________________________________________________________________ test_writer ___________________________________________________________________________________________________

wfile = <_io.BytesIO object at 0x7feb3d18e040>, writer = <pyls_jsonrpc.streams.JsonRpcStreamWriter object at 0x7feb3d194430>

    def test_writer(wfile, writer):
        writer.write({
            'id': 'hello',
            'method': 'method',
            'params': {}
        })
    
        if os.name == 'nt':
            assert wfile.getvalue() == (
                b'Content-Length: 49\r\n'
                b'Content-Type: application/vscode-jsonrpc; charset=utf8\r\n'
                b'\r\n'
                b'{"id": "hello", "method": "method", "params": {}}'
            )
        else:
>           assert wfile.getvalue() == (
                b'Content-Length: 44\r\n'
                b'Content-Type: application/vscode-jsonrpc; charset=utf8\r\n'
                b'\r\n'
                b'{"id":"hello","method":"method","params":{}}'
            )
E           assert b'Content-Len..."params": {}}' == b'Content-Leng...,"params":{}}'
E             At index 17 diff: 57 != 52
E             Left contains 5 more items, first extra item: 58
E             Full diff:
E             - (b'Content-Length: 49\r\nContent-Type: application/vscode-jsonrpc; charset=ut'
E             ?                     ^
E             + (b'Content-Length: 44\r\nContent-Type: application/vscode-jsonrpc; charset=ut'
E             ?                     ^
E             -  b'f8\r\n\r\n{"id": "hello", "method": "method", "params": {}}')
E             ?                    -        -         -         -         -
E             +  b'f8\r\n\r\n{"id":"hello","method":"method","params":{}}')

test/test_streams.py:90: AssertionError
_____________________________________________________________________________________________ test_writer_bad_message _____________________________________________________________________________________________

wfile = <_io.BytesIO object at 0x7feb3d1f5270>, writer = <pyls_jsonrpc.streams.JsonRpcStreamWriter object at 0x7feb3d1924c0>

    def test_writer_bad_message(wfile, writer):
        # A datetime isn't serializable(or poorly serializable),
        # ensure the write method doesn't throw
        import datetime
        writer.write(datetime.datetime(
            year=2019,
            month=1,
            day=1,
            hour=1,
            minute=1,
            second=1,
        ))
    
        if os.name == 'nt':
            assert wfile.getvalue() == b''
        else:
>           assert wfile.getvalue() == (
                b'Content-Length: 10\r\n'
                b'Content-Type: application/vscode-jsonrpc; charset=utf8\r\n'
                b'\r\n'
                b'1546304461'
            )
E           AssertionError: assert b'' == b'Content-Length: 10\r\nCon...rset=utf8\r\n\r\n1546304461'
E             Right contains 88 more items, first extra item: 67
E             Full diff:
E             - b''
E             + (b'Content-Length: 10\r\nContent-Type: application/vscode-jsonrpc; charset=ut'
E             +  b'f8\r\n\r\n1546304461')

test/test_streams.py:114: AssertionError
------------------------------------------------------------------------------------------------ Captured log call ------------------------------------------------------------------------------------------------
ERROR    pyls_jsonrpc.streams:streams.py:112 Failed to write message to output file 2019-01-01 01:01:01
Traceback (most recent call last):
  File "/pkg-python-jsonrpc-server/python-jsonrpc-server/pyls_jsonrpc/streams.py", line 98, in write
    body = json.dumps(message, **self._json_dumps_args)
  File "/usr/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type datetime is not JSON serializable
================================================================================================ warnings summary =================================================================================================
test/test_endpoint.py::test_bad_message
  /pkg-python-jsonrpc-server/python-jsonrpc-server/pyls_jsonrpc/endpoint.py:101: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
    log.warn("Unknown message type %s", message)

test/test_endpoint.py::test_consume_notification_method_not_found
  /pkg-python-jsonrpc-server/python-jsonrpc-server/pyls_jsonrpc/endpoint.py:138: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
    log.warn("Ignoring notification for unknown method %s", method)

test/test_endpoint.py::test_consume_request_cancel_unknown
  /pkg-python-jsonrpc-server/python-jsonrpc-server/pyls_jsonrpc/endpoint.py:168: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
    log.warn("Received cancel notification for unknown message id %s", msg_id)

-- Docs: https://docs.pytest.org/en/latest/warnings.html
----------------------------------------------------------------- generated xml file: /pkg-python-jsonrpc-server/python-jsonrpc-server/pytest.xml -----------------------------------------------------------------

----------- coverage: platform linux, python 3.8.5-final-0 -----------
Name                    Stmts   Miss  Cover
-------------------------------------------
test/__init__.py            0      0   100%
test/test_endpoint.py     133      9    93%
test/test_streams.py       46      2    96%
-------------------------------------------
TOTAL                     179     11    94%
Coverage HTML written to dir htmlcov

================================================================================= 4 failed, 23 passed, 3 warnings in 1.09 seconds =================================================================================

[elMor3no]

I was consulting and there are extra spaces between the JSON elements.
Im still dont solve this issue.

[bnavigator]

The error message for test_request_cancel looks like you did not apply the patch:

        with pytest.raises(exceptions.JsonRpcException) as exc_info:
>           assert future.result(timeout=2)

test/test_endpoint.py:119: 

It should look like this:

python-jsonrpc-server/test/test_endpoint.py

Lines 118 to 119 in 7987e30

	with pytest.raises((exceptions.JsonRpcException, futures.CancelledError)) as exc_info:
	assert future.result(timeout=2)

Also, did you try (already merged into develop but not released yet) #37 which is a small enhancement of @maximbaz's patch?

[elMor3no]

Im still have the same issue evens on version 0.4.0 with python 3.9 and ujson 4.0.1
Not sure why still happends this...

=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.9.0+, pytest-4.6.11, py-1.9.0, pluggy-0.13.0
rootdir: /root/python-jsonrpc-server, inifile: setup.cfg, testpaths: test
plugins: cov-2.8.1
collected 27 items                                                                                                                                                                                                

test/test_endpoint.py ......................                                                                                                                                                                [ 81%]
test/test_streams.py ....F                                                                                                                                                                                  [100%]Coverage.py warning: Module pyls was never imported. (module-not-imported)


==================================================================================================== FAILURES =====================================================================================================
_____________________________________________________________________________________________ test_writer_bad_message _____________________________________________________________________________________________

wfile = <_io.BytesIO object at 0x7f4734302a90>, writer = <pyls_jsonrpc.streams.JsonRpcStreamWriter object at 0x7f4734320790>

    def test_writer_bad_message(wfile, writer):
        # A datetime isn't serializable(or poorly serializable),
        # ensure the write method doesn't throw, but the result could be empty
        # or the correct datetime
        datetime.datetime = JsonDatetime
        writer.write(datetime.datetime(
            year=2019,
            month=1,
            day=1,
            hour=1,
            minute=1,
            second=1,
        ))
    
>       assert wfile.getvalue() in [
            b'',
            b'Content-Length: 10\r\n'
            b'Content-Type: application/vscode-jsonrpc; charset=utf8\r\n'
            b'\r\n'
            b'1546304461'
        ]
E       AssertionError: assert b'Content-Length: 10\r\nContent-Type: application/vscode-jsonrpc; charset=utf8\r\n\r\n1546311661' in [b'', b'Content-Length: 10\r\nContent-Type: application/vscode-jsonrpc; charset=utf8\r\n\r\n1546304461']
E        +  where b'Content-Length: 10\r\nContent-Type: application/vscode-jsonrpc; charset=utf8\r\n\r\n1546311661' = <built-in method getvalue of _io.BytesIO object at 0x7f4734302a90>()
E        +    where <built-in method getvalue of _io.BytesIO object at 0x7f4734302a90> = <_io.BytesIO object at 0x7f4734302a90>.getvalue

test/test_streams.py:124: AssertionError
--------------------------------------------------------------------------- generated xml file: /root/python-jsonrpc-server/pytest.xml ----------------------------------------------------------------------------

----------- coverage: platform linux, python 3.9.0-final-0 -----------
Name                    Stmts   Miss  Cover
-------------------------------------------
test/__init__.py            0      0   100%
test/test_endpoint.py     133      4    97%
test/test_streams.py       52      2    96%
-------------------------------------------
TOTAL                     185      6    97%
Coverage HTML written to dir htmlcov

======================================================================================= 1 failed, 26 passed in 2.00 seconds =======================================================================================

[elMor3no]

Also, did you try (already merged into develop but not released yet) #37 which is a small enhancement of @maximbaz's patch?

This works only one month https://salsa.debian.org/elMor3no-guest/python-jsonrpc-server/-/jobs/1005460#L1260

Now, a month later I get the same error https://salsa.debian.org/elMor3no-guest/python-jsonrpc-server/-/jobs/1005460#L1260

Im using the patch
https://salsa.debian.org/elMor3no-guest/python-jsonrpc-server/-/blob/debian/master/debian/patches/python-3.8.patch

[elMor3no]

As I can check the issue is on ujson version.
On version 1.35 was working fine, but now ujson in on version 4.0.1 and jsonrpc have the same error on test