Strange exception: TypeError: Message must be of type <class 'Foo'>, not <class 'Foo'>

[brunorijsman]

I am running into a strange error message when using the ChannelFor and Stub functionality in a pytest.

grpclib.encoding.proto.ProtoCodec.encode throws the following exception:

TypeError: Message must be of type <class 'key_store_pb2.AddBlockRequest'>, not <class 'key_store_pb2.AddBlockRequest'>

Note that the expected type and the actual type are the same, namely <class 'key_store_pb2.AddBlockRequest'>

Program:

import pytest
from grpclib.testing import ChannelFor
from key_store_server import KeyStoreServer
import protobuf_path  # pylint:disable=unused-import
import generated.protobuf.key_store_pb2 as pb2
import generated.protobuf.key_store_grpc as grpc

@pytest.mark.asyncio
async def test_key_store_server():
    server = KeyStoreServer()
    async with ChannelFor([server]) as channel:
        stub = grpc.KeyStoreStub(channel)
        request = pb2.AddBlockRequest(local_app_id="local-app-id",
                                      remote_app_ids=["remote-app-id"],
                                      key_matererial_blocks=[])
        response = await stub.AddBlock(request)
        assert response == "something..."

Proto file:

syntax = "proto3";

package keystore;

/* Was an RPC call successful, and if not, why not? */
message Status {
    /* Was the RPC call successful? */
    bool success = 1;
    /* HTTP status code (for easy mapping to REST APIs). */
    uint32 code = 2;
    /* Human-readable string describing the status. */
    string reason = 3;
}

/* A key with its associated meta-data. */
message KeyContainer {
    /* An opaque ID that uniquely identifies the key. */
    string key_id = 1;
    /* The length of the key in bits. */
    uint64 key_length = 2;
    /* The actual key bits. Prepended with zero bits in the most significant bit positions if
    the key is not a multiple of 8 bits. */
    bytes key = 3;
}

/* A key material block with its associated meta-data. */
message Block {
    /* An opaque ID that uniquely identifies the key material block. */
    string block_id = 1;
    /* The length of the key material block in bits. */
    uint64 length = 2;
    /* The actual key material block bits. Prepended with zero bits in the most significant bit
    positions if the key material block is not a multiple of 8 bits. */
    bytes bits = 3;
}

/* An initiator application sends the GetInitiatorKeysRequest message to the key store to take
one or more keys. The initiator application must take the key first. It gets back a key ID for each
key. These key IDs must be shared with the repsonder applications (the key ID need not be kept
confidential). The responders then invoke GetResponderKeys, providing the key IDs, to get the same
keys as the initiator. */
message GetInitiatorKeysRequest {
    /* The ID of the local initiator application. */
    string local_initiator_app_id = 1;
    /* The IDs of the remote responder application(s) that the key has been shared with. */
    repeated string remote_responder_app_ids = 2;
    /* The number of requested keys. Must be >= 1. */
    uint32 nr_keys = 3;
    /* The length of each requested key in bits. Must be >= 1. */
    uint64 key_length = 4;
}

/* The reply for GetInitiatorKeysRequest. */
message GetInitiatorKeysReply {
    /* Was the request sucessful and if not why not? In case of error, this is the only field. */
    Status status = 1;
    /* The requested keys with associated meta-data, including the key IDs. */
    repeated KeyContainer requested_keys = 2;
}

/* A responder application sends the GetResponderKeysRequest message to the key store to take one
or more key. The responder must provide the key IDs that it got from the intiator application. */
message GetResponderKeysRequest {
    /* The ID of the local responder application. */
    string local_responder_app_id = 1;
    /* The IDs of the remote responder application that the key has been shared with. */
    string remote_initiator_app_id = 2;
    /* The ID(s) of the key(s) that the responder application wants to get. These key IDs came from
    the initiator application. */
    repeated string key_ids = 3;
}

/* The reply for GetResponderKeysRequest. */
message GetResponderKeysReply {
    /* Was the request sucessful and if not why not? In case of error, this is the only field. */
    Status status = 1;
     /* The requested keys with associated meta-data. */
     repeated KeyContainer requested_keys = 2;
}

/* A key producer sends the AddBlockRequest message to store a newly produced block of
key material into the key store. */
message AddBlockRequest {
    /* The ID of the local application that the key has been shared with. */
    string local_app_id = 1;
    /* The IDs of the remote application(s) that the key has been shared with. */
    repeated string remote_app_ids = 2;
    /* The key material block(s) to be added to the key store. */
    repeated Block key_matererial_blocks = 3;
}

/* The reply for AddBlockRequest. */
message AddBlockReply {
    /* Was the request sucessful and if not why not? In case of error, this is the only field. */
    Status status = 1;
}

service KeyStore {
    rpc GetInitiatorKeys (GetInitiatorKeysRequest) returns (GetInitiatorKeysReply);
    rpc GetResponderKeys (GetResponderKeysRequest) returns (GetResponderKeysReply);
    rpc AddBlock (AddBlockRequest) returns (AddBlockReply);
}

Invoking the program:

pytest test_key_store_server.py 

Full error message:

================================================================== FAILURES ===================================================================
____________________________________________________________ test_key_store_server ____________________________________________________________

    @pytest.mark.asyncio
    async def test_key_store_server():
        """TODO"""
        server = KeyStoreServer()
        async with ChannelFor([server]) as channel:
            stub = grpc.KeyStoreStub(channel)
            request = pb2.AddBlockRequest(local_app_id="local-app-id",
                                          remote_app_ids=["remote-app-id"],
                                          key_matererial_blocks=[])
>           response = await stub.AddBlock(request)

test_key_store_server.py:35: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../venv/lib/python3.8/site-packages/grpclib/client.py:850: in __call__
    await stream.send_message(message, end=True)
../../venv/lib/python3.8/site-packages/grpclib/client.py:255: in send_message
    await send_message(self._stream, self._codec, message,
../../venv/lib/python3.8/site-packages/grpclib/stream.py:44: in send_message
    reply_bin = codec.encode(message, message_type)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = , message = local_app_id: "local-app-id"
remote_app_ids: "remote-app-id"

message_type = 

    def encode(
        self,
        message: 'IProtoMessage',
        message_type: Type['IProtoMessage'],
    ) -> bytes:
        if not isinstance(message, message_type):
>           raise TypeError('Message must be of type {!r}, not {!r}'
                            .format(message_type, type(message)))
E           TypeError: Message must be of type , not 

../../venv/lib/python3.8/site-packages/grpclib/encoding/proto.py:45: TypeError
-------------------------------------------------------------- Captured log call --------------------------------------------------------------
ERROR    asyncio:base_events.py:1707 Exception in callback H2Protocol.data_received(b'PRI * HTTP/...\x00@\x00\x00')
handle: 
Traceback (most recent call last):
  File "/Users/brunorijsman/.pyenv/versions/3.8.6/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/Users/brunorijsman/mdi-qkd-mark-4/venv/lib/python3.8/site-packages/grpclib/protocol.py", line 712, in data_received
    events = self.connection.feed(data)
  File "/Users/brunorijsman/mdi-qkd-mark-4/venv/lib/python3.8/site-packages/grpclib/protocol.py", line 189, in feed
    return self._connection.receive_data(data)  # type: ignore
  File "/Users/brunorijsman/mdi-qkd-mark-4/venv/lib/python3.8/site-packages/h2/connection.py", line 1463, in receive_data
    events.extend(self._receive_frame(frame))
  File "/Users/brunorijsman/mdi-qkd-mark-4/venv/lib/python3.8/site-packages/h2/connection.py", line 1486, in _receive_frame
    frames, events = self._frame_dispatch_table[frame.__class__](frame)
AttributeError: 'H2Connection' object has no attribute '_frame_dispatch_table'
ERROR    asyncio:base_events.py:1707 Exception in callback H2Protocol.data_received(b'\x00\x00*\x...\x00@\x00\x00')
handle: 
Traceback (most recent call last):
  File "/Users/brunorijsman/.pyenv/versions/3.8.6/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/Users/brunorijsman/mdi-qkd-mark-4/venv/lib/python3.8/site-packages/grpclib/protocol.py", line 712, in data_received
    events = self.connection.feed(data)
  File "/Users/brunorijsman/mdi-qkd-mark-4/venv/lib/python3.8/site-packages/grpclib/protocol.py", line 189, in feed
    return self._connection.receive_data(data)  # type: ignore
  File "/Users/brunorijsman/mdi-qkd-mark-4/venv/lib/python3.8/site-packages/h2/connection.py", line 1463, in receive_data
    events.extend(self._receive_frame(frame))
  File "/Users/brunorijsman/mdi-qkd-mark-4/venv/lib/python3.8/site-packages/h2/connection.py", line 1486, in _receive_frame
    frames, events = self._frame_dispatch_table[frame.__class__](frame)
AttributeError: 'H2Connection' object has no attribute '_frame_dispatch_table'
ERROR    asyncio:base_events.py:1707 Exception in callback H2Protocol.data_received(b"\x00\x00d\x...\xef.\xe7\xf7")
handle: 
Traceback (most recent call last):
  File "/Users/brunorijsman/.pyenv/versions/3.8.6/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/Users/brunorijsman/mdi-qkd-mark-4/venv/lib/python3.8/site-packages/grpclib/protocol.py", line 712, in data_received
    events = self.connection.feed(data)
  File "/Users/brunorijsman/mdi-qkd-mark-4/venv/lib/python3.8/site-packages/grpclib/protocol.py", line 189, in feed
    return self._connection.receive_data(data)  # type: ignore
  File "/Users/brunorijsman/mdi-qkd-mark-4/venv/lib/python3.8/site-packages/h2/connection.py", line 1463, in receive_data
    events.extend(self._receive_frame(frame))
  File "/Users/brunorijsman/mdi-qkd-mark-4/venv/lib/python3.8/site-packages/h2/connection.py", line 1486, in _receive_frame
    frames, events = self._frame_dispatch_table[frame.__class__](frame)
AttributeError: 'H2Connection' object has no attribute '_frame_dispatch_table'
ERROR    asyncio:base_events.py:1707 Exception in callback H2Protocol.data_received(b'\x00\x00\x0...0\x00\x00\x00')
handle: 
Traceback (most recent call last):
  File "/Users/brunorijsman/.pyenv/versions/3.8.6/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/Users/brunorijsman/mdi-qkd-mark-4/venv/lib/python3.8/site-packages/grpclib/protocol.py", line 712, in data_received
    events = self.connection.feed(data)
  File "/Users/brunorijsman/mdi-qkd-mark-4/venv/lib/python3.8/site-packages/grpclib/protocol.py", line 189, in feed
    return self._connection.receive_data(data)  # type: ignore
  File "/Users/brunorijsman/mdi-qkd-mark-4/venv/lib/python3.8/site-packages/h2/connection.py", line 1463, in receive_data
    events.extend(self._receive_frame(frame))
  File "/Users/brunorijsman/mdi-qkd-mark-4/venv/lib/python3.8/site-packages/h2/connection.py", line 1486, in _receive_frame
    frames, events = self._frame_dispatch_table[frame.__class__](frame)
AttributeError: 'H2Connection' object has no attribute '_frame_dispatch_table'
=========================================================== short test summary info ===========================================================
FAILED test_key_store_server.py::test_key_store_server - TypeError: Message must be of type , not 

[vmagamedov]

I assume that you generated two versions of the same messages or there are two ways of importing them. It is tricky to call protoc correctly. It is also easy to mess with the PYTHONPATH.

Take a look at the imports in the key_store_grpc.py file. Are they look the same as in your program?

import generated.protobuf.key_store_pb2

[vmagamedov]

Tip: it is not necessary to define your own specific Status message, you can use rich error details. You can also map gRPC errors to the corresponding HTTP statuses: https://github.com/googleapis/googleapis/blob/master/google/rpc/code.proto

[brunorijsman]

Thank you. That was indeed the problem. I had "import generated.protobuf.foo_pb2" in my code, where as the generated foo_grpc.py code had "import foo_pb2". I used a "sys.path.append" to be able to consistently use "import foo_pb2" everywhere. This involved some rather ugly hacks to make Visual Studio Code look in the right directories, but I got it to work.

Thanks also for the other tips.

I will close the issue.

[brunorijsman]

Closing (see above)