Research alternatives to google.protobuf.internal encoder and decoder

[Vasantha-Kumar-J]

Research on using the encoder and decoder alternatives from non internal locations so that our framework will not have risk when there is a breaking change with the APIs from the google.protobuf.internal.

AB#2267388

AB#2811755

[bkeryan]

Can we create descriptor objects directly? No.

descriptor_test.py:

from google.protobuf.descriptor import Descriptor, FieldDescriptor

int32_in = FieldDescriptor(
    name="int32_in",
    full_name="Configurations.int32_in",
    index=0,
    number=1,
    type=FieldDescriptor.TYPE_INT32,
    cpp_type=FieldDescriptor.CPPTYPE_INT32,
    label=FieldDescriptor.LABEL_OPTIONAL,
    default_value=123,
    message_type=None,
    enum_type=None,
    containing_type=None,
    is_extension=False,
    extension_scope=None,
)
configurations = Descriptor(
    name="Configurations",
    full_name="Configurations",
    filename=None,
    containing_type=None,
    fields=[int32_in],
    nested_types=[],
    enum_types=[],
    extensions=[],
)

PS D:\dev\measurementlink-python> poetry run python .\descriptor_test.py
Traceback (most recent call last):
  File "D:\dev\measurementlink-python\descriptor_test.py", line 3, in <module>
    int32_in = FieldDescriptor(
  File "D:\dev\measurementlink-python\.venv\lib\site-packages\google\protobuf\descriptor.py", line 561, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

Setting the environment variable allows creating descriptor objects directly, but this feature is deprecated.

PS D:\dev\measurementlink-python> $env:PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION='python'
PS D:\dev\measurementlink-python> poetry run python .\descriptor_test.py
D:\dev\measurementlink-python\descriptor_test.py:3: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  int32_in = FieldDescriptor(
D:\dev\measurementlink-python\descriptor_test.py:18: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  configurations = Descriptor(

Cc: @dixonjoel @pbirkhol-ni @jasonmreding

[bkeryan]

Can we create descriptor objects via the descriptor pool? Yes.

descriptor_test2.py:

from google.protobuf import descriptor_pool
from google.protobuf.descriptor_pb2 import (
    DescriptorProto,
    FieldDescriptorProto,
    FileDescriptorProto,
)
from google.protobuf.internal import builder

DESCRIPTOR = descriptor_pool.Default().Add(
    FileDescriptorProto(
        name="asdf.proto",
        package="asdf",
        message_type=[
            DescriptorProto(
                name="Configurations",
                field=[
                    FieldDescriptorProto(
                        name="int32_in",
                        number=1,
                        label=FieldDescriptorProto.LABEL_OPTIONAL,
                        type=FieldDescriptorProto.TYPE_INT32,
                        type_name=None,
                        default_value="123",
                    )
                ],
            )
        ],
    )
)
builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, globals())
builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, "asdf_pb2", globals())

configurations_type = descriptor_pool.Default().FindMessageTypeByName("asdf.Configurations")
print(repr(configurations_type))

msg = Configurations(int32_in=456)
print(type(msg))
print(msg)

PS D:\dev\measurementlink-python> poetry run python .\descriptor_test2.py
<google._upb._message.Descriptor object at 0x000001A38FF78EA0>
<class 'asdf_pb2.Configurations'>
int32_in: 456

Cc: @dixonjoel @pbirkhol-ni @jasonmreding

[bkeryan]

Generating proto file descriptor objects would automatically generate Python protobuf classes at run time, which could be useful for integration testing. Instead of having to write a .proto file to use when calling measurements in tests, clients could use the protobuf classes that are generated at run time. Note that this would require importing the measurement service as a module, which is fine for tests but not appropriate for non-test client use cases.

Generating proto file descriptor objects would not automatically generate a .proto file on disk without additional work. The protobuf C++ API has a Descriptor::DebugString() method that can be used to convert a proto file descriptor into .proto file text. This functionality was exposed in the "cpp" (pyext) implementation for Python, but that implementation is deprecated or obsolete and the "upb" implementation doesn't support this functionality.