Incorrect types are used to describe google.protobuf enums

[vmagamedov]

Let's take google.protobuf.descriptor_pb2.FieldDescriptorProto as an example.

In Python FieldDescriptorProto.TYPE_DOUBLE is instance of int
In mypy FieldDescriptorProto.TYPE_DOUBLE is instance of FieldDescriptorProto.Type

In Python FieldDescriptorProto.Type is instance of EnumTypeWrapper
In mypy FieldDescriptorProto.Type is subclass of int with additional methods

In Python FieldDescriptorProto.Type.keys() is of type List[str]
In mypy FieldDescriptorProto.Type.keys() is of type List[bytes]

In Python EnumTypeWrapper.items is an instancemethod
In mypy EnumTypeWrapper.items is a classmethod

In mypy this code works:

from google.protobuf.descriptor_pb2 import FieldDescriptorProto
print(FieldDescriptorProto.TYPE_DOUBLE.values())

In Python it fails:

Traceback (most recent call last):
  File "sandbox.py", line 5, in <module>
    print(FieldDescriptorProto.TYPE_DOUBLE.values())
AttributeError: 'int' object has no attribute 'values'

Currently this stub looks like this:

class FieldDescriptorProto(Message):
    class Type(int):
        @classmethod
        def Name(cls, number: int) -> bytes: ...

        @classmethod
        def Value(cls, name: bytes) -> FieldDescriptorProto.Type: ...

        @classmethod
        def keys(cls) -> List[bytes]: ...

        @classmethod
        def values(cls) -> List[FieldDescriptorProto.Type]: ...

        @classmethod
        def items(cls) -> List[Tuple[bytes, FieldDescriptorProto.Type]]: ...
    TYPE_DOUBLE: Type
    ...

I think it is possible to change it into this:

class FieldDescriptorProto(Message):
    FieldDescriptorProto_Type = typing.NewType('FieldDescriptorProto_Type', int)
    Type: EnumTypeWrapper[FieldDescriptorProto_Type]
    TYPE_DOUBLE: FieldDescriptorProto_Type
    ...

where google.protobuf.internal.enum_type_wrapper.EnumTypeWrapper looks like this:

class EnumTypeWrapper(Generic[T]):
    def Name(self, number: T) -> str: ...
    def Value(self, name: str) -> T: ...
    def keys(self) -> List[str]: ...
    def values(self) -> List[T]: ...
    def items(self) -> List[Tuple[str, T]]: ...

The only problem I see is how to generate unique type name for enum values, in the example above I'm using concatenation of message name and enum name: FieldDescriptorProto_Type.

Is it ok to create PR with this fix?

[JelleZijlstra]

This looks reasonable. I would put the NewType in the global scope instead of within the class and give it a _-prefixed name to ensure that it is private.

[vmagamedov]

Great, so I'm going to submit PR soon.

[vmagamedov]

@JelleZijlstra I've done some refactorings in vmagamedov@fe85fcd and it turns out that there are lots of changes needed. I've managed to complete them, but it is easy to make a mistake there and it would be hard to review :( I have an idea to automatically generate all *_pb2.py files from corresponding *.proto files by using mypy-protobuf, after adding some changes to it (maybe later?).

And there is a problem, enums declared on the module level can be imported into other modules, and enum value types (declared with NewType) should be imported as well. It would be odd if they remain to be _-prefixed.

[nipunn1313]

hi - mypy-protobuf recently made updates which fixes this issue - I can run mypy-protobuf on the google google.protobuf protos and try to commit them back into typeshed.

That'll make them more correct - though it might be a backward compatibility painpoint for those using them.