Compatibility problem 4.x vs 3.x

[ChameleonRed]

What version of protobuf and what language are you using?
Version: pip install protobuf==4.21.7
Language: Python

What operating system (Linux, Windows, ...) and version?

Windows 10

What runtime / compiler are you using (e.g., python version or gcc version)

3.10.7

What did you do?
Steps to reproduce the behavior:

1. Generate files from proto.
2. https://github.com/etcd-io/etcd/blob/main/api/versionpb/version.proto
3. Can be others from here: https://github.com/etcd-io/etcd/tree/main/api
4. Download:

setlocal
set previous_path=%cd%

cd  "%~dp0etcd\api\authpb"
wget -N https://raw.githubusercontent.com/etcd-io/etcd/main/api/authpb/auth.proto

cd  "%~dp0etcd\api\etcdserverpb"
wget -N https://raw.githubusercontent.com/etcd-io/etcd/main/api/etcdserverpb/rpc.proto

cd  "%~dp0etcd\api\mvccpb"
wget -N https://raw.githubusercontent.com/etcd-io/etcd/main/api/mvccpb/kv.proto

cd  "%~dp0etcd\api\versionpb"
wget -N https://raw.githubusercontent.com/etcd-io/etcd/main/api/versionpb/version.proto

cd  "%~dp0gogoproto"
wget -N https://raw.githubusercontent.com/gogo/protobuf/master/gogoproto/gogo.proto

cd  "%~dp0google\protobuf"
wget -N https://raw.githubusercontent.com/protocolbuffers/protobuf/main/src/google/protobuf/descriptor.proto

cd  "%~dp0google\api"
wget -N https://raw.githubusercontent.com/googleapis/googleapis/master/google/api/annotations.proto
wget -N https://raw.githubusercontent.com/googleapis/googleapis/master/google/api/http.proto

cd %previous_path%
endlocal

5. Write some *.cmd file like this.

set path_prefix=.
set output_path=.
python -m grpc_tools.protoc^
 --proto_path=.^
 --python_out=%output_path%^
 --grpc_python_out=%output_path%^
 --mypy_out=%output_path%^
 %path_prefix%/etcd/api/etcdserverpb/rpc.proto^
 %path_prefix%/etcd/api/authpb/auth.proto^
 %path_prefix%/etcd/api/mvccpb/kv.proto^
 %path_prefix%/etcd/api/versionpb/version.proto^
 %path_prefix%/gogoproto/gogo.proto^
 %path_prefix%/google/api/annotations.proto^
 %path_prefix%/google/api/http.proto

6. Try to import generated code -> version_pb2.py

What did you expect to see
Just working import ... or better explanation what is wrong and how to fix it if it error.

What did you see instead?

Traceback (most recent call last):
  File "C:\Users\Cezary Wagner\PycharmProjects\medptr-v2\test\authentication\sync_etcd3_client_authentication_server_localhost.py", line 8, in <module>
    from authentication.grpc import authentication_pb2_grpc, authentication_pb2
  File "C:\Users\Cezary Wagner\PycharmProjects\medptr-v2\src\authentication\__init__.py", line 9, in <module>
    from authentication.etcd3_client_connector import AuthenticationEtcd3ClientConnector
  File "C:\Users\Cezary Wagner\PycharmProjects\medptr-v2\src\authentication\etcd3_client_connector.py", line 21, in <module>
    from etcd3_client.client import EtcdClient
  File "C:\Users\Cezary Wagner\PycharmProjects\medptr-v2\src\etcd3_client\client.py", line 16, in <module>
    from etcd3_client.base_client import BaseEtcdClient
  File "C:\Users\Cezary Wagner\PycharmProjects\medptr-v2\src\etcd3_client\base_client.py", line 8, in <module>
    from etcd3_grpc.grpc.etcd.api.etcdserverpb import rpc_pb2
  File "C:\Users\Cezary Wagner\PycharmProjects\medptr-v2\src\etcd3_grpc\grpc\etcd\api\etcdserverpb\rpc_pb2.py", line 17, in <module>
    from etcd.api.versionpb import version_pb2 as etcd_dot_api_dot_versionpb_dot_version__pb2
  File "C:\Users\Cezary Wagner\PycharmProjects\medptr-v2\src\etcd3_grpc\grpc\etcd\api\versionpb\version_pb2.py", line 18, in <module>
    DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n etcd/api/versionpb/version.proto\x12\tversionpb\x1a\x14gogoproto/gogo.proto\x1a google/protobuf/descriptor.proto:>\n\x10\x65tcd_version_msg\x12\x1f.google.protobuf.MessageOptions\x18\xd0\x86\x03 \x01(\t\x88\x01\x01:>\n\x12\x65tcd_version_field\x12\x1d.google.protobuf.FieldOptions\x18\xd1\x86\x03 \x01(\t\x88\x01\x01:<\n\x11\x65tcd_version_enum\x12\x1c.google.protobuf.EnumOptions\x18\xd2\x86\x03 \x01(\t\x88\x01\x01:G\n\x17\x65tcd_version_enum_value\x12!.google.protobuf.EnumValueOptions\x18\xd3\x86\x03 \x01(\t\x88\x01\x01\x42\x08\xc8\xe2\x1e\x01\xd0\xe2\x1e\x01\x62\x06proto3')
TypeError: Couldn't build proto file into descriptor pool: field with proto3_optional was not in a oneof (versionpb.etcd_version_msg)

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

Anything else we should know about your project / environment

[ChameleonRed]

Proto is from etcd - nothing special:

syntax = "proto3";
package versionpb;

import "gogoproto/gogo.proto";
import "google/protobuf/descriptor.proto";

option (gogoproto.marshaler_all) = true;
option (gogoproto.unmarshaler_all) = true;

// Indicates etcd version that introduced the message, used to determine minimal etcd version required to interpret wal that includes this message.
extend google.protobuf.MessageOptions {
  optional string etcd_version_msg = 50000;
}

// Indicates etcd version that introduced the field, used to determine minimal etcd version required to interpret wal that sets this field.
extend google.protobuf.FieldOptions {
  optional string etcd_version_field = 50001;
}

// Indicates etcd version that introduced the enum, used to determine minimal etcd version required to interpret wal that uses this enum.
extend google.protobuf.EnumOptions {
  optional string etcd_version_enum = 50002;
}

// Indicates etcd version that introduced the enum value, used to determine minimal etcd version required to interpret wal that sets this enum value.
extend google.protobuf.EnumValueOptions {
  optional string etcd_version_enum_value = 50003;
}

[googleberg]

It looks like the python generator is not handling optional on custom descriptor options in proto3.

[Mokto]

Has anyone managed to fix this ? We're stuck with an old version of grpc (and we can't update to python 3.11) because of this.

[ChameleonRed]

etcd suggest that is problem with Python runtime library but I am not sure.

[Mokto]

I'm not sure why but it seems fixed for us.

Can anyone else confirm that?

[haberman]

This was fixed in protocolbuffers/upb#1034 and released in 4.21.12.