TypeError in to_pydict with optional fields

[Alexejhero]

See #475 (comment)

Outdated

I am using betterproto to load a ton of messages at once from a binary file, however this operation is very slow so I wanted to cache/parse it into json using json.dump with to_dict, and then load it directly from json. Problem is that from_json is also very slow, so my idea was to use to_dict(casing=Casing.SNAKE, include_default_values=true) to generate a dictionary that matches the attributes of the message dataclass, and then wrap it in an utility class that redirects dict.attribute calls to dict["attribute"] so that for the consumer (which is using the message dataclasses exclusively for their data) there's no difference.

However the problem is that when serializing to dictionary, enum values are serialized based on their name instead of value. Maybe an optional parameter can be added to change this behaviour? Thank you

Utility dictionary class for redirecting attributes to items

class AttrDict(dict):
    def __getattr__(self, key):
        try:
            return self[key]
        except KeyError:
            raise AttributeError(f"'{self.__class__.__name__}' object has no attribute '{key}'")

    def __setattr__(self, key, value):
        self[key] = value

    def __getitem__(self, key):
        value = super().__getitem__(key)
        if isinstance(value, dict) and not isinstance(value, AttrDict):
            value = AttrDict(value)
            self[key] = value
        if isinstance(value, list) and any(isinstance(x, dict) for x in value):
            value = [AttrDict(x) if isinstance(x, dict) else x for x in value]
            self[key] = value
        return value

[Gobot1234]

You should use to_pydict for this, using strings is for compatibility purposes with the google implementation

[Alexejhero]

to_pydict throws the following error:

  File "...", in <listcomp>
    json.dump([f.to_pydict(casing=Casing.SNAKE, include_default_values=True) for f in frames], file)
  File "...\Python\Python310\lib\site-packages\betterproto\__init__.py", line 1365, in to_pydict
    output[cased_name] = value.to_pydict(casing, include_default_values)
  File "...\Python\Python310\lib\site-packages\betterproto\__init__.py", line 1359, in to_pydict
    value._serialized_on_wire
AttributeError: 'NoneType' object has no attribute '_serialized_on_wire'

[Gobot1234]

You are probably breaking type safety somewhere which isn't really on the libraries shoulders to deal with. If you have a code snippet that passes type checking and reproduces this please do send it.

[Alexejhero]

Entry point

from data.proto import Frame

Frame().to_pydict()

data/proto.py (partial)

# Generated by the protocol buffer compiler.  DO NOT EDIT!

@dataclass(eq=False, repr=False)
class HeaderFrame(betterproto.Message):
    test: bool = betterproto.bool_field(2)

@dataclass(eq=False, repr=False)
class Frame(betterproto.Message):
    header: Optional["HeaderFrame"] = betterproto.message_field(
        2, optional=True, group="_Header"
    )

Causes

AttributeError: 'NoneType' object has no attribute '_serialized_on_wire'

I'm using betterproto version 2.0.0b5

This is just a quick way to reproduce the error, but it also happens with messages deserialized using parse

[nickderobertis]

I just ran into this problem as well, and it seems like it would be easy to fix by just adding a clause to check if the value is None here, something like:

elif value is None:
    output[cased_name] = value

I tried to pull down the repo to make this change an add a test, but I'm having some problems getting the development environment working. Poetry failed to install, so I ran poetry export -E compiler -o requirements.txt and was able to get the dependencies installed in a venv via pip. But then when I try to run poe generate, it outputs a bunch of errors like Failed to generate plugin output for 'service_separate_packages', and then the tests won't run due to the missing generated code. If someone can help me get my dev env working I'm happy to put up a PR for this.

[Gobot1234]

What version of python are you trying to build the repo with and what failed to build when you ran poetry install?

[nickderobertis]

I was originally trying with 3.10 on Mac OS, now just tried with Python 3.11.3, Poetry 1.4.2 on Ubuntu 22.04:

  third_party/protobuf/src/google/protobuf/util/internal/datapiece.cc:57:36: warning: comparison of integer expressions of different signedness: ‘long int’ and ‘long unsigned int’ [-Wsign-compare]
     57 |       MathUtil::Sign<From>(before) == MathUtil::Sign<To>(after)) {
        |       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~
  gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DHAVE_PTHREAD=1 -I. -Igrpc_root -Igrpc_root/include -Ithird_party/protobuf/src -I/tmp/tmptrqf78k0/.venv/include -I/root/.asdf/installs/python/3.11.3/include/python3.11 -c third_party/protobuf/src/google/protobuf/util/message_differencer.cc -o build/temp.linux-x86_64-cpython-311/third_party/protobuf/src/google/protobuf/util/message_differencer.o -std=c++14 -fno-wrapv -frtti
  gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DHAVE_PTHREAD=1 -I. -Igrpc_root -Igrpc_root/include -Ithird_party/protobuf/src -I/tmp/tmptrqf78k0/.venv/include -I/root/.asdf/installs/python/3.11.3/include/python3.11 -c third_party/protobuf/src/google/protobuf/util/internal/protostream_objectwriter.cc -o build/temp.linux-x86_64-cpython-311/third_party/protobuf/src/google/protobuf/util/internal/protostream_objectwriter.o -std=c++14 -fno-wrapv -frtti
  third_party/protobuf/src/google/protobuf/util/internal/utility.cc: In function ‘void google::protobuf::util::converter::InitWellKnownTypes()’:
  third_party/protobuf/src/google/protobuf/util/internal/utility.cc:348:21: warning: comparison of integer expressions of different signedness: ‘int’ and ‘long unsigned int’ [-Wsign-compare]
    348 |   for (int i = 0; i < GOOGLE_ARRAYSIZE(well_known_types_name_array_); ++i) {
  third_party/protobuf/src/google/protobuf/util/internal/protostream_objectwriter.cc: In member function ‘void google::protobuf::util::converter::ProtoStreamObjectWriter::AnyWriter::StartAny(const google::protobuf::util::converter::DataPiece&)’:
  third_party/protobuf/src/google/protobuf/util/internal/protostream_objectwriter.cc:380:21: warning: comparison of integer expressions of different signedness: ‘int’ and ‘std::vector<google::protobuf::util::converter::ProtoStreamObjectWriter::AnyWriter::Event>::size_type’ {aka ‘long unsigned int’} [-Wsign-compare]
    380 |   for (int i = 0; i < uninterpreted_events_.size(); ++i) {
        |                   ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DHAVE_PTHREAD=1 -I. -Igrpc_root -Igrpc_root/include -Ithird_party/protobuf/src -I/tmp/tmptrqf78k0/.venv/include -I/root/.asdf/installs/python/3.11.3/include/python3.11 -c third_party/protobuf/src/google/protobuf/compiler/js/well_known_types_embed.cc -o build/temp.linux-x86_64-cpython-311/third_party/protobuf/src/google/protobuf/compiler/js/well_known_types_embed.o -std=c++14 -fno-wrapv -frtti
  gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DHAVE_PTHREAD=1 -I. -Igrpc_root -Igrpc_root/include -Ithird_party/protobuf/src -I/tmp/tmptrqf78k0/.venv/include -I/root/.asdf/installs/python/3.11.3/include/python3.11 -c third_party/protobuf/src/google/protobuf/timestamp.pb.cc -o build/temp.linux-x86_64-cpython-311/third_party/protobuf/src/google/protobuf/timestamp.pb.o -std=c++14 -fno-wrapv -frtti
  gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DHAVE_PTHREAD=1 -I. -Igrpc_root -Igrpc_root/include -Ithird_party/protobuf/src -I/tmp/tmptrqf78k0/.venv/include -I/root/.asdf/installs/python/3.11.3/include/python3.11 -c third_party/protobuf/src/google/protobuf/wire_format_lite.cc -o build/temp.linux-x86_64-cpython-311/third_party/protobuf/src/google/protobuf/wire_format_lite.o -std=c++14 -fno-wrapv -frtti
  gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DHAVE_PTHREAD=1 -I. -Igrpc_root -Igrpc_root/include -Ithird_party/protobuf/src -I/tmp/tmptrqf78k0/.venv/include -I/root/.asdf/installs/python/3.11.3/include/python3.11 -c third_party/protobuf/src/google/protobuf/descriptor.pb.cc -o build/temp.linux-x86_64-cpython-311/third_party/protobuf/src/google/protobuf/descriptor.pb.o -std=c++14 -fno-wrapv -frtti
  error: command '/usr/bin/gcc' failed with exit code 1
  

  at ~/.local/pipx/venvs/poetry/lib/python3.11/site-packages/poetry/installation/chef.py:152 in _prepare
      148│ 
      149│                 error = ChefBuildError("\n\n".join(message_parts))
      150│ 
      151│             if error is not None:
    → 152│                 raise error from None
      153│ 
      154│             return path
      155│ 
      156│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with grpcio-tools (1.48.2) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "grpcio-tools (==1.48.2)"'.

There is a lot more output before that as well, let me know if it's helpful.

[Gobot1234]

Try using 3.9 a lot of projects in the toolchain still aren't 3.10+ compatible

[nickderobertis]

Ok great, thanks, poetry install worked on 3.9. However, I am still getting the same errors when running poe generate:

(betterproto-py3.9) root@shulgin:~/repos/python-betterproto# poe generate
Poe => generate
Generated reference output for 'bool'
Failed to generate plugin output for 'bool'
Failed to generate plugin (pydantic compatible) output for 'bool'
Generated reference output for 'bytes'
Failed to generate plugin output for 'bytes'
Failed to generate plugin (pydantic compatible) output for 'bytes'
Generated reference output for 'casing'
Failed to generate plugin output for 'casing'
Failed to generate plugin (pydantic compatible) output for 'casing'
Generated reference output for 'casing_inner_class'
Failed to generate plugin output for 'casing_inner_class'
Failed to generate plugin (pydantic compatible) output for 'casing_inner_class'
Generated reference output for 'casing_message_field_uppercase'
Failed to generate plugin output for 'casing_message_field_uppercase'
Failed to generate plugin (pydantic compatible) output for 'casing_message_field_uppercase'
Generated reference output for 'deprecated'
Failed to generate plugin output for 'deprecated'
Failed to generate plugin (pydantic compatible) output for 'deprecated'
Generated reference output for 'double'
Failed to generate plugin output for 'double'
Failed to generate plugin (pydantic compatible) output for 'double'
Generated reference output for 'empty_repeated'
Failed to generate plugin output for 'empty_repeated'
Failed to generate plugin (pydantic compatible) output for 'empty_repeated'
Generated reference output for 'empty_service'
Failed to generate plugin output for 'empty_service'
Failed to generate plugin (pydantic compatible) output for 'empty_service'
Generated reference output for 'entry'
Failed to generate plugin output for 'entry'
Failed to generate plugin (pydantic compatible) output for 'entry'
Generated reference output for 'enum'
Failed to generate plugin output for 'enum'
Failed to generate plugin (pydantic compatible) output for 'enum'
Generated reference output for 'example'
Failed to generate plugin output for 'example'
Failed to generate plugin (pydantic compatible) output for 'example'
Generated reference output for 'example_service'
Failed to generate plugin output for 'example_service'
Failed to generate plugin (pydantic compatible) output for 'example_service'
Generated reference output for 'field_name_identical_to_type'
Failed to generate plugin output for 'field_name_identical_to_type'
Failed to generate plugin (pydantic compatible) output for 'field_name_identical_to_type'
Generated reference output for 'fixed'
Failed to generate plugin output for 'fixed'
Failed to generate plugin (pydantic compatible) output for 'fixed'
Generated reference output for 'float'
Failed to generate plugin output for 'float'
Failed to generate plugin (pydantic compatible) output for 'float'
Generated reference output for 'google_impl_behavior_equivalence'
Failed to generate plugin output for 'google_impl_behavior_equivalence'
Failed to generate plugin (pydantic compatible) output for 'google_impl_behavior_equivalence'
Generated reference output for 'googletypes'
Failed to generate plugin output for 'googletypes'
Failed to generate plugin (pydantic compatible) output for 'googletypes'
Generated reference output for 'googletypes_request'
Failed to generate plugin output for 'googletypes_request'
Failed to generate plugin (pydantic compatible) output for 'googletypes_request'
Generated reference output for 'googletypes_response'
Failed to generate plugin output for 'googletypes_response'
Failed to generate plugin (pydantic compatible) output for 'googletypes_response'
Generated reference output for 'googletypes_response_embedded'
Failed to generate plugin output for 'googletypes_response_embedded'
Failed to generate plugin (pydantic compatible) output for 'googletypes_response_embedded'
Generated reference output for 'googletypes_service_returns_empty'
Failed to generate plugin output for 'googletypes_service_returns_empty'
Failed to generate plugin (pydantic compatible) output for 'googletypes_service_returns_empty'
Generated reference output for 'googletypes_service_returns_googletype'
Failed to generate plugin output for 'googletypes_service_returns_googletype'
Failed to generate plugin (pydantic compatible) output for 'googletypes_service_returns_googletype'
Generated reference output for 'googletypes_struct'
Failed to generate plugin output for 'googletypes_struct'
Failed to generate plugin (pydantic compatible) output for 'googletypes_struct'
Generated reference output for 'googletypes_value'
Failed to generate plugin output for 'googletypes_value'
Failed to generate plugin (pydantic compatible) output for 'googletypes_value'
Generated reference output for 'import_capitalized_package'
Failed to generate plugin output for 'import_capitalized_package'
Failed to generate plugin (pydantic compatible) output for 'import_capitalized_package'
Generated reference output for 'import_child_package_from_package'
Failed to generate plugin output for 'import_child_package_from_package'
Failed to generate plugin (pydantic compatible) output for 'import_child_package_from_package'
Generated reference output for 'import_child_package_from_root'
Failed to generate plugin output for 'import_child_package_from_root'
Failed to generate plugin (pydantic compatible) output for 'import_child_package_from_root'
Generated reference output for 'import_circular_dependency'
Failed to generate plugin output for 'import_circular_dependency'
Failed to generate plugin (pydantic compatible) output for 'import_circular_dependency'
Generated reference output for 'import_cousin_package'
Failed to generate plugin output for 'import_cousin_package'
Failed to generate plugin (pydantic compatible) output for 'import_cousin_package'
Generated reference output for 'import_cousin_package_same_name'
Failed to generate plugin output for 'import_cousin_package_same_name'
Failed to generate plugin (pydantic compatible) output for 'import_cousin_package_same_name'
Generated reference output for 'import_packages_same_name'
Failed to generate plugin output for 'import_packages_same_name'
Failed to generate plugin (pydantic compatible) output for 'import_packages_same_name'
Generated reference output for 'import_parent_package_from_child'
Failed to generate plugin output for 'import_parent_package_from_child'
Failed to generate plugin (pydantic compatible) output for 'import_parent_package_from_child'
Generated reference output for 'import_root_package_from_child'
Failed to generate plugin output for 'import_root_package_from_child'
Failed to generate plugin (pydantic compatible) output for 'import_root_package_from_child'
Generated reference output for 'import_root_sibling'
Failed to generate plugin output for 'import_root_sibling'
Failed to generate plugin (pydantic compatible) output for 'import_root_sibling'
Generated reference output for 'import_service_input_message'
Failed to generate plugin output for 'import_service_input_message'
Failed to generate plugin (pydantic compatible) output for 'import_service_input_message'
Generated reference output for 'int32'
Failed to generate plugin output for 'int32'
Failed to generate plugin (pydantic compatible) output for 'int32'
Generated reference output for 'map'
Failed to generate plugin output for 'map'
Failed to generate plugin (pydantic compatible) output for 'map'
Generated reference output for 'mapmessage'
Failed to generate plugin output for 'mapmessage'
Failed to generate plugin (pydantic compatible) output for 'mapmessage'
Generated reference output for 'namespace_builtin_types'
Failed to generate plugin output for 'namespace_builtin_types'
Failed to generate plugin (pydantic compatible) output for 'namespace_builtin_types'
Generated reference output for 'namespace_keywords'
Failed to generate plugin output for 'namespace_keywords'
Failed to generate plugin (pydantic compatible) output for 'namespace_keywords'
Generated reference output for 'nested'
Failed to generate plugin output for 'nested'
Failed to generate plugin (pydantic compatible) output for 'nested'
Generated reference output for 'nested2'
Failed to generate plugin output for 'nested2'
Failed to generate plugin (pydantic compatible) output for 'nested2'
Generated reference output for 'nestedtwice'
Failed to generate plugin output for 'nestedtwice'
Failed to generate plugin (pydantic compatible) output for 'nestedtwice'
Generated reference output for 'oneof'
Failed to generate plugin output for 'oneof'
Failed to generate plugin (pydantic compatible) output for 'oneof'
Generated reference output for 'oneof_default_value_serialization'
Failed to generate plugin output for 'oneof_default_value_serialization'
Failed to generate plugin (pydantic compatible) output for 'oneof_default_value_serialization'
Generated reference output for 'oneof_empty'
Failed to generate plugin output for 'oneof_empty'
Failed to generate plugin (pydantic compatible) output for 'oneof_empty'
Generated reference output for 'oneof_enum'
Failed to generate plugin output for 'oneof_enum'
Failed to generate plugin (pydantic compatible) output for 'oneof_enum'
Generated reference output for 'proto3_field_presence'
Failed to generate plugin output for 'proto3_field_presence'
Failed to generate plugin (pydantic compatible) output for 'proto3_field_presence'
Generated reference output for 'proto3_field_presence_oneof'
Failed to generate plugin output for 'proto3_field_presence_oneof'
Failed to generate plugin (pydantic compatible) output for 'proto3_field_presence_oneof'
Generated reference output for 'recursivemessage'
Failed to generate plugin output for 'recursivemessage'
Failed to generate plugin (pydantic compatible) output for 'recursivemessage'
Generated reference output for 'ref'
Failed to generate plugin output for 'ref'
Failed to generate plugin (pydantic compatible) output for 'ref'
Generated reference output for 'regression_387'
Failed to generate plugin output for 'regression_387'
Failed to generate plugin (pydantic compatible) output for 'regression_387'
Generated reference output for 'regression_414'
Failed to generate plugin output for 'regression_414'
Failed to generate plugin (pydantic compatible) output for 'regression_414'
Generated reference output for 'repeated'
Failed to generate plugin output for 'repeated'
Failed to generate plugin (pydantic compatible) output for 'repeated'
Generated reference output for 'repeated_duration_timestamp'
Failed to generate plugin output for 'repeated_duration_timestamp'
Failed to generate plugin (pydantic compatible) output for 'repeated_duration_timestamp'
Generated reference output for 'repeatedmessage'
Failed to generate plugin output for 'repeatedmessage'
Failed to generate plugin (pydantic compatible) output for 'repeatedmessage'
Generated reference output for 'repeatedpacked'
Failed to generate plugin output for 'repeatedpacked'
Failed to generate plugin (pydantic compatible) output for 'repeatedpacked'
Generated reference output for 'service'
Failed to generate plugin output for 'service'
Failed to generate plugin (pydantic compatible) output for 'service'
Generated reference output for 'service_separate_packages'
Failed to generate plugin output for 'service_separate_packages'
Failed to generate plugin (pydantic compatible) output for 'service_separate_packages'
Generated reference output for 'service_uppercase'
Failed to generate plugin output for 'service_uppercase'
Failed to generate plugin (pydantic compatible) output for 'service_uppercase'
Generated reference output for 'signed'
Failed to generate plugin output for 'signed'
Failed to generate plugin (pydantic compatible) output for 'signed'
Generated reference output for 'timestamp_dict_encode'
Failed to generate plugin output for 'timestamp_dict_encode'
Failed to generate plugin (pydantic compatible) output for 'timestamp_dict_encode'

Failed to generate the following test cases:
- bool
- bytes
- casing
- casing_inner_class
- casing_message_field_uppercase
- deprecated
- double
- empty_repeated
- empty_service
- entry
- enum
- example
- example_service
- field_name_identical_to_type
- fixed
- float
- google_impl_behavior_equivalence
- googletypes
- googletypes_request
- googletypes_response
- googletypes_response_embedded
- googletypes_service_returns_empty
- googletypes_service_returns_googletype
- googletypes_struct
- googletypes_value
- import_capitalized_package
- import_child_package_from_package
- import_child_package_from_root
- import_circular_dependency
- import_cousin_package
- import_cousin_package_same_name
- import_packages_same_name
- import_parent_package_from_child
- import_root_package_from_child
- import_root_sibling
- import_service_input_message
- int32
- map
- mapmessage
- namespace_builtin_types
- namespace_keywords
- nested
- nested2
- nestedtwice
- oneof
- oneof_default_value_serialization
- oneof_empty
- oneof_enum
- proto3_field_presence
- proto3_field_presence_oneof
- recursivemessage
- ref
- regression_387
- regression_414
- repeated
- repeated_duration_timestamp
- repeatedmessage
- repeatedpacked
- service
- service_separate_packages
- service_uppercase
- signed
- timestamp_dict_encode

[Gobot1234]

Did you install all the developer requirements as well?

[nickderobertis]

Ahh thank you I missed adding the -E compiler this time. Now it works! poe test is passing.

I'll go ahead and first create failing test cases for this and then work on a fix. Should I put up the PR once I have failing test cases or all together?

[Gobot1234]

I'd probably wait for both before making a pr

[nickderobertis]

Sounds good, PR is finished with both! Happy to make adjustments for any feedback but it seems pretty straightforward.