proto3 JSON required field confusion

[osa1]

proto3 does not have required fields, so proto3 JSON format spec does not specify how required fields should be handled.

This is fine, but we allow serializing proto2 messages as proto3 JSON. So users do it and get confusing results. Example:

syntax = "proto2";

message MyMessage {
    optional MyOtherMessage my_message_field = 1;
}

message MyOtherMessage {
    required int32 field1 = 1 [default = 123];
    required int32 field2 = 2 [default = 456];
}

import '../test.pb.dart';

void main() {
  {
    final json = {'my_message_field': null};
    final myMessage = MyMessage()..mergeFromProto3Json(json);
    print(myMessage.myMessageField.field1);
    print(myMessage.toProto3Json());
  }

  {
    final json = {'my_message_field': {}}; // required fields missing
    final myMessage = MyMessage()..mergeFromProto3Json(json);
    print(myMessage.myMessageField.field1);
    print(myMessage.toProto3Json());
  }
}

Output:

123
{myMessageField: {}}
123
{myMessageField: {}}

Problems:

1. Deser->ser changes message from {myMessageField: null} to {myMessageField: {}}
2. Deserialization accepts missing required fields
3. Serialization omits required fields

I think (2) and (3) are consistent with binary serialization and work as intended, but (1) is potentially a problem as depending on the implementation on the receiving end it changes a message with non-existent optional field into a message with an invalid field value.

[osa1]

We have the same problem in other serializers as well. Repro for binary serialization:

import '../test.pb.dart';

void main() {
  final json = {'my_message_field': null};
  final myMessage = MyMessage()..mergeFromProto3Json(json);
  print(myMessage.writeToBuffer());
}

Prints [10, 0].

I think we need one of these:

1. Initialize required fields in makeDefault
2. Have another message initializer function that initializes required fields, use it when parsing null in proto3 JSON (makeDefaultWithRequiredFields)
3. Check field presence when serializing a message, use default value if field is required but the message does not have the value explicitly set (this may not work in some cases as presence checks are not always correct: Incorrect field presence with null JSON values #751, Presence checks (hazzers) for optional bytes fields don't work #690)

My understanding is empty message (message with empty field set) represents the default message, not the empty message. If this is right, then we don't need to do (1) as getters already work as expected. For example, getters below work as expected:

final json = {'my_message_field': null};
final myMessage = MyMessage()..mergeFromProto3Json(json);
print(myMessage.myMessageField.field1);  // 123
print(myMessage.myMessageField.getField(1)); // 456

(Not sure if bug or not: hasField1 returns false)

I don't know if (1) will interact with some of the other features. We should be careful not to initialize required fields when parsing binary messages. I think the only case where we need to initialize required fields is when we see a null in proto3 JSON, so maybe (2) is better than (1) in that it won't cause any unintentional changes elsewhere.

@sigurdm any thoughts on this issue?