Provide a way to explicitly set fields to null

[houglum]

Context: gsutil uses apitools to create messages and send them in HTTP requests to our JSON API. In the case of PATCH requests used to change object metadata (see [1]), one must set a field's value to null to clear it (as mentioned in the patch semantics at [2]). However, this isn't possible to do for predefined message fields. For example, an Object message, which describes a GCS object's metadata, has the field cacheControl:

class Object(_messages.Message):
    ...
    cacheControl = _messages.StringField(3)
    ...

but if we attempt to set that field to None, it's treated as if the field was unset:

(Pdb) obj_metadata = apitools_messages.Object()
(Pdb) obj_metadata
<Object
 acl: []>
(Pdb) obj_metadata.cacheControl = 'public'
(Pdb) obj_metadata
<Object
 acl: []
 cacheControl: 'public'>
(Pdb) obj_metadata.cacheControl = None
(Pdb) obj_metadata
<Object
 acl: []>

... and it won't be included in the JSONified representation of the message that's sent as part of the HTTP request to our API. In order to clear these fields' values via PATCH requests, we need a way to set them to some value such that they'll be represented as null when translated to JSON.

Note: This isn't the first time we've run into this limitation, and others have run into it as well, e.g. GoogleCloudPlatform/endpoints-proto-datastore#65 (comment).

[1] https://cloud.google.com/storage/docs/json_api/v1/objects/patch
[2] https://cloud.google.com/storage/docs/json_api/v1/how-tos/performance#patch

[houglum]

Digging a bit further, I'm not certain if this issue should be filed here or in the google/apitools repository.

I've found that some fields actually encode this as expected -- our Object messages contain a separate metadata field -- this is a repeated field containing instances of a separate message type, MetadataValue, which is decorated with an apitools function called MapUnrecognizedFields. It seems that if you a supply a value of None for one of these unrecognized fields, it will be JSON-encoded as null, rather than omitting the key/value pair altogether.

As an example, when encoding an Object message containing one of these (with a key of "foo" and a value of None) to JSON, the value of null will be present, rather than omitting the attribute:

(Pdb) obj.metadata = apitools_messages.Object.MetadataValue()
(Pdb) obj.metadata.additionalProperties = []
(Pdb) obj
<Object
 acl: []
 metadata: <MetadataValue
 additionalProperties: []>>

(Pdb) obj.metadata.additionalProperties.append(apitools_messages.Object.MetadataValue.AdditionalProperty(key='foo', value=None))
(Pdb) obj
<Object
 acl: []
 metadata: <MetadataValue
 additionalProperties: [<AdditionalProperty
 key: 'foo'>]>>

(Pdb) encoding.MessageToJson(obj)
'{"metadata": {"foo": null}}'

Although this encoding module is defined within apitools here, and looks to be its own extension of protorpc's ProtoJson class, so probably outside the scope of this repository.

[houglum]

Upon further investigation, it looks like apitools has an IncludeFields method that takes care of this:
https://github.com/google/apitools/blob/2bfa257a65e335ef67afc9f135c7796f47f85535/apitools/base/py/base_api.py#L356