Add metadata header to all blob payload fields

[mfateev]

Currently all blob payload fields do not have any associated metadata. It works only in monolithic applications, but might cause a lot of compatibility issues when Temporal is used as an asynchronous service mesh that connects multiple services. Another problem is that migrating applications from one encoding scheme to another is practically impossible while workflows are running.
The strawman solution is to change all the binary blobs to Payload (name is up to the discussion) which includes Header field.
This would allow to specify something like: this is an encrypted field with ## version of certificate, gziped, JSON encoded.

The API proposal (Header already exists at common.proto)

message Header {
    map<string, bytes> fields = 1;
}

message Payload {
    Header header = 1;
    bytes data = 2;
}

Then, for example,ActivityTaskCompletedEventAttributes change from:

message ActivityTaskCompletedEventAttributes {
    bytes result = 1;
    int64 scheduledEventId = 2;
    int64 startedEventId = 3;
    string identity = 4;
}

to

message ActivityTaskCompletedEventAttributes {
    Payload result = 1;
    int64 scheduledEventId = 2;
    int64 startedEventId = 3;
    string identity = 4;
}

A possible alternative that potentially reduces the size of the Payload is to use an enum instead of string for metadata key names. But having that the size of payloads measures in hundreds of kilobytes I'm not sure if it is worth the complexity.

[alexshtin]

Previous issue #25.

[alexshtin]

I don't want to reuse Header as soon as they are TransportHeader essentially (perhaps we should rename it). Actually we don't need separate message at all. Let it be just:

message Payload {
    map<string,bytes> metadata = 1;
    bytes data = 2;
}

Let's also define constants for system metadata keys/values. For now we have only encoding:

const encoding = "encoding"
const encodingJson = "json"
const encodingProto = "proto"

bytes usages in proto files fall into following categories:

1. details: 18 fields (history.proto, workflow_execution.proto, decision.proto),
2. result: 6 fields (history.proto, decision.proto),
3. control: 10 fields (history.proto, decision.proto),
4. input: 10 fields (history.proto, decision.proto).
5. Also there are 2 specific cased in common.proto:

message WorkflowQuery {
    string queryType = 1;
    bytes queryArgs = 2;
}

message WorkflowQueryResult {
    enums.QueryResultType resultType = 1;
    bytes answer = 2;
    string errorMessage = 3;
}

Most usages, I've seen just pass whatever is in request to attributes or from attributes to attributes. Changing all at once shouldn't be hard.

My general plan is to convert 4 cases from list above (1,2,4,5) to Payload and convert control to string (and likely removed in future). There might be some exceptions, like details might be used as string, then it will be converted to string. Basically goal is not to have bytes at all but Payload or string.

Also we have similar message type common.DataBlob which is used 11 times. It is uses mostly internally and for rawHistory. I would convert this to Payload also.

[alexshtin]

After initial implementation, I came up with another idea. Current implementation has special case for single argument of type []byte. The idea is to extend this optimization and encode every argument separately with its own metadata. Set encoding to raw for []byte arguments.

Also add argument name as one more metadata value. It won't be used by deserializer to allow arguments rename: code will only rely on arguments sequence but not names.

Giving this, new proto structure will look like:

message Payload {
    repeated PayloadItem items = 1;
}

message PayloadItem {
    map<string,bytes> metadata = 1;
    bytes data = 2;
}

Code constants:

const encodingMetadata = "encoding"
const encodingMetadataJson = "json"
const encodingMetadataRaw = "raw"
const encodingMetadataProto = "proto"

const nameMetadata = "name"

Logically serialization of userId int, profile UserProfile, image []byte will look like:

items: [{
  metadata: {
    name: "userId",
    encoding: "json",
  },
  data: <userId value serialized using json.Marshal>,
},
{
  metadata: {
    name: "profile",
    encoding: "json",
  },
  data: <profile value serialized using json.Marshal>,
},
{
  metadata: {
    name: "image",
    encoding: "raw",
  },
 data: <image value as it is>,
}]

[alexshtin]

One interesting case is localActivityMarkerData:

localActivityMarkerData struct {
	ActivityID   string        `json:"activityId,omitempty"`
	ActivityType string        `json:"activityType,omitempty"`
	ErrReason    string        `json:"errReason,omitempty"`
	ErrJSON      string        `json:"errJson,omitempty"` // string instead of []byte so the encoded blob is human readable
	ResultJSON   string        `json:"resultJson,omitempty"`
	ReplayTime   time.Time     `json:"replayTime,omitempty"`
	Attempt      int32         `json:"attempt,omitempty"` // record attempt, starting from 0.
	Backoff      time.Duration `json:"backoff,omitempty"` // retry backoff duration
}

This struct is serialized to JSON and has JSON fields. The idea here is to convert it to:

localActivityMarkerData struct {
	ActivityID   string
	ActivityType string
	ErrReason    string
	Err          *commonpb.Payload
	Result       *commonpb.Payload
	ReplayTime   time.Time
	Attempt      int32
	Backoff      time.Duration
}

And serialize to commopb.Payload (the same as other fields) creating commonpb.PayloadItem for every field, specifying name and encoding (json) metadata. The only question is *commonpb.Payload types. I see two options:

1. Add special case for this type and call the same encoding/decoding funcs recursively. New encoding type need to be introduced: payload, proto-payload, or we can use full proto type name common.Payload. In future, proto type itself can be saved to metadata.
2. Add special case for all proto objects and serialize them to JSON using JSONPB.

First approach gives us unlimited level of enveloping and more generic and robust code.
With second approach Err and Result fields will be human readable (although JSON will look like pseudo example above and I am not sure about data field).

I would go with first option but will be glad to hear any feedback/input.

[mfateev]

Let's go with 1. Any payload should be human readable having that UI/CLI understand how to show them given metadata.

[alexshtin]

After more investigation and trying different approaches I end up with another idea. I removed Err *commonpb.Payload and Result *commonpb.Payload from the localActivityMarkerData struct and pass them as separate parameters to ToData func. This allows me not to deal with struct reflection. Custom struct deserialization is not an easy problem which I don't want to solve as part of this change.
I added another encodingType=proto and modified DataConverter.ToData and DataConverter.FromData to support any proto type. Proto types are serialized using binary proto serialization.
Giving this, marker is saved as common.Payload with 3 items:

• marker localActivityMarkerData (w/o Result and Err fields), encodingType=json,
• result *commonpb.Payload, encodingType=proto,
• err *commonpb.Payload, encodingType=proto.

This naturally fits all other Payload related changes.

[alexshtin]

Future discussions here are:

1. Replace similar types with common.Payload:

• common.Memo
• common.Header
• common.SearchAttributes
• common.DataBlob

2. In addition to custom DataConverter which knows how to convert any set of arguments to *commonpb.Payload we might want to add field/type level converters for specific encoding types, i.e. convert one argument to *commonpb.PayloadItem. This will allow customers to override only specific logic but not entire DataConverter (which is big now).