Zeebe Worker payload parsing needs to be lossless

[jwulf]

Similar to #80, the Zeebe worker variable payload may contain int64 number values.

Without a user-provided Dto class to leverage #78, we can't know the variable type - but we can detect an unsafe number value if we parse with lossless-json.

An option could be to specify via configuration how variables with a number value that is unsafe are handled.

If there is no Dto provided, we could default to throwing during the parse operation. This stops the application from operating with bad data.

And we could allow the user to set a default behaviour, "coerce to BigInt" or "coerce to string".

[jwulf]

The Zeebe worker payload comes over as stringified JSON, so int64 values will be numbers to the parser.

[jwulf]

As the first step towards this, the Zeebe worker input and output (the job.variables received by the handler function, and the input to the complete function) are now parsed and stringified using the lossless-number library.

This means that int64 number values outside the safe range of a JavaScript number sent over the wire from the Zeebe broker will cause the SDK to throw an exception when it parses the stringified JSON.

This is lossless, but not predictable for consumers.

The next phase is to allow the user to specify a LosslessDto class as a decoder schema.

[jwulf]

OK, the LosslessDto for the worker variable decoding is implemented.

Typing of Zeebe Worker Variables

The variable payload in a Zeebe Worker task handler is available as an object job.variables. By default, this is of type any.

The ZBClient.createWorker() method accepts an inputVariableDto to control the parsing of number values and provide design-time type information. Passing an inputVariableDto class to a Zeebe worker is optional. If a Dto class is passed to the Zeebe Worker, it is used for two purposes:

1. To provide design-time type information on the job.variables object.
2. To specify the parsing of JSON number fields. These can potentially represent int64 values that cannot be represented accurately by the JavaScript number type. With a Dto, you can specify that a specific JSON number fields be parsed losslessly to a string or BigInt.

With no Dto specified, there is no design-time type safety. At run-time, all JSON numbers are converted to the JavaScript number type. If a variable field has a number value that cannot be safely represented using the JavaScript number type (a value greater than 2^53 -1) then an exception is thrown.

To provide a Dto, extend the LosslessDto class, like so:

class MyVariableDto extends LosslessDto {
  name!: string
  maybeAge?: number
  @Int64String
  veryBigNumber!: string
  @BigIntValue
   veryBigInteger!: bigint
}

In this case, veryBigNumber is an int64 value. It is transferred as a JSON number on the wire, but the parser will parse it into a string so that no loss of precision occurs. Similarly, veryBigInteger is a very large integer value. In this case, we direct the parser to parse this variable field as a bigint.

You can nest Dtos like this:

class MyLargerDto extends LosslessDto {
  id!: string
  @ChildDto(MyVariableDto)
  entry!: MyVariableDto
}

Typing of Custom Headers

The Zeebe worker receives custom headers as job.customHeaders. The ZBClient.createWorker() method accepts a customHeadersDto to control the behaviour of custom header parsing of number values and provide design-time type information.

This follows the same strategy as the job variables, as previously described.

[jwulf]

OK, let's make this simple.

These are the scenarios that we want to handle:

1. We do not want to have JavaScript numbers in the application that claim to represent a value but are incorrect.
2. We want to allow the consumer to specify how these values should be represented in the application, as bigint or string.
3. We want this to be consistent - in other words, we don't want a variable field that has a different type depending on its value.
4. We don't want to force the consumer to make all numbers bigint or string, only the ones that could carry an unsafe int64 value.

To accomplish these:

1. The parser should throw if it encounters an unsafe int64 value that is not mapped to either string or bigint.
2. We should accept a Dto that can declare that number fields are mapped to either string or bigint.
3. Any number that is not mapped and is safe to represent as number will be parsed to number. This means it is always an accurate number, or the application throws.
4. See 3.

So there is one type of parser, one type of Dto.

The parser, if not supplied with a Dto, will simply parse the variables using lossless-json, then iterate over the resulting object, converting all LosslessNumbers to number or throwing if they are unsafe. This requires no type information or mapping.

This will also be the last stage for the parser when supplied with a Dto. When it is supplied with a Dto, the parser will iterate over the Dto, and convert the LosslessNumber of the parsed object to the mapped type of the Dto.

A question arises here: How should the parser behave if it encounters a value that is not a LosslessNumber when a mapping has been specified?

Answer: it should throw an error, because the expectation expressed in the Dto does not meet reality, and as a consequence the application behaviour is not predictable from the code.

Final stage: the object should be iterated with the parser to convert any remaining (unmapped) LosslessNumbers to number type or throw.

A question arises here: a field may be specified in the Dto as number but in fact it has an int64 value in it. If this is detected by the presence of an unsafe number, the parser should definitely throw.

But what about when the Dto is not complete? After the application is deployed, the payload is extended and now contains an int64 field that is not in the Dto. The application gets all the variables, and an unsafe, unmapped value shows up. How should the parser behave? If it throws, it forces the application developer to either constrain the variables requested to those in the Dto, or to extend the Dto.

Answer: it should throw if it there is an unmapped, unsafe value.

[jwulf]

Next step:

Deal with Output Variable Dto and stringifying variables.