Re-evaluate Avro logical type union

[tuliren]

Tell us about the problem you're trying to solve

• When converting a Json built-in format to Avro logical type, our Json to Avro schema converter always generate a nullable union of the logical type and string. This is necessary because the incoming Json built-in format can often be invalid (because they are just strings and there is usually no automatic validation). These invalid values can fail the conversion code. In that case, the object converter will fall back to a string.
• This rule can cause problems for some users on the destination side. One example: https://github.com/airbytehq/oncall/issues/643 We may want to re-evaluate this rule to fix the problem for those users.

Code reference

• Generation of the union with string for logical types in the schema converter
  • https://github.com/airbytehq/airbyte/blob/v0.40.7/airbyte-integrations/connectors/destination-s3/src/main/java/io/airbyte/integrations/destination/s3/avro/JsonToAvroSchemaConverter.java#L447-L455
• Falling back to a string when the Json built-in format is invalid
  • https://github.com/airbytehq/json-avro-converter/blob/master/converter/src/main/java/tech/allegro/schema/json2avro/converter/JsonGenericRecordReader.java#L174

Solution

• This is especially a problem for BigQuery. Other destinations look fine with this union. So the solution can be to avoid creating the union for logical types in the schema converter.
  • Related issue: https://github.com/airbytehq/oncall/issues/784

[isabelamaro-hotmart]

Hi
Any update on this?
I am facing this error syncing Jira > S3
2023-01-04 19:27:03 ERROR c.n.s.DateTimeValidator(tryParse):82 - Invalid date-time: Invalid timezone offset: -0300
Example date: "2022-12-29T16:04:29.627-0300"

[ethagnawl]

Did you figure out a workaround, @isabelamaro-hotmart?

[edgao]

closing as duplicate of #14028.