Compatibility checks with unions

[mausch]

Hi, I'm running schema registry 5.2.1 and seeing some weird behaviour around Avro compatibility checks in unions.
I'm new to Kafka and Avro so I'm not sure if this is a bug or not. Any explanation of this behaviour would be appreciated!

Starting with this schema:

Version 1 (default compatibility mode "Backward")

{
  "name": "aschema",
  "type": "record",
  "namespace": "",
  "fields": [
    {
      "name": "Message",
      "type": [
        {
          "type": "record",
          "name": "case1",
          "fields": [
            {
              "name": "field1",
              "type": "string"
            }
          ]
        }
      ]
    }
  ]
}

Then I submit this, adding fields to the union case, which the registry accepts: (version 2)

{
  "type": "record",
  "name": "aschema",
  "fields": [
    {
      "name": "Message",
      "type": [
        {
          "type": "record",
          "name": "case1",
          "fields": [
            {
              "name": "field1",
              "type": "string"
            },
            {
              "name": "field2",
              "type": "string"
            },
            {
              "name": "field3",
              "type": "string"
            }            
          ]
        }
      ]
    }
  ]
}

Then submit this, removing fields from the union case, which the registry also accepts (version 3):

{
  "type": "record",
  "name": "aschema",
  "fields": [
    {
      "name": "Message",
      "type": [
        {
          "type": "record",
          "name": "case1",
          "fields": [
            {
              "name": "field1",
              "type": "string"
            },
            {
              "name": "field3",
              "type": "string"
            }            
          ]
        }
      ]
    }
  ]
}

This looks wrong? When not using unions I get seemingly different behaviour under the same compatibility mode:

Initial:

{
  "type": "record",
  "name": "aschema",
  "fields": [
    {
      "name": "field1",
      "type": "string"
    },
    {
      "name": "field2",
      "type": "string"
    }
  ]
}

Trying to add a new field fails as it's not compatible:

{
  "type": "record",
  "name": "aschema",
  "fields": [
    {
      "name": "field1",
      "type": "string"
    },
    {
      "name": "field2",
      "type": "string"
    },
    {
      "name": "field3",
      "type": "string"
    }    
  ]
}

Removing a field succeeds though:

{
  "type": "record",
  "name": "aschema",
  "fields": [
    {
      "name": "field1",
      "type": "string"
    },
    {
      "name": "field2",
      "type": "string"
    }
  ]
}

Being new to everything here I don't know if this is something I'm misunderstanding or if it's a bug in the schema registry or a bug in Avro.
Attempting to make the same changes in a non-union schema applies the compatibility mode as expected.
Is this related somehow to https://issues.apache.org/jira/browse/AVRO-1883 ? Maybe the schema registry needs to update its version of Avro?

I've tried to work around this issue by encoding unions as a set of nullable fields instead. But nullable fields are unions too so they show the same weird/incorrect behaviour.
So in effect I can't seem to be able to do compatibility checks with unions at all!

[mausch]

There's a couple of open issues here about union compatibility checks, not sure if this is related?

#253
#391