Implement support of Schema Registry's schema references

[FrancescoPessina]

Description

When consuming an Avro message with a schema which has a schema reference, the Avrora.decode/1 function fails with {:error, {:not_found, "type"}}.

Schema references is a new feature introduced in Confluent Platform version 5.5.0 (see here https://www.confluent.io/blog/multiple-event-types-in-the-same-kafka-topic/#avro-unions-with-schema-references)

How to reproduce

Given a schema containing a schema reference registered into the schema registry, such as:

Nested schema:

{
  "namespace": "org.schema_registry_test_app.avro",
  "type": "record",
  "name": "Result",
  "fields": [
    {
      "name": "up",
      "type": "string"
    },
    {
      "name": "down",
      "type": "string"
    }
  ]
}

Schema with reference:

{
  "namespace": "org.schema_registry_test_app.avro",
  "type": "record",
  "name": "AvroTest",
  "fields": [
    {
      "name": "id",
      "type": {
        "name": "Uuid",
        "type": "fixed",
        "size": 16
      }
    },
    {
      "name": "results",
      "type": {
        "type": "array",
        "items": "org.schema_registry_test_app.avro.Result"
      }
    }
  ]
}

Try to consume and decode a message produced with AvroTest schema.

I tried with Avrora v0.13.0.

[Strech]

@FrancescoPessina Hey, thanks for bringing it up. In short – Avrora supports Inter-Schema references, so you can refer to nested schemas.

In case when a writer has local schema files it should be possible to find it. And in the case when a reader gets the schema the nested one should be present inside the top-level schema.

I didn't read how does that new CSR functionality work yet.

As I understand the issue happens when you receive a message encoded with a schema containing the reference, which schema is not present in the message schema, right?

[FrancescoPessina]

@Strech Hello and thanks for your response.

The new CSR functionality enables the "reference" from an Avro schema to another, without "copying" it entirely into the schema.
Now CSR, when a schema with references is requested via API, returns a json like this:

{
    "subject": "org.schema_registry_test_app.avro.AvroTest",
    "version": 1,
    "id": 20,
    "references": [
        {
            "name": "org.schema_registry_test_app.avro.Result",
            "subject": "org.schema_registry_test_app.avro.Result",
            "version": 1
        }
    ],
    "schema": "{
  \"namespace\": \"org.schema_registry_test_app.avro\",
  \"type\": \"record\",
  \"name\": \"AvroTest\",
  \"fields\": [
    {
      \"name\": \"id\",
      \"type\": {
        \"name\": \"Uuid\",
        \"type\": \"fixed\",
        \"size\": 16
      }
    },
    {
      \"name\": \"results\",
      \"type\": {
        \"type\": \"array\",
        \"items\": \"org.schema_registry_test_app.avro.Result\"
      }
    }
  ]
}"
}

So when Avrora finds the items with type org.schema_registry_test_app.avro.Result fails because is not a recognized type.

I pushed a PR which solves this, pulling the reference schema from CSR and inserting it into the parent schema json.
I'm a newbie with Elixir and this is my very first contribution, so I may have made some mistakes, sorry for that.
Looking at the PR I think you could also understand better the problem and how I solved it :)

[Strech]

Closed via #50