Reference schema from other subject

[treziac]

Hi,

Currently, it's not possible to reference type from an other schema, which would really be useful to manage schema across entreprise

example:
Subject "test.Product"

{
    "namespace":"test",
    "type":"record",
    "name":"Product",
    "fields":[
        {"name":"product_id","type": "long"},
        {"name":"product_name","type": "string"},
        {"name":"price","type": "float"}
    ]
}

Subject "test.Order"
reference Product

{
    "namespace":"test",
    "type":"record",
    "name" : "Order",
    "fields" : [
        {"name":"quantity","type": "int"},
        {"name":"total","type": "float"},
        {"name":"product_detail", "type": "Product"}
    ]
}

Subject "otherTest.Catalog"
reference Product

{
    "namespace":"otherTest",
    "type":"record",
    "name" : "Catalog",
    "fields" : [
        {"name":"products","type":{"type":"array", "items": "test.Product"}}
    ]
}

We should be able to have three topics to not have duplicates
A solution would be, at registration of schema, to provide a referenced subjects

POST /subjects/test/versions HTTP/1.1
Host: schemaregistry.example.com
Accept: application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json

{
    "schema" :"{
        \"namespace\":\"test\",
        \"type\":\"record\",
        \"name\" : \"Order\",
        \"fields\" : [
            {\"name\":\"quantity\",\"type\": \"int\"},
            {\"name\":\"total\",\"type\": \"float\"},
            {\"name\":\"product_detail\", \"type\": \"Product\"}
        ]
    }",
    "referencedTopics":
    [
        {
            "subject":"test.Product",
            "version":2
        },
        {
            "subject":"otherTest.Catalog",
            "version":1
        }
    ]
}

Then, at reception, for all unknown types, it would search if the type exists in the referencedTopics and would associate them with the given version.

When asking for a schema, it would concatenate all the schemas in a single one (internally, we can keep both schemas: the reference-one and the plain-one)

{
    "namespace": "test",
    "type": "record",
    "name": "Order",
    "fields": [{
        "name": "quantity",
        "type": "int"
    },
    {
        "name": "total",
        "type": "float"
    },
    {
        "name": "product_detail",
        "type": {
            "namespace": "test",
            "type": "record",
            "name": "Product",
            "fields": [{
                "name": "product_id",
                "type": "long"
            },
            {
                "name": "product_name",
                "type": "string"
            },
            {
                "name": "price",
                "type": "float"
            }]
        }
    }]
}

Compatibility is an other issue which can be solved with some rules - either have to update manually the schema with new versions in referencedTopics, or automatically with compatibility of each subject (more complicated)

[ramachpa]

We have the same situation described above by @treziac. We have also organized our Avro schemas in multiple files to enable reuse. But can't use Schema registry to resolve complex types that has references. Would be nice to have schema registry handle the references internally during registration/POST and respond back with fully built schema in response to GET.

[leifg]

So is there any way to reference other types? We currently have a currency type that would be nice to have in multiple different schemas.

[aschepis]

any movement here? would love to know if i should kick this problem down the road for a bit (if there is something coming) or determine another way of solving it permanently for my needs

[sachnk]

Ditto. We have the same problem. Any updates?

[z4f1r0v]

It is strange that schema-registry is not taking referencing across schemas into consideration. This is something native to the avro format.
Not having this feature results in a great amount of code duplication and maintenance of boilerplate classes that have to unify the replicated classes.

[steghio]

+1 same problem here when trying to create schemas for specific message types that share common fields

[rayokota]

Fixed by #1280