Best way to handle named/dependent schemas

[edgarrmondragon]

Hi, thanks for creating and maintaining this package. I'm trying to write Avro files with schema dependencies, this is the only way I've found to write Avro files with valid schemas (I'm using Avro Viewer and rq to validate the format):

import fastavro

records = [
    {"id": 1, "bars": [{"id": 1, "baz": "abc"}, {"id": 2, "baz": "def"}]},
    {"id": 2, "bars": [{"id": 2, "baz": "uvw"}, {"id": 2, "baz": "xyz"}]},
]

# [{"name": "com.namespace.Foo", "__named_schemas": {...}, ...},
#  {"name": "com.dependencies.Bar", "__named_schemas": {...}, ...}]
schemas = fastavro.schema.load_schema("com.namespace.Foo.avsc")

# [{"name": "com.namespace.Foo", "__named_schemas": {...}, "fields": [
#    {"name": "id", "type": "int"}
#    {"name": "bars", "type": {"items": {"items": {"__named_schemas": {...}, "name": "com.dependencies.Bar"}}}}
# ]},
#  {"name": "com.dependencies.Bar", "__named_schemas": {...}}]
expanded_schema = fastavro.schema.expand_schema(schemas)

# Pick only the `Foo` schema
foo_schema = next(filter(lambda s: s["name"] == "com.namespace.Foo", expanded_schema))

with open("foo.avro", "wb") as f:
    fastavro.writer(f, foo_schema, records)

com.namespace.Foo.avsc

{
  "name": "Foo",
  "namespace": "com.namespace",
  "type": "record",
  "fields": [
    {"name": "id", "type": "int"},
    {"name": "bars", "type": {"type": "array", "items": "com.dependencies.Bar"}}
  ]
}

com.dependencies.Bar.avsc

{
  "name": "Bar",
  "namespace": "com.dependencies",
  "type": "record",
  "fields": [
    {"name": "id", "type": "int"},
    {"name": "baz", "type": "string"}
  ]
}

Avro Viewer seems to be OK with files written with schemas or foo_schema, but rq only likes foo_schema. Is this how I'm supposed to be doing this at all?

[scottbelden]

If I were doing this, I would just use schemas like so:

import fastavro

records = [
    {"id": 1, "bars": [{"id": 1, "baz": "abc"}, {"id": 2, "baz": "def"}]},
    {"id": 2, "bars": [{"id": 2, "baz": "uvw"}, {"id": 2, "baz": "xyz"}]},
]

schemas = fastavro.schema.load_schema("com.namespace.Foo.avsc")

with open("foo.avro", "wb") as f:
    fastavro.writer(f, schemas, records)

Correct me if I'm wrong, but it seems like the main problem is that rq only likes foo_schema and doesn't accept schemas as a valid schema. If so, this seems like a bug in that library. The specification states that a schema can indeed be a JSON array which represents a union of embedded types.

[scottbelden]

I took a look at rq and it looks like it internally uses avro-rs and it looks like someone has brought up the same problem that you are running into: flavray/avro-rs#147. I'm going to close this as I believe this library is doing the right thing and the problem is in avro-rs.

[edgarrmondragon]

Thank you very much @scottbelden!