Cannot generate code from directories with cross dependencies

[batconjurer]

Suppose I have a directory with two files: Thing.avsc and UUID.avsc with the following contents:

// Thing.avsc
{
	"name": "Thing",
	"type": "record",
	"fields": [
		{"name": "id", "type": "UUID"},
		{"name": "other", "type": "float"}
	]
}

// UUID.avsc
{
	"name": "UUID",
	"type": "record",
	"fields": [
		{"name": "bytes", "type": "bytes"}
	]
}

Code generation fails with Avro failure: Failed to parse schema: Unknown type: UUID because the current code generation does not seem to support cross depencies.

[lerouxrgd]

Indeed right now all files are expected to be standalone and cross file definition is not supported at the moment.

[lerouxrgd]

I added a note about this limitation in the README.md.

In practice though Avro schemas are better when self-contained as this is how they are written as header for big compressed files (typically stored in Hadoop). Just out of curiosity, what would be your use-case ?

A first simple solution for this would be to require the schemas in the input directory to have a naming that reflects their inter-dependencies. Something like 01_foo.avsc and 02_bar.avsc. And then just generate them in order.

A better solution would be to make a proper dependency graph (with petgraph for instance). The tricky part is that schema can be recursively nested. This is currently handled for self-contained schemas through the function deps_stacks defined as follows:

/// Utility function to find the ordered, nested dependencies of an Avro `schema`.
/// Explores nested `schema`s in a breadth-first fashion, pushing them on a stack
/// at the same time in order to have them ordered.
/// It is similar to traversing the `schema` tree in a post-order fashion.
fn deps_stack(schema: &Schema) -> Vec<&Schema> { }

In any case handling cross-files defined schemas would start from there I think.

[batconjurer]

It came up at my work. We were reworking the architecture of a monolithic piece of software to a series of microservices all talking through Kakfa. We needed to choose between protobuf or avro for the (de)serialization and our entity models have lots of cross dependencies. So the most obvious way to write it in avro didnt work out of the box ( so the decision was made to use protobuf) However, I am working on a fork of this repo now now to build up a dependency graph and resolve dependencies that way. But at the moment it would not support recursive dependencies. As you mentioned, that is much trickier.

…

On Mon, Nov 30, 2020, 1:11 AM Romain Leroux ***@***.***> wrote: I added a note about this limitation in the README.md. In practice though Avro schemas are better when self-contained as this is how they are written as header for big compressed files (typically stored in Hadoop). Just out of curiosity, what would be your use-case ? A first simple solution for this would be to require the schemas in the input directory to have a naming that reflects their inter-dependencies. Something like 01_foo.avsc and 02_bar.avsc. And then just generate them in order. A better solution would be to make a proper dependency graph (with petgraph <https://github.com/petgraph/petgraph> for instance). The tricky part is that schema can be recursively nested. This is currently handled for self-contained schemas through the function deps_stacks defined as follows: /// Utility function to find the ordered, nested dependencies of an Avro `schema`./// Explores nested `schema`s in a breadth-first fashion, pushing them on a stack/// at the same time in order to have them ordered./// It is similar to traversing the `schema` tree in a post-order fashion.fn deps_stack(schema: &Schema) -> Vec<&Schema> { } In any case handling cross-files defined schemas would start from there I think. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#6 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFICTV5344O43W53HDE727DSSLPM7ANCNFSM4T3FJXIQ> .

[lerouxrgd]

Following your PR to avro-rs released in 0.13.0 I added basic support for cross dependencies in the branch cross-deps.

Could you give it a try and let me know if that works for you ?

[batconjurer]

Wow, you got that done very quickly! Yes, it looks like it works very well.

[lerouxrgd]

Ok, nice !

I'll try to use a glob pattern instead of a directory path to be able to take into account nested dir.

After that I'll merge it to master.

[lerouxrgd]

This has been merged to master and released as 0.9.0.