AVRO-3275: Clarify specification on naming

[opwvhk]

Make sure you have checked all steps below.

Jira

• My PR addresses the following Avro Jira issues and references them in the PR title. For example, "AVRO-1234: My Avro PR"
  • https://issues.apache.org/jira/browse/AVRO-3275
  • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason: it's a text change in specification

Commits

• My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain Javadoc that explain what it does

[RyanSkraba]

This LGTM! :D

I made the same suggestion on https://github.com/martin-g/avro-website/pull/7/files -- it's so much easier to review the markdown than the XML here!

[RyanSkraba]

Ugh -- this is another problem, isn't it? At our company we just ran into the inconsistent treatment of a record named "record" (a non-primitive type, OK in Java, forbidden in Python). I'll bring this up on the mailing list!

[RyanSkraba]

https://lists.apache.org/thread/vmx2ocyvt6nbrff9lsdt4plyx5foyzcb

[davydog187]

@RyanSkraba I really appreciate this writeup as I'm reimplementing schema parsing for Elixir, and struggling with the inconsistencies of how existing libraries resolve references.

Can you answer a question for me?

Imagine the following schema

{
  "name": "a",
  "namespace": "top",
  "type": "record",
  "fields": [
      {"name": "b", "type": {"name": "c", "type": "string"}}
  ]
}

Would the fullnames here be either:

1. top.a, top.b, and top.c
2. top.a, top.b, c

It's unclear to me how namespaces should propagate. It seems like 1 should be the correct answer, but unclear as most libraries seem to not enforce this

[RyanSkraba]

Hello! I think this is a good question, and probably could also be clarified in the specification.

1. Field names don't have namespaces, and
2. Unnamed types such as string don't have names and don't inherit namespaces.

So there is only one named type in your example top.a and one field name b. The attribute "name": "c" on the string looks like it is accepted, but silently dropped in the Java SDK.

Where the specification is vague: "Attributes not defined in this document are permitted as metadata, but must not affect the format of serialized data." I think it makes sense to drop metadata that are applied to primitive schemas, if we consider that the type string is an immutable singleton instance in the SDK.

[RyanSkraba]

I created AVRO-3430 for discussion on whether Java is correct to silently drop the "name": "c" attribute on the primitive string.

[davydog187]

I believe it is correct as you pointed out that it's technically metadata here. The canonical parsing form would drop it down to just the primitive.

Can you clarify where in the spec it specifies that Record fields do not have namespaces and thus do not have full names? Clearly they do not have a namespace field, but that nuance was not obvious.

Thank you so much for answering this out of band! This all seems like useful data to go back into the spec

[davydog187]

Also, if you're interested, this is the PR that brought up these questions beam-community/avro_ex#62

[davydog187]

Another question is that if field names do not have namespaces, how do field aliases come into play?

[opwvhk]

It's all about context.

A namespace is a context for (named) schemata. Named schemata (of types record/error, enum or fixed) must have a unique name within their context. This results in globally unique full names (i.e., names prefixed with the namespace and a dot).

Fields cannot share a context with schemata, because they're not schemata. The context where fields are known is their record schema. Field names must be unique there.

Aliases are alternate names for schemata and fields, and the same rules as for names apply. Aliases for schemata are interpreted in the namespace of the schema, or globally if the alias includes a namespace (i.e., it's a full name). Aliases for fields are interpreted within the record that contains the field (and like field names, cannot reference another context/record schema).

[RyanSkraba]

Can you clarify where in the spec it specifies that Record fields do not have namespaces and thus do not have full names?

I agree that it's not great, and specified "by omission" here To paraphrase: Record, enum and fixed each have a fullname composed of namespace and name... The name portion of a fullname, record field names, and enum symbols must match [an alphanumeric regex without dot] I raised AVRO-3436 because it would be simple to clarify!

[RyanSkraba]

Oh, I also agree with Oscar's interpretation : named types have namespaces because they are reused (as type references), while fields (and enum symbols) can only be used in the context of their enclosing record (or enum).

If you have a use case for fields with a namespace, I'd be interested in hearing it. Maybe we could suggest an alternative? But user@avro.apache.org or a new JIRA might get more visibility!