Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avro Compact Format #1213

Merged
merged 5 commits into from
Jul 5, 2023
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ incubator project on [Oct 24, 2019](https://github.com/cncf/toc/pull/297).
| **Optional Specifications:** |
| AMQP Protocol Binding | [v1.0.2](https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/bindings/amqp-protocol-binding.md) | [WIP](cloudevents/bindings/amqp-protocol-binding.md) |
| AVRO Event Format | [v1.0.2](https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/formats/avro-format.md) | [WIP](cloudevents/formats/avro-format.md) |
| AVRO Compact Event Format | | [WIP](cloudevents/formats/avro-compact-format.md) |
| HTTP Protocol Binding | [v1.0.2](https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/bindings/http-protocol-binding.md) | [WIP](cloudevents/bindings/http-protocol-binding.md) |
| JSON Event Format | [v1.0.2](https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/formats/json-format.md) | [WIP](cloudevents/formats/json-format.md) |
| Kafka Protocol Binding | [v1.0.2](https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/bindings/kafka-protocol-binding.md) | [WIP](cloudevents/bindings/kafka-protocol-binding.md) |
Expand Down
5 changes: 3 additions & 2 deletions cloudevents/SDK.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ following SDKs:
- [Java SDK](https://github.com/cloudevents/sdk-java)
- [JavaScript SDK](https://github.com/cloudevents/sdk-javascript)
- [PHP SDK](https://github.com/cloudevents/sdk-php)
- [PowerShell](https://github.com/cloudevents/sdk-powershell)
- [PowerShell SDK](https://github.com/cloudevents/sdk-powershell)
- [Python SDK](https://github.com/cloudevents/sdk-python)
- [Ruby SDK](https://github.com/cloudevents/sdk-ruby)
- [Rust SDK](https://github.com/cloudevents/sdk-rust)
Expand Down Expand Up @@ -168,11 +168,12 @@ Undo it when done:
-->

| Feature | C# | Go | Java | JS | PHP | PS | Python | Ruby | Rust |
| :------ | :-: | :-: | :--: | :-: | :-: | :-: | :----: | :--: | :--: |
| :------ | :-: | :-: | :--: | :-: | :-: | :-: | :----: | :--: | :--: |
| **[v1.0](https://github.com/cloudevents/spec/tree/v1.0)** |
| [CloudEvents Core](https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/spec.md) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| Event Formats |
| [Avro](https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/formats/avro-format.md) | :heavy_check_mark: | | :x: | :x: | | | | :x: | :x: |
| [Avro Compact](https://github.com/cloudevents/spec/blob/main/cloudevents/formats/avro-compact-format.md) | :heavy_check_mark: | | :x: | :x: | | | | | :x: |
| [JSON](https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/formats/json-format.md) | :heavy_check_mark: | | :heavy_check_mark: | :heavy_check_mark: | | | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| [Protobuf ](https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/formats/protobuf-format.md) | :heavy_check_mark: | | :heavy_check_mark: | :x: | | | | :x: | :x: |
| Bindings / Content Modes |
Expand Down
109 changes: 109 additions & 0 deletions cloudevents/formats/avro-compact-format.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
# Avro Compact Event Format for CloudEvents - Version 1.0.3-wip

## Abstract

The Avro Compact Format for CloudEvents defines how events are expressed in
the [Avro 1.9.0 Specification][avro-spec].

This differs from the [Avro format](avro-format.md) in that:

- It is optimized for performance, preferring a more compact representation.
- It only supports spec version 1.0 (any changes to spec version requires changes to the Avro schema,
which changes the fingerprint, breaking compatibility).
- It does not natively support JSON (JSON can be straight-forwardly serialized
to bytes and this was therefore not considered necessary).

## Table of Contents

1. [Introduction](#1-introduction)
2. [Attributes](#2-attributes)
3. [Transport](#3-transport)
4. [Examples](#4-examples)

## 1. Introduction

[CloudEvents][ce] is a standardized and protocol-agnostic definition of the
structure and metadata description of events. This specification defines how the
CloudEvents are to be represented as [Avro 1.9.0][avro-primitives].

The [Attributes](#2-attributes) section describes the naming conventions and
data type mappings for CloudEvents attributes for use as Avro message
properties.

This specification does not define an envelope format. The Avro type system's
intent is primarily to provide a consistent type system for Avro itself and not
for message payloads.

The Avro event format does not currently define a batch mode format.

### 1.1. Conformance

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD",
"SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be
interpreted as described in [RFC2119][rfc2119].

## 2. Attributes

This section defines how CloudEvents attributes are mapped to the Avro
type-system. This specification explicitly maps each attribute.

### 2.1 Type System Mapping

The CloudEvents type system MUST be mapped to Avro types as follows.

| CloudEvents | Avro |
|---------------|------------------------------------------------------------------------|
| Boolean | [boolean][avro-primitives] |
| Integer | [int][avro-primitives] |
| String | [string][avro-primitives] |
| Binary | [bytes][avro-primitives] |
| URI | [string][avro-primitives] following [RFC 3986 §4.3][rfc3986-section43] |
| URI-reference | [string][avro-primitives] following [RFC 3986 §4.1][rfc3986-section41] |
| Timestamp | [long][avro-primitives] using `timestamp-micros` logical type |

Extension specifications MAY define secondary mapping rules for the values of
attributes they define, but MUST also include the previously defined primary
mapping.

### 2.2 Definition

Users of Avro MUST use a message whose binary encoding is identical to the one
described by the [CloudEvent Avro Compact Schema](cloudevents-compact.avsc).

## 3 Transport

Transports that support content identification MUST use the following designation:

```text
application/cloudevents+avro-compact
```

## 4 Examples

The following table shows exemplary mappings:

| CloudEvents | Type | Exemplary Avro Value |
|-----------------|--------|-------------------------------------------|
| id | string | `7a0dc520-c870-4193c8` |
| source | string | `https://github.com/cloudevents` |
| specversion | N/A | Spec version is always `1.0`. |
| type | string | `com.example.object.deleted.v2` |
| datacontenttype | string | `application/octet-stream` |
| dataschema | string | `http://registry.com/schema/v1/much.json` |
| subject | string | `mynewfile.jpg` |
| time | long | `1685121689691000` |
| data | bytes | `[bytes]` |

## References

- [Avro 1.9.0][avro-spec] Apache Avro™ 1.9.0 Specification

[avro-spec]: http://avro.apache.org/docs/1.9.0/spec.html
[avro-primitives]: http://avro.apache.org/docs/1.9.0/spec.html#schema_primitive
[avro-logical-types]: http://avro.apache.org/docs/1.9.0/spec.html#Logical+Types
[avro-unions]: http://avro.apache.org/docs/1.9.0/spec.html#Unions
[ce]: ../spec.md
[rfc2119]: https://tools.ietf.org/html/rfc2119
[rfc3986-section41]: https://tools.ietf.org/html/rfc3986#section-4.1
[rfc3986-section43]: https://tools.ietf.org/html/rfc3986#section-4.3
[rfc3339]: https://tools.ietf.org/html/rfc3339
45 changes: 25 additions & 20 deletions cloudevents/formats/avro-format.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,22 @@

## Abstract

The Avro Format for CloudEvents defines how events attributes are expressed in
The Avro Format for CloudEvents defines how events are expressed in
the [Avro 1.9.0 Specification][avro-spec].

## Table of Contents

1. [Introduction](#1-introduction)
2. [Attributes](#2-attributes)
3. [Data](#3-data)
4. [Examples](#4-examples)
4. [Transport](#4-transport)
4. [Examples](#5-examples)

## 1. Introduction

[CloudEvents][ce] is a standardized and protocol-agnostic definition of the
structure and metadata description of events. This specification defines how the
elements defined in the CloudEvents specification are to be represented in the
[Avro 1.9.0][avro-primitives].
CloudEvents are to be represented as [Avro 1.9.0][avro-primitives].

The [Attributes](#2-attributes) section describes the naming conventions and
data type mappings for CloudEvents attributes for use as Avro message
Expand Down Expand Up @@ -45,7 +45,7 @@ type-system. This specification explicitly maps each attribute.
The CloudEvents type system MUST be mapped to Avro types as follows.

| CloudEvents | Avro |
| ------------- | ---------------------------------------------------------------------- |
|---------------|------------------------------------------------------------------------|
| Boolean | [boolean][avro-primitives] |
| Integer | [int][avro-primitives] |
| String | [string][avro-primitives] |
Expand Down Expand Up @@ -148,24 +148,29 @@ For other types (non-binary data without a `datacontenttype` attribute), the
implementation MUST translate the data value into a representation of the JSON
value using the union types described for the `data` record.

## 4 Examples
## 4 Transport

Transports that support content identification MUST use the following designation:

```text
application/cloudevents+avro
```

## 5 Examples

The following table shows exemplary mappings:

| CloudEvents | Type | Exemplary Avro Value |
| ----------- | ------ | ---------------------------------------------- |
| type | string | `"com.example.someevent"` |
| specversion | string | `"1.0"` |
| source | string | `"/mycontext"` |
| id | string | `"7a0dc520-c870-4193c8"` |
| time | string | `"2019-06-05T23:45:00Z"` |
| dataschema | string | `"http://registry.com/schema/v1/much.json"` |
| contenttype | string | `"application/json"` |
| data | bytes | `"{"much":{"wow":"json"}}"` |
| | | |
| dataschema | string | `"http://registry.com/subjects/ce/versions/1"` |
| contenttype | string | `"application/avro"` |
| data | bytes | `[avro-serialized-bytes]` |
| CloudEvents | Type | Exemplary Avro Value |
|-----------------|--------|-------------------------------------------|
| id | string | `7a0dc520-c870-4193c8` |
| source | string | `https://github.com/cloudevents` |
| specversion | string | `1.0` |
| type | string | `com.example.object.deleted.v2` |
| datacontenttype | string | `application/octet-stream` |
| dataschema | string | `http://registry.com/schema/v1/much.json` |
alexec marked this conversation as resolved.
Show resolved Hide resolved
| subject | string | `mynewfile.jpg` |
| time | long | `2019-06-05T23:45:00Z` |
alexec marked this conversation as resolved.
Show resolved Hide resolved
| data | bytes | `[bytes]` |

## References

Expand Down
82 changes: 82 additions & 0 deletions cloudevents/formats/cloudevents-compact.avsc
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
{
"namespace": "io.cloudevents.v1.avro.compact",
"type": "record",
"name": "CloudEvent",
"version": "1.0",
"doc": "Avro Compact Event Format for CloudEvents",
"fields": [
{
"name": "id",
"type": "string"
},
{
"name": "source",
"type": "string"
},
{
"name": "type",
"type": "string"
},
{
"name": "datacontenttype",
"type": [
"null",
"string"
],
"default": null
},
{
"name": "dataschema",
"type": [
"null",
"string"
],
"default": null
},
{
"name": "subject",
"type": [
"null",
"string"
],
"default": null
},
{
"name": "time",
"type": [
"null",
{
"type": "long",
"logicalType": "timestamp-micros"
}
],
"default": null
},
{
"name": "attributes",
"type": {
"type": "map",
"values": [
"null",
alexec marked this conversation as resolved.
Show resolved Hide resolved
"boolean",
"int",
{
"type": "long",
"logicalType" : "timestamp-micros"
},
"string",
"bytes"
]
},
"default": {}
},
{
"name": "data",
"type": [
"bytes",
"null"
],
"default": "null"
}
]
}
2 changes: 2 additions & 0 deletions cloudevents/languages/he/formats/avro-compact-format.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Avro Compact Event Format for CloudEvents - Version 1.0.3-wip
מסמך זה טרם תורגם. בבקשה תשתמשו [בגרסה האנגלית של המסמך](../../../formats/avro-compact-format.md) לבינתיים.
6 changes: 6 additions & 0 deletions cloudevents/languages/zh-CN/formats/avro-compact-format.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Avro Compact Event Format for CloudEvents - Version 1.0.3-wip

本文档尚未被翻译,请先阅读英文[原版文档](../../../formats/avro-compact-format.md) 。

如果您迫切地需要此文档的中文翻译,请[提交一个issue](https://github.com/cloudevents/spec/issues) ,
我们会尽快安排专人进行翻译。
2 changes: 1 addition & 1 deletion cloudevents/languages/zh-CN/primer.md
Original file line number Diff line number Diff line change
Expand Up @@ -781,4 +781,4 @@ AWS 上的很大一部分事件处理系统都在使用这种格式。
}
```

[Documentation](https://www.adobe.io/apis/cloudplatform/events/documentation.html)
[Documentation](https://developer.adobe.com/events/docs/)
2 changes: 1 addition & 1 deletion cloudevents/primer.md
Original file line number Diff line number Diff line change
Expand Up @@ -1033,4 +1033,4 @@ of this format.
}
```

[Documentation](https://www.adobe.io/apis/cloudplatform/events/documentation.html)
[Documentation](https://developer.adobe.com/events/docs/)
Loading