Avro Compact Format #1213

alexec · 2023-06-01T15:29:48Z

No description provided.

alexec · 2023-06-01T15:32:14Z

@duglin I've had a negative contributor experience with this PR:

Fighting the DCO check. There is no need to sign every commit, just one really should enough to meet this check. This is make-work.
Merge conflicts. I've been editing files that are also edited on master branch. This is also make-work.

This has resulted in a messy PR very unlike my usual work.

duglin · 2023-06-01T18:09:12Z

@duglin I've had a negative contributor experience with this PR:
* Fighting the DCO check. There is no need to sign every commit, just one really should enough to meet this check. This is make-work.
* Merge conflicts. I've been editing files that are also edited on master branch. This is also make-work.
This has resulted in a messy PR very unlike my usual work.

Signing is a commit-level thing, so the DCO checker needs to check each commit. One option is to --amend your commit so you only have one. No one will complain (or should since I do it and no one has complained).

Not sure what to say about conflicts - that's just part of a collaborative tool like git and I don't see how someone can avoid it if there are busy files. One thing I found helpful is to rebase often. On the positive side, since we only (usually) merge PRs once a week, rebasing shouldn't be needed more than that.

alexec · 2023-06-06T15:04:16Z

I've traced the cause of the conflict. The instruction for fixing the DCO check are wrong. You cannot push using --force-with-lease once you git merge master, this corrupts the git history.

alexec · 2023-06-06T15:11:38Z

cloudevents/sdk-java#578

alexec · 2023-06-06T15:14:45Z

#1210

alexec · 2023-06-07T00:29:10Z

@duglin open questions:

Existing event formats loose type information (time is rounded to microseconds and offset lost, URIs are de. Both avro and json formats lose URI type information. Ee to limitations in the spec and limitation in JSON respectively.

It is not clear if the HTTP formats support this, but I'm not sure how they would. Say you head custom context attribute that was foo, how can you turn this into a HTTP header and back and preserve type?

Given how common the json format is, it is now de-facto URIs are not supported in custom context attributes.

Avro Turbo chooses performance over complexity. Supporting these types is <= O(9) cost (the number of CE SDKs is 9). Complex systems tend to perform poorly.

For time , the argument is not so strong. we could use an union, but it makes implementation costly, complex and buggy.

My Java implementation is 126 lines long.

json is 923 lines.
protobuf is 700 lines.
xml is 1017 lines long

It is a full order of magnitude smaller. This is achieved by keenly eschewing complexity.

Less code means less bugs. I don't want to make implementing it harder and more buggy than needed.

@pierDipi is v3x of the Java SDK backwards incompatible with v2.x? If that is the case, then I'll put this work on hold because we will not have resource to do a version upgrade anytime in the next 3-6 months.

pierDipi · 2023-06-08T15:28:26Z

@duglin open questions:

Existing event formats loose type information (time is rounded to microseconds and offset lost, URIs are de. Both avro and json formats lose URI type information. Ee to limitations in the spec and limitation in JSON respectively.

It is not clear if the HTTP formats support this, but I'm not sure how they would. Say you head custom context attribute that was foo, how can you turn this into a HTTP header and back and preserve type?

Given how common the json format is, it is now de-facto URIs are not supported in custom context attributes.

Avro Turbo chooses performance over complexity. Supporting these types is <= O(9) cost (the number of CE SDKs is 9). Complex systems tend to perform poorly.

For time , the argument is not so strong. we could use an union, but it makes implementation costly, complex and buggy.

My Java implementation is 126 lines long.

json is 923 lines.

protobuf is 700 lines.

xml is 1017 lines long

It is a full order of magnitude smaller. This is achieved by keenly eschewing complexity.

Less code means less bugs. I don't want to make implementing it harder and more buggy than needed.

@pierDipi is v3x of the Java SDK backwards incompatible with v2.x? If that is the case, then I'll put this work on hold because we will not have resource to do a version upgrade anytime in the next 3-6 months.

In v3, there will be a few breaking changes (current list of breaking changes), however, avro-turbo will be a new module, so we can backport it for a 2.x release

cloudevents/formats/avro-format.md

duglin · 2023-06-09T10:00:02Z

Existing event formats loose type information (time is rounded to microseconds and offset lost, URIs are de. Both avro and json formats lose URI type information. Ee to limitations in the spec and limitation in JSON respectively.

I think I understand how avro loses precision on "time", but how does JSON lose URI type info? Do you mean because it's just a string and there's no way to know if hello is a URI or just a random string? But, also is there a question here?

It is not clear if the HTTP formats support this, but I'm not sure how they would. Say you head custom context attribute that was foo, how can you turn this into a HTTP header and back and preserve type?

Everything is converted to a string. You need to know what the extension's spec-defined type is to know how to deserialize it back to the right type.

Given how common the json format is, it is now de-facto URIs are not supported in custom context attributes.

I'm not sure what you mean by this one.

Avro Turbo chooses performance over complexity. Supporting these types is <= O(9) cost (the number of CE SDKs is 9). Complex systems tend to perform poorly.

For time , the argument is not so strong. we could use an union, but it makes implementation costly, complex and buggy.

My Java implementation is 126 lines long.
* `json` is 923 lines.
* `protobuf` is 700 lines.
* `xml` is 1017 lines long
It is a full order of magnitude smaller. This is achieved by keenly eschewing complexity.

Less code means less bugs. I don't want to make implementing it harder and more buggy than needed.

Is there a question here? It seems to me that if people choose to use Avro then they're accepting the limitations you've mentioned to gain performance.

alexec · 2023-06-10T15:09:33Z

Thank you. You answered my questions (even if not intentionally).

URIs custom context attributes not being back to URIs is acceptable.
Time being truncated to micros is acceptable.
The main breaking change in the Java SDK appears to be the upgrade to Kafka 3 libraries.

cloudevents/formats/avro-format.md

cloudevents/formats/avro-turbo-format.md

duglin · 2023-06-14T23:17:45Z

@alexec is this one ready for final review?
Also, in the previous version you updated the SDK.md file - did you want to do that again for this one?

alexec · 2023-06-15T14:05:19Z

ready for review

duglin · 2023-06-15T17:01:03Z

Discussed on the 6/15 call and the general direction looks good.
Just a few outstanding comments that need to be dealt with, and some folks wanted a little more time to review.
Hopefully we'll vote next week.

jskeet

I agree with the suggestion to rename this to "compact" - I'm happy to do that in an extra commit in this PR if you'd like, so that we're not adding work for you.

Other than that:

One wording nitpick which can be addressed later
One question about type differentiation for String/URI/URI-Ref
One suggested renaming

cloudevents/formats/avro-turbo-format.md

jskeet · 2023-06-16T09:39:20Z

cloudevents/formats/cloudevents-turbo.avsc

+      "default": null
+    },
+    {
+      "name": "attributes",


Perhaps this should be extensionattributes or extensions? (The other values are all attributes too, apart from data.)

I don't feel strongly - @clemensv _

Changed to “extensions”

cloudevents/formats/avro-turbo-format.md

alexec · 2023-06-20T16:55:39Z

I've renamed this to "Avro Compact" because 3 of us preferred that.

duglin · 2023-06-20T16:59:33Z

Also, in the previous version you updated the SDK.md file - did you want to do that again for this one?

duglin · 2023-06-20T17:23:35Z

can you rebase? There's a conflict on the SDK.md

alexec · 2023-06-20T17:39:01Z

there is always a conflict with that file, I'll fix conflicts once approved

duglin · 2023-06-20T18:58:27Z

I fixed the rebase for you so we can have the checker do it's job

Signed-off-by: Alex Collins <alex_collins@intuit.com> ok Signed-off-by: Alex Collins <alex_collins@intuit.com> Update cloudevents-turbo.avsc Signed-off-by: Alex Collins <alexec@users.noreply.github.com> Update avro-format.md Signed-off-by: Alex Collins <alexec@users.noreply.github.com> turbo -> compact Signed-off-by: Alex Collins <alex_collins@intuit.com> turbo -> compact Signed-off-by: Alex Collins <alex_collins@intuit.com> turbo -> compact Signed-off-by: Alex Collins <alex_collins@intuit.com>

duglin · 2023-06-20T19:02:31Z

I also added a change to the primer since the Adobe docs moved. Might as well just reuse this PR. Now the only failing check is an ok one.

duglin · 2023-06-22T15:06:15Z

@alexec sorry for the late notice, but I just realized that for consistency I think you should put the new spec into: cloudevents/working-drafts dir, like we did for XML.

JemDay

Minor tweak ....

cloudevents/formats/cloudevents-compact.avsc

Signed-off-by: Alex Collins <alexec@users.noreply.github.com>

jskeet

Ping @clemensv for the naming of "attributes" vs "extensions" for the map of extension attributes. I'd personally like to see that resolved before merging. One new thing I've noticed as well around nullability... but we're getting there :)

cloudevents/formats/cloudevents-compact.avsc

Signed-off-by: Alex Collins <alexec@users.noreply.github.com>

jskeet

Thanks for being patient with us :)

duglin · 2023-06-27T14:41:54Z

@alexec can you please move the new spec into the cloudevents/working-drafts dir?

Signed-off-by: Alex Collins <alex_collins@intuit.com>

duglin · 2023-06-29T15:57:02Z

@alexec couple of bad refs now:

cloudevents/SDK.md: 'https://github.com/cloudevents/spec/blob/main/cloudevents/formats/avro-compact-format.md' was not found
cloudevents/working-drafts/avro-compact-format.md: Translation file cloudevents/languages/he/working-drafts/avro-compact-format.md does not exist
cloudevents/working-drafts/avro-compact-format.md: Translation file cloudevents/languages/zh-CN/working-drafts/avro-compact-format.md does not exist

actually, I think the first one is supposed to fail but it needs to point to "working-drafts" instead of "formats". The other 2 probably need the files moved to the working-drafts dir

Signed-off-by: Alex Collins <alex_collins@intuit.com>

duglin · 2023-07-05T13:43:46Z

@alexec thanks - merging as it was approved during last week's call.

alexec force-pushed the avro-turbo branch 2 times, most recently from e6020d6 to c74889f Compare June 1, 2023 15:31

alexec marked this pull request as ready for review June 6, 2023 14:54

duglin reviewed Jun 9, 2023

View reviewed changes

cloudevents/formats/avro-format.md Outdated Show resolved Hide resolved

duglin reviewed Jun 12, 2023

View reviewed changes

cloudevents/formats/avro-format.md Show resolved Hide resolved

duglin reviewed Jun 12, 2023

View reviewed changes

cloudevents/formats/avro-format.md Show resolved Hide resolved

clemensv reviewed Jun 14, 2023

View reviewed changes

cloudevents/formats/avro-turbo-format.md Outdated Show resolved Hide resolved

jskeet reviewed Jun 16, 2023

View reviewed changes

alexec changed the title ~~Avro Turbo~~ Avro Compact Format Jun 20, 2023

duglin force-pushed the avro-turbo branch from 30d13c1 to eb3275e Compare June 20, 2023 18:54

duglin force-pushed the avro-turbo branch from eb3275e to f70934b Compare June 20, 2023 19:00

JemDay suggested changes Jun 22, 2023

View reviewed changes

cloudevents/formats/cloudevents-compact.avsc Outdated Show resolved Hide resolved

alexec commented Jun 23, 2023

View reviewed changes

cloudevents/formats/cloudevents-compact.avsc Outdated Show resolved Hide resolved

Update cloudevents/formats/cloudevents-compact.avsc

4c63a68

Signed-off-by: Alex Collins <alexec@users.noreply.github.com>

jskeet reviewed Jun 26, 2023

View reviewed changes

cloudevents/formats/cloudevents-compact.avsc Outdated Show resolved Hide resolved

Update cloudevents-compact.avsc

b61f5d8

Signed-off-by: Alex Collins <alexec@users.noreply.github.com>

jskeet approved these changes Jun 27, 2023

View reviewed changes

move into working-drafts folder

4eabe43

Signed-off-by: Alex Collins <alex_collins@intuit.com>

ok

ff1a37d

Signed-off-by: Alex Collins <alex_collins@intuit.com>

duglin merged commit 777d0c0 into cloudevents:main Jul 5, 2023
1 of 2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avro Compact Format #1213

Avro Compact Format #1213

alexec commented Jun 1, 2023

alexec commented Jun 1, 2023 •

edited

Loading

duglin commented Jun 1, 2023

alexec commented Jun 6, 2023 •

edited

Loading

alexec commented Jun 6, 2023

alexec commented Jun 6, 2023

alexec commented Jun 7, 2023 •

edited

Loading

pierDipi commented Jun 8, 2023 •

edited

Loading

duglin commented Jun 9, 2023 •

edited

Loading

alexec commented Jun 10, 2023

duglin commented Jun 14, 2023

alexec commented Jun 15, 2023

duglin commented Jun 15, 2023

jskeet left a comment

jskeet Jun 16, 2023

alexec Jun 20, 2023

alexec Jun 27, 2023

alexec commented Jun 20, 2023

duglin commented Jun 20, 2023

duglin commented Jun 20, 2023

alexec commented Jun 20, 2023

duglin commented Jun 20, 2023

duglin commented Jun 20, 2023

duglin commented Jun 22, 2023

JemDay left a comment

jskeet left a comment

jskeet left a comment

duglin commented Jun 27, 2023

duglin commented Jun 29, 2023

duglin commented Jul 5, 2023

Avro Compact Format #1213

Avro Compact Format #1213

Conversation

alexec commented Jun 1, 2023

alexec commented Jun 1, 2023 • edited Loading

duglin commented Jun 1, 2023

alexec commented Jun 6, 2023 • edited Loading

alexec commented Jun 6, 2023

alexec commented Jun 6, 2023

alexec commented Jun 7, 2023 • edited Loading

pierDipi commented Jun 8, 2023 • edited Loading

duglin commented Jun 9, 2023 • edited Loading

alexec commented Jun 10, 2023

duglin commented Jun 14, 2023

alexec commented Jun 15, 2023

duglin commented Jun 15, 2023

jskeet left a comment

Choose a reason for hiding this comment

jskeet Jun 16, 2023

Choose a reason for hiding this comment

alexec Jun 20, 2023

Choose a reason for hiding this comment

alexec Jun 27, 2023

Choose a reason for hiding this comment

alexec commented Jun 20, 2023

duglin commented Jun 20, 2023

duglin commented Jun 20, 2023

alexec commented Jun 20, 2023

duglin commented Jun 20, 2023

duglin commented Jun 20, 2023

duglin commented Jun 22, 2023

JemDay left a comment

Choose a reason for hiding this comment

jskeet left a comment

Choose a reason for hiding this comment

jskeet left a comment

Choose a reason for hiding this comment

duglin commented Jun 27, 2023

duglin commented Jun 29, 2023

duglin commented Jul 5, 2023

alexec commented Jun 1, 2023 •

edited

Loading

alexec commented Jun 6, 2023 •

edited

Loading

alexec commented Jun 7, 2023 •

edited

Loading

pierDipi commented Jun 8, 2023 •

edited

Loading

duglin commented Jun 9, 2023 •

edited

Loading