Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

changefeedccl: support remaining simple types in Avro #34417

Closed
rolandcrosby opened this issue Jan 30, 2019 · 0 comments

Comments

@rolandcrosby
Copy link
Collaborator

commented Jan 30, 2019

#32472 tracks the work to support time-related types (timestamp/timestamptz, date, time, and interval) in Avro. I'm going to open separate issues for Avro support of JSONB and ARRAY. That just leaves the following types as unsupported:

  • uuid
  • inet

uuid
The Avro docs don't reference an official convention for UUID values, but according to this issue, UUIDs are actually implemented in Avro as a string type with a logicalType of uuid. We should do the same.

inet
The CockroachDB/Postgres inet type stores an IPv4 or IPv6 address, optionally including a netmask. Absent an Avro convention for this sort of value, I recommend just representing these as strings in Avro.

@rolandcrosby rolandcrosby added the A-cdc label Jan 30, 2019

@danhhz danhhz added this to 2.2 in CDC Feb 5, 2019

danhhz added a commit to danhhz/cockroach that referenced this issue Feb 14, 2019
changefeedccl: support more columns types with avro
This adds support for sql DATE, TIME, UUID, INET, and JSONB columns with
the `experimental_avro` CHANGEFEED format.

It also adds a set of "golden" tests for what avro schema each of our
sql column types map to.

INTERVAL has a corresponding avro logical type, but is left
unimplemented for now because our INTERVALs have a larger domain of
values than avro's: 8 bytes vs 4.

ARRAY, BIT, and COLLATEDSTRING are a little tricker, so we'll wait for
customer demand. These are expected to be safe enough to backport if we
need them.

Closes cockroachdb#32472
Closes cockroachdb#34421
Closes cockroachdb#34417

Release note (enterprise change): the `CHANGEFEED` `experimental_avro`
format now supports sql columns of type DATE, TIME, UUID, INET, and
JSONB.
danhhz added a commit to danhhz/cockroach that referenced this issue Feb 14, 2019
changefeedccl: support more columns types with avro
This adds support for sql DATE, TIME, UUID, INET, and JSONB columns with
the `experimental_avro` CHANGEFEED format.

It also adds a set of "golden" tests for what avro schema each of our
sql column types map to.

INTERVAL has a corresponding avro logical type, but is left
unimplemented for now because our INTERVALs have a larger domain of
values than avro's: 8 bytes vs 4.

ARRAY, BIT, and COLLATEDSTRING are a little tricker, so we'll wait for
customer demand. These are expected to be safe enough to backport if we
need them.

Closes cockroachdb#32472
Closes cockroachdb#34421
Closes cockroachdb#34417

Release note (enterprise change): the `CHANGEFEED` `experimental_avro`
format now supports sql columns of type DATE, TIME, UUID, INET, and
JSONB.
danhhz added a commit to danhhz/cockroach that referenced this issue Feb 14, 2019
changefeedccl: support more columns types with avro
This adds support for sql DATE, TIME, UUID, INET, and JSONB columns with
the `experimental_avro` CHANGEFEED format.

It also adds a set of "golden" tests for what avro schema each of our
sql column types map to.

INTERVAL has a corresponding avro logical type, but is left
unimplemented for now because our INTERVALs have a larger domain of
values than avro's: 8 bytes vs 4.

ARRAY, BIT, and COLLATEDSTRING are a little tricker, so we'll wait for
customer demand. These are expected to be safe enough to backport if we
need them.

Closes cockroachdb#32472
Closes cockroachdb#34421
Closes cockroachdb#34417

Release note (enterprise change): the `CHANGEFEED` `experimental_avro`
format now supports sql columns of type DATE, TIME, UUID, INET, and
JSONB.
danhhz added a commit to danhhz/cockroach that referenced this issue Feb 15, 2019
changefeedccl: support more columns types with avro
This adds support for sql DATE, TIME, UUID, INET, and JSONB columns with
the `experimental_avro` CHANGEFEED format.

It also adds a set of "golden" tests for what avro schema each of our
sql column types map to.

INTERVAL has a corresponding avro logical type, but is left
unimplemented for now because our INTERVALs have a larger domain of
values than avro's: 8 bytes vs 4.

ARRAY, BIT, and COLLATEDSTRING are a little tricker, so we'll wait for
customer demand. These are expected to be safe enough to backport if we
need them.

Closes cockroachdb#32472
Closes cockroachdb#34421
Closes cockroachdb#34417

Release note (enterprise change): the `CHANGEFEED` `experimental_avro`
format now supports sql columns of type DATE, TIME, UUID, INET, and
JSONB.
danhhz added a commit to danhhz/cockroach that referenced this issue Feb 15, 2019
changefeedccl: support more columns types with avro
This adds support for sql DATE, TIME, UUID, INET, and JSONB columns with
the `experimental_avro` CHANGEFEED format.

It also adds a set of "golden" tests for what avro schema each of our
sql column types map to. Ditto for some example values for each type.

INTERVAL has a corresponding avro logical type, but is left
unimplemented for now because our INTERVALs have a larger domain of
values than avro's: 8 bytes vs 4.

ARRAY, BIT, and COLLATEDSTRING are a little tricker, so we'll wait for
customer demand. These are expected to be safe enough to backport if we
need them.

Closes cockroachdb#32472
Closes cockroachdb#34421
Closes cockroachdb#34417

Release note (enterprise change): the `CHANGEFEED` `experimental_avro`
format now supports sql columns of type DATE, TIME, UUID, INET, and
JSONB.
craig bot pushed a commit that referenced this issue Feb 15, 2019
Merge #34918
34918: changefeedccl: support more columns types with avro r=nvanbenschoten a=danhhz

This adds support for sql DATE, TIME, UUID, INET, and JSONB columns with
the `experimental_avro` CHANGEFEED format.

It also adds a set of "golden" tests for what avro schema each of our
sql column types map to.

INTERVAL has a corresponding avro logical type, but is left
unimplemented for now because our INTERVALs have a larger domain of
values than avro's: 8 bytes vs 4.

ARRAY, BIT, and COLLATEDSTRING are a little tricker, so we'll wait for
customer demand. These are expected to be safe enough to backport if we
need them.

Closes #32472
Closes #34421
Closes #34417

Release note (enterprise change): the `CHANGEFEED` `experimental_avro`
format now supports sql columns of type DATE, TIME, UUID, INET, and
JSONB.

Co-authored-by: Daniel Harrison <daniel.harrison@gmail.com>

@craig craig bot closed this in #34918 Feb 15, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.