Add support of UUID values when encoding avro by shnapz · Pull Request #971 · spotify/dbeam

shnapz · 2025-03-18T02:02:56Z

Checklist for PR author(s)

Changes are covered by unit tests (no major decrease in code coverage %) and/or integration tests.
Ensure code formating (use mvn com.coveo:fmt-maven-plugin:format org.codehaus.mojo:license-maven-plugin:update-file-header)
Document any relevant additions/changes in the appropriate spot in javadocs/docs/README.

codecov · 2025-03-18T02:05:22Z

Codecov Report

Attention: Patch coverage is 94.11765% with 5 lines in your changes missing coverage. Please review.

Project coverage is 91.92%. Comparing base (e4b739f) to head (c7960a9).
Report is 7 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master     #971      +/-   ##
============================================
+ Coverage     91.67%   91.92%   +0.24%     
- Complexity      253      283      +30     
============================================
  Files            26       27       +1     
  Lines           949     1015      +66     
  Branches         71       86      +15     
============================================
+ Hits            870      933      +63     
- Misses           52       54       +2     
- Partials         27       28       +1

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

farzad-sedghi · 2025-03-18T17:59:50Z

e2e/ddl.sql

  E'\\000'::bytea AS bytes_field,
-  ARRAY['foo', 'bar']::text[] AS arr1
+  ARRAY['foo', 'bar']::text[] AS arr1,
+  ARRAY[gen_random_uuid(), gen_random_uuid()]::uuid[] AS arr2


I assume the test(s) failed because of this addition, before the actual fix in the code

dbeam-core/src/main/java/com/spotify/dbeam/avro/JdbcAvroSchema.java

charles-tan

lgtm, should be able to close #959 after these changes as well.

README.md

charles-tan · 2025-05-01T20:04:09Z

dbeam-core/src/main/java/com/spotify/dbeam/avro/JdbcAvroSchema.java

+  private static SchemaBuilder.UnionAccumulator<SchemaBuilder.NullDefault<Schema>>
+      buildAvroFieldTypeFromPGType(
+      final String columnTypeName,
+      final boolean useLogicalTypes,
+      final SchemaBuilder.BaseTypeBuilder<
+          SchemaBuilder.UnionAccumulator<SchemaBuilder.NullDefault<Schema>>>
+          field) {
+    switch (columnTypeName) {
+      case "uuid":
+        if (useLogicalTypes) {
+          return field.stringBuilder().prop("logicalType", "uuid").endString();
+        } else {


Does this handle nested array type? for instance array[array[int]]

I think it won't handle that

Do we want to handle it? If not, maybe we should be clear in docs

Good point, will add to docs

Co-authored-by: Charles Tan <ctan417@gmail.com>

Bumps [org.mariadb.jdbc:mariadb-java-client](https://github.com/mariadb-corporation/mariadb-connector-j) from 3.5.1 to 3.5.3. - [Release notes](https://github.com/mariadb-corporation/mariadb-connector-j/releases) - [Changelog](https://github.com/mariadb-corporation/mariadb-connector-j/blob/master/CHANGELOG.md) - [Commits](mariadb-corporation/mariadb-connector-j@3.5.1...3.5.3) --- updated-dependencies: - dependency-name: org.mariadb.jdbc:mariadb-java-client dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

….3 (#975) Bumps [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire) from 3.5.2 to 3.5.3. - [Release notes](https://github.com/apache/maven-surefire/releases) - [Commits](apache/maven-surefire@surefire-3.5.2...surefire-3.5.3) --- updated-dependencies: - dependency-name: org.apache.maven.plugins:maven-surefire-plugin dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [org.mockito:mockito-core](https://github.com/mockito/mockito) from 5.15.2 to 5.17.0. - [Release notes](https://github.com/mockito/mockito/releases) - [Commits](mockito/mockito@v5.15.2...v5.17.0) --- updated-dependencies: - dependency-name: org.mockito:mockito-core dependency-version: 5.17.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [com.puppycrawl.tools:checkstyle](https://github.com/checkstyle/checkstyle) from 10.21.1 to 10.23.0. - [Release notes](https://github.com/checkstyle/checkstyle/releases) - [Commits](checkstyle/checkstyle@checkstyle-10.21.1...checkstyle-10.23.0) --- updated-dependencies: - dependency-name: com.puppycrawl.tools:checkstyle dependency-version: 10.23.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [org.jacoco:jacoco-maven-plugin](https://github.com/jacoco/jacoco) from 0.8.12 to 0.8.13. - [Release notes](https://github.com/jacoco/jacoco/releases) - [Commits](jacoco/jacoco@v0.8.12...v0.8.13) --- updated-dependencies: - dependency-name: org.jacoco:jacoco-maven-plugin dependency-version: 0.8.13 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Add support of UUID column type

e3fc723

shnapz changed the title ~~Add support of UUID column type~~ Add support of UUID values when writing avro bytes Mar 18, 2025

shnapz changed the title ~~Add support of UUID values when writing avro bytes~~ Add support of UUID values when encoding avro Mar 18, 2025

Added e2e test case

2bbcbd6

farzad-sedghi reviewed Mar 18, 2025

View reviewed changes

farzad-sedghi approved these changes Mar 18, 2025

View reviewed changes

shnapz added 2 commits April 22, 2025 23:48

try unit test

3239f1e

Fix style

f2533b1

charles-tan reviewed Apr 24, 2025

View reviewed changes

dbeam-core/src/main/java/com/spotify/dbeam/avro/JdbcAvroSchema.java Outdated Show resolved Hide resolved

shnapz added 2 commits April 27, 2025 20:28

Added unit tests

3bd18b4

Fix style errors

6d5a0f6

charles-tan approved these changes Apr 28, 2025

View reviewed changes

shnapz added 2 commits May 1, 2025 15:35

Handle null arrays

f978b11

Fix

979901a

charles-tan reviewed May 1, 2025

View reviewed changes

shnapz and others added 14 commits May 2, 2025 09:54

Update README.md

0bfb1f6

Co-authored-by: Charles Tan <ctan417@gmail.com>

Fix build

ad38b12

fixing tests

473a09f

Added NullableArrayItems

22b70ed

add avro-tools verification

539ef57

try to fix it

2ece77e

try more

5261b64

fix e2e script

3eb3178

fix

f460369

fix

c24675d

fix

76e03da

try +1 more time

971a0f6

try

6a7a7d0

fix

0635c2b

shnapz and others added 14 commits May 7, 2025 20:10

fix

dee5af0

cover tests

2f625a8

fix style

d176782

Fix coverage

5542032

Fix coverage

9d1d488

fi

25b7a56

fix style

0dc87d4

Automate deploy with GHA (#946)

3b66300

fix coverage

c7960a9

shnapz merged commit 817ceb9 into master May 27, 2025
8 checks passed

shnapz deleted the akabas/fix-uuid-in-array branch May 27, 2025 18:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support of UUID values when encoding avro#971

Add support of UUID values when encoding avro#971
shnapz merged 36 commits intomasterfrom
akabas/fix-uuid-in-array

shnapz commented Mar 18, 2025

Uh oh!

codecov bot commented Mar 18, 2025 •

edited

Loading

Uh oh!

farzad-sedghi Mar 18, 2025 •

edited

Loading

Uh oh!

Uh oh!

charles-tan left a comment

Uh oh!

Uh oh!

charles-tan May 1, 2025

Uh oh!

shnapz May 2, 2025

Uh oh!

charles-tan May 2, 2025

Uh oh!

shnapz May 6, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

shnapz commented Mar 18, 2025

Checklist for PR author(s)

Uh oh!

codecov bot commented Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

farzad-sedghi Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

charles-tan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

charles-tan May 1, 2025

Choose a reason for hiding this comment

Uh oh!

shnapz May 2, 2025

Choose a reason for hiding this comment

Uh oh!

charles-tan May 2, 2025

Choose a reason for hiding this comment

Uh oh!

shnapz May 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Mar 18, 2025 •

edited

Loading

farzad-sedghi Mar 18, 2025 •

edited

Loading