Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat][sql] Support UUID for json and avro #21267

Merged
merged 3 commits into from Oct 9, 2023

Conversation

liangyepianzhou
Copy link
Contributor

Motivation

As https://pulsar.apache.org/docs/3.1.x/sql-overview/, Pulsar SQL is based on Trino (formerly Presto SQL), which supports UUID type. But now, the UUID field in Avro or JSON schema will be interpreted as VARCHAR.

Modifications

Support decoding UUID form AVRO or JSON schema.

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository:

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Sep 28, 2023
@liangyepianzhou
Copy link
Contributor Author

liangyepianzhou commented Sep 28, 2023

Reopen #21244

@codecov-commenter
Copy link

Codecov Report

Merging #21267 (2887ddb) into master (66271e3) will increase coverage by 0.40%.
Report is 13 commits behind head on master.
The diff coverage is 62.50%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #21267      +/-   ##
============================================
+ Coverage     72.73%   73.14%   +0.40%     
- Complexity    32465    32660     +195     
============================================
  Files          1875     1887      +12     
  Lines        139929   141010    +1081     
  Branches      15413    15650     +237     
============================================
+ Hits         101781   103136    +1355     
+ Misses        30067    29695     -372     
- Partials       8081     8179      +98     
Flag Coverage Δ
inttests 24.14% <ø> (-0.04%) ⬇️
systests 24.89% <ø> (?)
unittests 72.40% <62.50%> (-0.10%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
...ql/presto/decoder/json/PulsarJsonFieldDecoder.java 63.54% <ø> (+0.52%) ⬆️
...l/presto/decoder/avro/PulsarAvroColumnDecoder.java 60.82% <50.00%> (+0.40%) ⬆️
...esto/decoder/avro/PulsarAvroRowDecoderFactory.java 85.71% <66.66%> (-0.86%) ⬇️
...esto/decoder/json/PulsarJsonRowDecoderFactory.java 75.36% <66.66%> (-0.40%) ⬇️

... and 173 files with indirect coverage changes

@liangyepianzhou liangyepianzhou merged commit 8c70943 into apache:master Oct 9, 2023
45 checks passed
@liangyepianzhou liangyepianzhou deleted the SQL-uuid branch October 9, 2023 04:07
liangyuanpeng pushed a commit to liangyuanpeng/pulsar that referenced this pull request Oct 11, 2023
### Motivation
As https://pulsar.apache.org/docs/3.1.x/sql-overview/, Pulsar SQL is based on [Trino (formerly Presto SQL)](https://trino.io/), which supports UUID type. But now, the UUID field in Avro or JSON schema will be interpreted as VARCHAR.

### Modifications

Support decoding UUID form AVRO or JSON schema.
vinayakmalik95 pushed a commit to tmdc-io/pulsar that referenced this pull request Oct 12, 2023
### Motivation
As https://pulsar.apache.org/docs/3.1.x/sql-overview/, Pulsar SQL is based on [Trino (formerly Presto SQL)](https://trino.io/), which supports UUID type. But now, the UUID field in Avro or JSON schema will be interpreted as VARCHAR.

### Modifications

Support decoding UUID form AVRO or JSON schema.
liangyepianzhou added a commit that referenced this pull request Oct 17, 2023
### Motivation
As https://pulsar.apache.org/docs/3.1.x/sql-overview/, Pulsar SQL is based on [Trino (formerly Presto SQL)](https://trino.io/), which supports UUID type. But now, the UUID field in Avro or JSON schema will be interpreted as VARCHAR.

### Modifications

Support decoding UUID form AVRO or JSON schema.

(cherry picked from commit 8c70943)
liangyepianzhou added a commit that referenced this pull request Oct 17, 2023
### Motivation
As https://pulsar.apache.org/docs/3.1.x/sql-overview/, Pulsar SQL is based on [Trino (formerly Presto SQL)](https://trino.io/), which supports UUID type. But now, the UUID field in Avro or JSON schema will be interpreted as VARCHAR.

### Modifications

Support decoding UUID form AVRO or JSON schema.

(cherry picked from commit 8c70943)
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Dec 20, 2023
### Motivation
As https://pulsar.apache.org/docs/3.1.x/sql-overview/, Pulsar SQL is based on [Trino (formerly Presto SQL)](https://trino.io/), which supports UUID type. But now, the UUID field in Avro or JSON schema will be interpreted as VARCHAR.

### Modifications

Support decoding UUID form AVRO or JSON schema.

(cherry picked from commit 8c70943)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Dec 20, 2023
### Motivation
As https://pulsar.apache.org/docs/3.1.x/sql-overview/, Pulsar SQL is based on [Trino (formerly Presto SQL)](https://trino.io/), which supports UUID type. But now, the UUID field in Avro or JSON schema will be interpreted as VARCHAR.

### Modifications

Support decoding UUID form AVRO or JSON schema.

(cherry picked from commit 8c70943)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants