Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix][io] add protobuf ByteString to pulsar-io jdbc core #20259

Merged

Conversation

bpereto
Copy link
Contributor

@bpereto bpereto commented May 8, 2023

  • tested with jdbc clickhouse on pulsar 2.11.1

Fixes #20247

Motivation

Modifications

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository:

@github-actions
Copy link

github-actions bot commented May 8, 2023

@bpereto Please add the following content to your PR description and select a checkbox:

- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->

<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>3.22.4</version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use the version in dependencyManagement in parent pom.

Also, cloud we use provided scope? (N.B. I don't take a considerate thought).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 we need to use provided to avoid runtime clashes
Also get the already declared version

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I set the version to ${protobuf3.version} from pulsar/pom.xml and added the provided scope.
Is this what you ment by dependencyManagement ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bpereto Just remove the <version> line. We already import:

      <dependency>
        <groupId>com.google.protobuf</groupId>
        <artifactId>protobuf-bom</artifactId>
        <version>${protobuf3.version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bpereto Also, please verify that provided statisfy your use case. One downside of provided is that - I'm not sure - if we don't pack protobuf into the NARball, it's possible to meet NoClassDefFoundError.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will test it with data, currently the sink is starting with the proposed changes.

Copy link
Contributor

@nicoloboschi nicoloboschi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@bpereto
Copy link
Contributor Author

bpereto commented May 9, 2023

Perfect, currently the sink is starting, need to test further with data that will be sinked.

@bpereto
Copy link
Contributor Author

bpereto commented May 9, 2023

works as intended for me.

@codecov-commenter
Copy link

Codecov Report

Merging #20259 (d3c3cd0) into master (00f17e8) will increase coverage by 35.34%.
The diff coverage is 25.00%.

Impacted file tree graph

@@              Coverage Diff              @@
##             master   #20259       +/-   ##
=============================================
+ Coverage     37.61%   72.95%   +35.34%     
- Complexity    12589    31962    +19373     
=============================================
  Files          1691     1868      +177     
  Lines        129028   138588     +9560     
  Branches      14066    15237     +1171     
=============================================
+ Hits          48530   101107    +52577     
+ Misses        74183    29444    -44739     
- Partials       6315     8037     +1722     
Flag Coverage Δ
inttests 24.14% <25.00%> (-0.04%) ⬇️
systests 24.74% <25.00%> (-0.03%) ⬇️
unittests 72.24% <25.00%> (+39.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
.../apache/pulsar/io/jdbc/BaseJdbcAutoSchemaSink.java 72.85% <0.00%> (+72.85%) ⬆️
.../apache/bookkeeper/mledger/impl/MetaStoreImpl.java 85.91% <100.00%> (+44.01%) ⬆️

... and 1424 files with indirect coverage changes

@tisonkun
Copy link
Member

Thanks for your verification! Merging...

@tisonkun tisonkun merged commit 849cbf3 into apache:master May 10, 2023
41 checks passed
@tisonkun
Copy link
Member

BTW, I notice another PR of yours #14584 that we may make progress if you review the comments and rebase on master.

@poorbarcode poorbarcode added this to the 3.1.0 milestone May 30, 2023
poorbarcode pushed a commit that referenced this pull request May 30, 2023
Signed-off-by: tison <wander4096@gmail.com>
Co-authored-by: tison <wander4096@gmail.com>
(cherry picked from commit 849cbf3)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] Pulsar IO JDBC Clickhouse and Protobuf native with "ByteString" throws exception
6 participants