Skip to content

Tiered Storage: add JClouds HttpClient driver#15105

Merged
eolivelli merged 2 commits intoapache:masterfrom
eolivelli:impl/jclouds-httpclient
Apr 11, 2022
Merged

Tiered Storage: add JClouds HttpClient driver#15105
eolivelli merged 2 commits intoapache:masterfrom
eolivelli:impl/jclouds-httpclient

Conversation

@eolivelli
Copy link
Contributor

Motivation

Apache JClouds supports Apache HttpClient as library to perform HTTP class, that is better of the standard JDK HttpUrlConnection based provider.

See https://github.com/jclouds/jclouds/tree/master/drivers/apachehc

Modifications

Modifications:

  • add ApacheHCHttpCommandExecutorServiceModule module
  • add jclouds driver based on Apache HTTP Client
  • add a new ShadedJCloudsUtils that is needed to workaround some problems with the shading of Guice (AbstractModule class is problematic)
  • add system property pulsar.jclouds.use_apache_hc to disable the new driver in case of problems

Verifying this change

I have tested this manually.

Does this pull request potentially affect one of the following parts:

Tiered Storage

@eolivelli eolivelli added type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages area/tieredstorage doc-not-needed Your PR changes do not impact docs labels Apr 10, 2022
@eolivelli eolivelli self-assigned this Apr 10, 2022
@lhotari
Copy link
Member

lhotari commented Apr 10, 2022

@eolivelli please share the stack trace of the blocked thread which is the original motivation to switch to use Apace HttpComponents/HttpClient for JClouds.

@HQebupt
Copy link
Contributor

HQebupt commented Apr 11, 2022

/pulsarbot run-failure-checks

Copy link
Member

@dave2wave dave2wave left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested a build with this PR with an AWS offload test of S3 offloads:

  1. Appeared to continue to work properly.
  2. Drained the backlog very slightly faster.
    Let's evaluate the different http client drivers we use so that we can be more consistent.

@eolivelli eolivelli merged commit e34448c into apache:master Apr 11, 2022
@eolivelli eolivelli deleted the impl/jclouds-httpclient branch April 11, 2022 06:40
lhotari pushed a commit to datastax/pulsar that referenced this pull request Apr 11, 2022
aparajita89 pushed a commit to aparajita89/pulsar that referenced this pull request Apr 18, 2022
Nicklee007 pushed a commit to Nicklee007/pulsar that referenced this pull request Apr 20, 2022
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Apr 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/tieredstorage doc-not-needed Your PR changes do not impact docs type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants