[improve][client] Add SOCKS5 proxy support for PulsarAdmin and for PulsarClient HTTP lookups#25575
Conversation
Signed-off-by: xiaolongran <xiaolongran@tencent.com>
lhotari
left a comment
There was a problem hiding this comment.
Please also add a end-to-end test to verify that the Socks proxy can be configured in the PulsarAdminBuilder. That doesn't seem to be possible at the moment.
|
In addition, it would be useful to create a separate PR to add Socks proxy support for https serviceUrls. The Socks proxy in the Pulsar client is only used for the Pulsar binary protocol at the moment. In that case, there would be a need to specify whether the socks proxy should be used for both http lookups as well as Pulsar binary protocol connections to brokers. The Pulsar client has also http clients in failover implementations. |
Thanks lhotari. That is a great suggestion. Adding a toggle switch here to control different behaviors is indeed a better implementation approach. I will readjust and revise accordingly. |
Signed-off-by: xiaolongran <xiaolongran@tencent.com>
Signed-off-by: xiaolongran <xiaolongran@tencent.com>
Signed-off-by: xiaolongran <xiaolongran@tencent.com>
|
@lhotari Please take a look again, thanks |
Signed-off-by: xiaolongran <xiaolongran@tencent.com>
Fixes #xyz
Motivation
Historically the
PulsarClientJava client supported a SOCKS5 proxy, but the proxy waswired into the Pulsar binary protocol path only (
PulsarChannelInitializer). This led totwo gaps that made Pulsar unusable in network environments where brokers are only
reachable through a SOCKS5 jump host:
PulsarAdminhad no SOCKS5 support at all.PulsarAdminBuilderexposed nosocks5ProxyAddress(...)/socks5ProxyUsername(...)/socks5ProxyPassword(...)methods, and the underlying
AsyncHttpConnectornever configured a proxy on itsasync-http-client. Users in restricted networks therefore could not run any admin
operation (topic/tenant/namespace/cluster REST calls) against the broker.
HTTP lookups inside
PulsarClientsilently ignored the SOCKS5 proxy. When a userconfigured
PulsarClientwith anhttp(s)://serviceUrl, or when the failoverHTTP client kicked in, the
HttpClientused by the lookup service built anasync-http-client without the proxy, so HTTP lookups bypassed SOCKS5 even though the
user had explicitly asked for it. This was surprising and poorly documented.
This PR closes both gaps while preserving 100% backward compatibility with the
pre-existing
PulsarClientbehavior.Modifications
New public API:
Socks5ProxyScopeenum (pulsar-client-api).A selector that tells the builder which kinds of connections should go through the
configured SOCKS5 proxy:
BINARY_ONLY— only Pulsar binary protocol connections (pre-existingPulsarClientbehavior, and the default forClientBuilder).HTTP_ONLY— only HTTP/HTTPS traffic (HTTP lookups, failover HTTP clients, andPulsarAdminREST calls). Default forPulsarAdminBuilder.BOTH— both binary and HTTP traffic.Helper methods
appliesToBinary()/appliesToHttp()keep the call sites tidy.PulsarAdminSOCKS5 support.socks5ProxyAddress,socks5ProxyUsername,socks5ProxyPassword, andsocks5ProxyScopebuilder methods onPulsarAdminBuilder, implemented inPulsarAdminBuilderImpl.PulsarAdminBuilderImpldefaults the scope toHTTP_ONLYbecause admin trafficis HTTP-only, so a user who only configures
socks5ProxyAddress(...)gets theintuitive behavior out of the box.
AsyncHttpConnectornow builds an async-http-clientProxyServerof typeSOCKS_V5when a proxy address is set and the scope includes HTTP. OptionalBASIC auth is applied when a username is configured.
PulsarClientHTTP SOCKS5 support (backward-compatible).HttpClient(used by the HTTP lookup path and the failover HTTP clients insidePulsarClient) now wires a SOCKS5ProxyServeronto its async-http-clientexactly the same way
AsyncHttpConnectordoes.ClientBuilderstaysBINARY_ONLY, so existing users whoconfigured
socks5ProxyAddress(...)onPulsarClientcontinue to see the oldbehavior — HTTP lookups do NOT go through the proxy. Users who want the new
behavior opt in explicitly via
socks5ProxyScope(Socks5ProxyScope.HTTP_ONLY)orSocks5ProxyScope.BOTH.PulsarChannelInitializer.initSocks5IfConfig(...)now additionally checkssocks5ProxyScope.appliesToBinary(), so users can disable the proxy for thebinary protocol while keeping it for HTTP.
Config plumbing.
socks5ProxyScopeonClientConfigurationData(defaults toBINARY_ONLY, overridden toHTTP_ONLYbyPulsarAdminBuilderImpl).ClientBuilderandPulsarAdminBuilderwith full Javadocexplaining the backward-compatibility contract.
Verifying this change
This change added tests and can be verified as follows:
PulsarAdminBuilderImplTest): verify that every new builder methodon
PulsarAdminBuildercorrectly stores its value intoClientConfigurationData,including the default
HTTP_ONLYscope, address/username/password round-trips, andscope overrides.
admin endpoint, and drives a real
PulsarAdmininstance throughsocks5ProxyAddress(...). The test asserts that the admin call succeeds only whenit is routed through the proxy (the proxy increments a counter) and fails when the
proxy is taken down, proving that
PulsarAdminBuilderactually honours the SOCKS5configuration end-to-end. This covers the reviewer's request to verify that SOCKS5
"can be configured in the PulsarAdminBuilder".
PulsarChannelInitializer/ binary-protocol SOCKS5 stillpass unchanged, which is the regression signal that
BINARY_ONLYis indeed thedefault on
ClientBuilder.Manual verification: run
PulsarAdminandPulsarClientagainst a localssh -DSOCKS5 tunnel with every
Socks5ProxyScopevalue and confirm that binary/HTTPtraffic is (or is not) routed through the tunnel as documented.
Does this pull request potentially affect one of the following parts:
Highlighted public-API changes:
org.apache.pulsar.client.api.Socks5ProxyScope(public, stable).org.apache.pulsar.client.api.ClientBuilder:ClientBuilder socks5ProxyScope(Socks5ProxyScope scope)org.apache.pulsar.client.admin.PulsarAdminBuilder:PulsarAdminBuilder socks5ProxyAddress(InetSocketAddress address)PulsarAdminBuilder socks5ProxyUsername(String username)PulsarAdminBuilder socks5ProxyPassword(String password)PulsarAdminBuilder socks5ProxyScope(Socks5ProxyScope scope)All additions are purely additive. No existing signatures are removed or changed, and
the default
Socks5ProxyScope.BINARY_ONLYonClientBuilderguarantees thatPulsarClientinstances that only callsocks5ProxyAddress(...)keep the exactpre-existing behavior (binary protocol only, HTTP lookups unproxied).