Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ssh tunnelling support for es-hadoop #166

Closed
mbaig opened this Issue Mar 12, 2014 · 6 comments

Comments

Projects
None yet
2 participants
@mbaig
Copy link

mbaig commented Mar 12, 2014

Our hadoop cluster connects to our prod ES cluster via ssh tunnels on localhost:9200. It seems the es-hadoop client instead of connecting to localhost, connects instead to the resolved ip of the ES shards (more accurately it connects to localhost the 1st time but switches to ips for all subsequent calls). Since those ips aren't visible I'm seeing ConnectExceptions, see here for logs https://gist.github.com/mbaig/9397119
I tried setting the config param "es.nodes.discovery" to false to force the client to only use localhost, but, this doesn't seem to be doing what I hoped.
Btw, I'm using the shapshot of es-hadoop for this.

@mbaig

This comment has been minimized.

Copy link
Author

mbaig commented Mar 13, 2014

Hi Costin,
Here the M2 and snapshot logs you requested, hopefully they'll help explain why M2 seems to support tunnelling and not the snapshots: https://gist.github.com/mbaig/9537620

costin added a commit that referenced this issue Mar 14, 2014

@costin

This comment has been minimized.

Copy link
Member

costin commented Mar 14, 2014

Hi,
I've added http proxy (not Socks) support in master. I don't think it will work with the ssh tunnel (which is socks) but just so you know.

@costin

This comment has been minimized.

Copy link
Member

costin commented Mar 14, 2014

Hi,

I added a barely tested support for SOCKS in a dedicated branch called socks. See the commit on the params needed (es.net.proxy.socks.host) and ('es.net.proxy.socks.port`). Note that authentication is not supported (and will likely not be).

Cheers,

@costin costin closed this in fb68fe1 Mar 15, 2014

@costin

This comment has been minimized.

Copy link
Member

costin commented Mar 15, 2014

Master supports now both HTTP and SOCKS proxying. See the previous comment on the settings Es Hadoop uses.

@costin costin added enhancement and removed enhancement labels Mar 15, 2014

@mbaig

This comment has been minimized.

Copy link
Author

mbaig commented Mar 18, 2014

Hi Costin: great, thanks for getting the proxy support in, I test the new stuff later today against our clusters. Btw, the 2 params for socks support, es.net.proxy.socks.host es.net.proxy.socks.port, should they be passed into the EsTap constructor via a properties obj, or set as hadoop job config params? Also, will I need to set es.nodes.discovery to false for the proxy?

@costin

This comment has been minimized.

Copy link
Member

costin commented Mar 18, 2014

I don't know where you got the idea to disable es.nodes.discovery but you should leave it as it is.
The idea of proxying is to allow transparent network communication.
As for es.net..., these are parameters just like the rest. Since you're using Cascading, it's probably easier to just pass them to the constructor since then it works in both local and hadoop mode.

costin added a commit that referenced this issue Apr 8, 2014

costin added a commit that referenced this issue Apr 8, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.