Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ssh tunnelling support for es-hadoop #166

Closed
mbaig opened this issue Mar 12, 2014 · 6 comments
Closed

ssh tunnelling support for es-hadoop #166

mbaig opened this issue Mar 12, 2014 · 6 comments
Labels
feature :Rest v1.3.0.M3

Comments

@mbaig
Copy link

mbaig commented Mar 12, 2014

Our hadoop cluster connects to our prod ES cluster via ssh tunnels on localhost:9200. It seems the es-hadoop client instead of connecting to localhost, connects instead to the resolved ip of the ES shards (more accurately it connects to localhost the 1st time but switches to ips for all subsequent calls). Since those ips aren't visible I'm seeing ConnectExceptions, see here for logs https://gist.github.com/mbaig/9397119
I tried setting the config param "es.nodes.discovery" to false to force the client to only use localhost, but, this doesn't seem to be doing what I hoped.
Btw, I'm using the shapshot of es-hadoop for this.
@mbaig
Copy link
Author

mbaig commented Mar 13, 2014

Hi Costin,
Here the M2 and snapshot logs you requested, hopefully they'll help explain why M2 seems to support tunnelling and not the snapshots: https://gist.github.com/mbaig/9537620

costin added a commit that referenced this issue Mar 14, 2014
@costin
Add HTTP proxy support in es-hadoop
263ebe3 
relates #166
@costin
Copy link
Member

costin commented Mar 14, 2014

Hi,
I've added http proxy (not Socks) support in master. I don't think it will work with the ssh tunnel (which is socks) but just so you know.

@costin
Copy link
Member

costin commented Mar 14, 2014

Hi,

I added a barely tested support for SOCKS in a dedicated branch called socks. See the commit on the params needed (es.net.proxy.socks.host) and ('es.net.proxy.socks.port`). Note that authentication is not supported (and will likely not be).

Cheers,

@costin costin closed this as completed in fb68fe1 Mar 15, 2014
@costin
Copy link
Member

costin commented Mar 15, 2014

Master supports now both HTTP and SOCKS proxying. See the previous comment on the settings Es Hadoop uses.

@mbaig
Copy link
Author

mbaig commented Mar 18, 2014

Hi Costin: great, thanks for getting the proxy support in, I test the new stuff later today against our clusters. Btw, the 2 params for socks support, es.net.proxy.socks.host es.net.proxy.socks.port, should they be passed into the EsTap constructor via a properties obj, or set as hadoop job config params? Also, will I need to set es.nodes.discovery to false for the proxy?

@costin
Copy link
Member

costin commented Mar 18, 2014

I don't know where you got the idea to disable es.nodes.discovery but you should leave it as it is.
The idea of proxying is to allow transparent network communication.
As for es.net..., these are parameters just like the rest. Since you're using Cascading, it's probably easier to just pass them to the constructor since then it works in both local and hadoop mode.

costin added a commit that referenced this issue Apr 8, 2014
@costin
Add HTTP proxy support in es-hadoop
acc6392 
relates #166
costin added a commit that referenced this issue Apr 8, 2014
@costin
Add SOCKS support (w/o authentication)
d46d7f5 
fix #166
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature :Rest v1.3.0.M3
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@costin @mbaig