New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
EsTap is not working #162
Comments
Can you expand on that - what exception do you encounter? |
Hello Costin: thanks for the quick response! Ok, so the exception I'm seeing is below, I pared down everything to keep things simple for debug purposes, but, basically I'm creating an EsTap to an index/type together with an array of fields of interest, then simply outputting all data from the tap to stdout, no queries to complicate matters, no "es.query" config param either since this also doesn't seem to be working any longer. Note, if I use 1.3.0.M2, everything works as expected, but, not so with the snapshots. |
Sorry, hit Close by accident, please ignore. |
The error indicates a network error, that is es-hadoop cannot connect to your host. |
Just tried that nightly, sorry it didn't work, same exception stack(s). |
Can you turn on logging (TRACE level) in log4j.properties on Thanks! |
Hi Costin: here is the trace level log output you requested (see gist below), I had to redact some parts of the logs, so if you see something like, Received [200-OK] [], the empty [] actually was populated correctly, hope you understand. Thanks again for all your help with this and the great work you guys are doing with ES in general! |
Hi, There are several things suspicious in the logs. There's the network error but there's also the NoSuchMethodError at the There were several improvements made to cascading so I've pushed a nightly build [1] - can you please check it out once Are you available on IRC? This would make things a lot easier to debug - I'm 'costin' on #elasticsearch. Let's connect Thanks! On 3/6/2014 9:19 PM, mbaig wrote:
Costin |
By the way, the #221 build has been published. On 3/6/2014 9:19 PM, mbaig wrote:
Costin |
The id between the build plan and maven are not synchronized. #331 is the Maven number, #221 the number of the build plan. On 3/6/2014 11:03 PM, mbaig wrote:
Costin |
As for IRC, give me a ping once you get online - I'll be available for the next 1.5h or so. Cheers! On 3/6/2014 11:03 PM, mbaig wrote:
|
Hi, I've pushed some changes on a new branch - cfg-refactor. Cheers! On 3/6/2014 11:03 PM, mbaig wrote:
Costin |
Hi, Can you please try the latest build #333 ? Also please post the update logs just in case. Thanks, On 3/6/2014 11:03 PM, mbaig wrote:
Costin |
Costin: not sure what you changed, but, it looks like build #333 is working. That is, reading from ES looks good. Haven't tried writing, will do that next. |
That's good to know. Getting some logs between M2 and current master would still be useful - we can chat on IRC more if you'd like. thanks! |
Yeah, I'll definitely get you those logs. I was trying to filter the dataset for the logs using the es.query job config param, which incidentally didn't work, however, passing the filter query to the EsTap constructor did work, so that should get me over that obstacle. I'm going to deploy #333 now to our cluster for larger dataset test, fingers crossed. |
I'm deploying the new jar to our cluster, but, I just realized we upgraded our ES to 1.0.1 (successfully) last night. Will this be a problem for the es-hadoop client? |
It's not a problem. es-hadoop since M2 supports both ES 1.0 and 9.x On 3/8/2014 12:54 AM, mbaig wrote:
Costin |
The Configuration option should work in master just like on M2. Thanks! On 3/8/2014 12:39 AM, mbaig wrote:
Costin |
Hey: sorry, didn't mean to disappear on you like that, I was busy firefighting issues with our ES upgrade to 1.0.1. One of the issues I ran into was that es-hadoop nightly stopped working again, albeit due to a different issue. Let me know if I should open another issue for it, meantime I'll try to describe it here. Thanks again Costin. Oh and I still owe you those M2 logs... |
@mbaig Best to open another issue. Tunnelling is not supported by es-hadoop and I'm not sure whether it will ever be. Without a direct network connection, the parallel read/write don't make sense since there's no direct connection to each shard and thus the performance goes down the drain. |
@mbaig Those M2 logs would still be nice .... |
By the way, you could try setting up the JDK property for proxies, in particular the SOCKS one: |
Please note EsTap was/is working as expected in 1.3.0.M2, however, it seems to be broken in the last ~2 weeks of nightly builds. Also note, our usage pattern or code did not change between the release of 1.3.0.M2 and today (2014-03-05).
The text was updated successfully, but these errors were encountered: