What version of the repository-hdfs plugin are you using? Any information on where the TCP connections point to? Also what version of hadoop are you using?

Thanks

I'm using :

• hadoop 2.0.0 (cdh 4.1.2)
• repository-hdfs 2.0.0-light
• Debian Wheezy

The TCP connections are pointing to the nodes of my hadoop cluster. Here is an extract of "lsof" output for the elasticsearch process :

java    68073 elasticsearch 6747u  IPv4           15823974       0t0      TCP es08:57227->cdh4worker04:50010 (CLOSE_WAIT)
java    68073 elasticsearch 6748u  IPv4           15824908       0t0      TCP es08:57651->cdh4worker05:50010 (CLOSE_WAIT)
java    68073 elasticsearch 6749u  IPv4           15818656       0t0      TCP es08:54883->cdh4worker12:50010 (CLOSE_WAIT)

I noticed that even few hours after the last snapshot, the connections are still not closed.

Thanks for your help

I've done some quick searches and it looks like this is likely caused by Hadoop itself. For example, see this thread and this issue.
I'll try to rework the code so that the filesystem instance is disposed and created per action but if the connections are leaking, this will not help much...
http://archive.cloudera.com/cdh4/cdh/4/mr1-0.20.2+1215.releasenotes.html

@jubagarie Can you try a quick fix? Hope it forces hadoop to close the connections. After you do the backup, can you try unregistering the repository and see whether it has any effect on the number of connections opened?
This causes the plugin to close the underlying HDFS FS which is the only possible fix we can apply.

Let me know how it goes - thanks!

Unfortunately, unregistering the repository and registering it again doesn't affect the number of connections.

I can also add that my server version of hadoop-hdfs is a CDH 4.1.2 but my Elasticsearch servers are running a CDH 4.6.0. Both of them seem to include the patch created for the issue you found.

Thanks

Hmm, I'm afraid I'm not sure what else can be done on this front. Can you confirm in the ES logs that the file-system is created again when you register it again after unregistering it?
As I've mentioned the only thing we can do is close the FS faster than we are currently doing it. We are not opening or closing connections manually, rather we are just clients of Hadoop's FileSystem interface...
I'm wondering whether using concurrent_streams of 1 has any impact on the number of opened connections and thus potentially slows the growth down...

The only thing I can think of is restart the node which is clearly not ideal...

Here are the ES logs when I unregister/register the repository so it seems fine :

[2014-06-19 16:17:07,732][INFO ][repositories             ] [Turner Century] delete repository [hdfs]
[2014-06-19 16:31:56,739][INFO ][repositories             ] [Turner Century] put repository [hdfs]

I will dig on the Hadoop side.

Thanks for your time !

As a work-around, you could potentially add some firewall rules to kill connections in CLOSE_WAIT state, idle for more than X minutes.

Closing this with won't fix since there's no much we can do unfortunately...

As it seems this bug in Hadoop keeps occurring some pointers in the docs would help on how to try to fix it.

Hmmm. As a point of reference, Elasticsearch 1.1.1 + CDH 5.2.1 (Hadoop 2.5.x) here, and I don't see any CLOSE_WAIT connections in lsof after taking HDFS snapshots.

Aside: is there a recommended way to setup the light plugin with Hadoop jars (like CDH) for startup? Is ES_CLASSPATH the way to go? Might be worth a mention in the documentation, especially since on CentOS/RHEL provided files, its not obvious since the variable not in the sysconfig file or init script. Found it in the Elasticsearch shell script itself.

Oh wait, totally lying, the connections were on the master nodes:

java      13452 elasticsearch  211u     IPv6            7964297      0t0        TCP esmaster01.example.com:58187->cdh02.example.com:50010 (CLOSE_WAIT)
java      13452 elasticsearch  212u     IPv6            7964298      0t0        TCP esmaster01.example.com:52732->cdh03.example.com:50010 (CLOSE_WAIT)
java      13452 elasticsearch  213u     IPv6            7964299      0t0        TCP esmaster01.example.com:57888->cdh01.example.com:50010 (CLOSE_WAIT)

So yes, still an issue!

@bflad Sorry to heart that.
As for the classpath, ES_CLASSPATH definitely works. I'll update the readme to make this clearer.

@bflad @jubagarie you might want to try 2.1.0.rc1 since it improves the creation and closing of the FileSystem.

As there hasn't been any update, I'm closing the issue.