New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dump ends with EADDRNOTAVAIL #138

Closed
rysiekpl opened this Issue Nov 30, 2015 · 3 comments

Comments

Projects
None yet
2 participants
@rysiekpl
Copy link

rysiekpl commented Nov 30, 2015

A dump between two ES 1.7.3 nodes (not in the same cluster, of course) ends in:

Mon, 30 Nov 2015 15:19:58 GMT | sent 600 objects to destination elasticsearch, wrote 600
Mon, 30 Nov 2015 15:19:58 GMT | got 600 objects from source elasticsearch (offset: 353683)
Mon, 30 Nov 2015 15:20:01 GMT | Error Emitted => connect EADDRNOTAVAIL
Mon, 30 Nov 2015 15:20:01 GMT | Total Writes: 353683
Mon, 30 Nov 2015 15:20:01 GMT | dump ended with error (set phase)  => Error: connect EADDRNOTAVAIL

This happens consistently after transferring ~3/5 for the documents (the full index is 549339) and seems to be related with the number of connections being open.

@evantahler

This comment has been minimized.

Copy link
Collaborator

evantahler commented Nov 30, 2015

That error indicates that node cannot connect to your ES cluster. Are you running your dump from within the same cluster? Are you timing out ? Are there any proxy servers in play?

Also as a note, we do not use the elsticsearch-js client in this project.

@evantahler evantahler closed this Nov 30, 2015

@rysiekpl

This comment has been minimized.

Copy link
Author

rysiekpl commented Nov 30, 2015

There are no proxy servers in play. I am dumping from single-node cluster to a different single-node cluster, both running on the same physical machine (yet in separate docker containers), and both running ES 1.7.3. It does not seem to be a time-out, and I can re-start the dump (with --skip) right after this happens (it will then go on for ~80-100k records and die with this error again).

I recognize that you are not using elasticsearch-js in this project, and I did not assume so. However, the bug that I linked seems to be exactly what is happening here, and I hoped it might be helpful to see such a similar bug in a different ES tool, as it might help debugging and solving the issue.

@evantahler

This comment has been minimized.

Copy link
Collaborator

evantahler commented Dec 1, 2015

You actually have the option to modify how node uses maxSockets within this project via options: https://github.com/taskrabbit/elasticsearch-dump/blob/master/elasticdump.js#L24-L28. In newer versions of node (>= v3.0.0/ioJS), maxScokets's defaults have been returned to Infinity, so it is back to the OS to control socket allocation.

That said, this tool operates serially, and should only ever have 2 sockets open at once (a reader and a writer for the input and output respectively). It might be true that the OS isn't returning sockets to node to use for subsequent requests, but that seems really unlikely on any major OS.

Can you share and reproduce your docker build? What is the base OS? What version of node?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment