New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

This socket has been ended by the other party #4

Closed
shnhrrsn opened this Issue Jan 9, 2018 · 6 comments

Comments

Projects
None yet
3 participants
@shnhrrsn

shnhrrsn commented Jan 9, 2018

Getting this during the heartbeat for long(ish) running workers:

1/9/2018 4:42:21 AM{ Error: This socket has been ended by the other party
1/9/2018 4:42:21 AM    at Socket.writeAfterFIN [as write] (net.js:351:12)
1/9/2018 4:42:21 AM    at Promise (/apps/api/node_modules/faktory-client/lib/client.js:164:19)
1/9/2018 4:42:21 AM    at new Promise (<anonymous>)
1/9/2018 4:42:21 AM    at Client.send (/apps/api/node_modules/faktory-client/lib/client.js:160:12)
1/9/2018 4:42:21 AM    at Client.beat (/apps/api/node_modules/faktory-client/lib/client.js:223:17)
1/9/2018 4:42:21 AM    at withConnection.c (/apps/api/node_modules/faktory-worker/lib/manager.js:112:34)
1/9/2018 4:42:21 AM    at Manager.withConnection (/apps/api/node_modules/faktory-worker/lib/manager.js:46:14)
1/9/2018 4:42:21 AM    at <anonymous> code: 'EPIPE' }

@jbielick jbielick self-assigned this Jan 11, 2018

@jbielick

This comment has been minimized.

Show comment
Hide comment
@jbielick

jbielick Jan 11, 2018

Owner

Do you know how long the process was running when this happened?
How about the number of jobs processed—or were they mostly idle?

Owner

jbielick commented Jan 11, 2018

Do you know how long the process was running when this happened?
How about the number of jobs processed—or were they mostly idle?

@shnhrrsn

This comment has been minimized.

Show comment
Hide comment
@shnhrrsn

shnhrrsn Jan 15, 2018

~500k jobs processed across about 5 different workers, timed out after 12 hours or so I believe (I didn't discover until after the fact). Based on the logs, it looked like it may have been idle for a little while before it timed out.

shnhrrsn commented Jan 15, 2018

~500k jobs processed across about 5 different workers, timed out after 12 hours or so I believe (I didn't discover until after the fact). Based on the logs, it looked like it may have been idle for a little while before it timed out.

@lypanov

This comment has been minimized.

Show comment
Hide comment
@lypanov

lypanov Jan 17, 2018

Seeing the same issue here also.

lypanov commented Jan 17, 2018

Seeing the same issue here also.

@jbielick

This comment has been minimized.

Show comment
Hide comment
@jbielick

jbielick Jan 17, 2018

Owner

This is helpful, thanks for the info.

Do you have logging or debug enabled for the workers? You don't have any heartbeat log statements do you? That would also be helpful if they were available.

I'll take a look into this.

Owner

jbielick commented Jan 17, 2018

This is helpful, thanks for the info.

Do you have logging or debug enabled for the workers? You don't have any heartbeat log statements do you? That would also be helpful if they were available.

I'll take a look into this.

@jbielick

This comment has been minimized.

Show comment
Hide comment
@jbielick

jbielick Jan 21, 2018

Owner

Did the node process die when this happened or was it just hanging around?

Owner

jbielick commented Jan 21, 2018

Did the node process die when this happened or was it just hanging around?

@jbielick

This comment has been minimized.

Show comment
Hide comment
@jbielick

jbielick Feb 6, 2018

Owner

I added some fixes to gracefully reconnect and continue working on connection interruptions. Could you try v0.7.0?
It's now published to npm.

31184d6

Please reopen if this persists.

Owner

jbielick commented Feb 6, 2018

I added some fixes to gracefully reconnect and continue working on connection interruptions. Could you try v0.7.0?
It's now published to npm.

31184d6

Please reopen if this persists.

@jbielick jbielick closed this Feb 6, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment