Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Possible failure to detect etcd lease destruction #13

Closed
connor4312 opened this issue Jun 9, 2017 · 0 comments
Closed

Possible failure to detect etcd lease destruction #13

connor4312 opened this issue Jun 9, 2017 · 0 comments
Assignees
Labels

Comments

@connor4312
Copy link
Member

I noticed this morning that etcd lost the lease used for service discovery on my local instance of our chat server. Chat's last report of the lease being lost and reestablished yesterday. Etcd was empty and when sending SIGINT (which tears down the lease in a graceful shutdown) it errored that the lease was not found in etcd. Seems like it never detected that yesterday's lease was timed out.

{"name":"chat","hostname":"hera.peet.io","pid":18186,"level":30,"err":{"message":"Lease 7587822556486125056 is expired or revoked","name":"Error","stack":"Error: Lease 7587822556486125056 is expired or revoked\n    at EtcdLeaseInvalidError (/Users/copeet/Github/chat/node_modules/etcd3/lib/src/errors.js:71:9)\n    at ClientDuplexStream.stream.on.res (/Users/copeet/Github/chat/node_modules/etcd3/lib/src/lease.js:158:33)\n    at emitOne (events.js:96:13)\n    at ClientDuplexStream.emit (events.js:188:7)\n    at readableAddChunk (_stream_readable.js:176:18)\n    at ClientDuplexStream.Readable.push (_stream_readable.js:134:10)\n    at readCallback (/Users/copeet/Github/chat/node_modules/etcd3/node_modules/grpc/src/node/src/client.js:245:14)"},"msg":"Etcd3 lease lost, attempting to re-grant","time":"2017-06-08T17:22:58.238Z","v":0}
{"name":"chat","hostname":"hera.peet.io","pid":18186,"level":30,"msg":"Ectd3 lease re-established","time":"2017-06-08T17:22:58.290Z","v":0}
^CUncaught error: EtcdError: etcdserver: requested lease not found
    at /Users/copeet/Github/chat/node_modules/etcd3/node_modules/grpc/src/node/src/client.js:434:17
Uncaught error: EtcdError: etcdserver: requested lease not found
    at /Users/copeet/Github/chat/node_modules/etcd3/node_modules/grpc/src/node/src/client.js:434:17
Failed to handle shutdown: EtcdError: etcdserver: requested lease not found
    at /Users/copeet/Github/chat/node_modules/etcd3/node_modules/grpc/src/node/src/client.js:434:17
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! chat@1.0.0 start: `tsc && node dist/index`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the chat@1.0.0 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/copeet/.npm/_logs/2017-06-09T16_09_43_535Z-debug.log
@connor4312 connor4312 added the bug label Jun 9, 2017
@connor4312 connor4312 self-assigned this Jun 9, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant