Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nsqd: cannot delete topic #343

Closed
tj opened this issue May 8, 2014 · 14 comments
Closed

nsqd: cannot delete topic #343

tj opened this issue May 8, 2014 · 14 comments
Labels
question

Comments

@tj
Copy link
Contributor

tj commented May 8, 2014

Not really sure what's going on here, our primary topic is "events", and I have a bunch of unrelated nsqd nodes set up which did have "events" at one point, however it won't let me delete the topic. I get a 200 response, but it's created again immediately after even though we're not publishing to an "events" topic on any of those nodes.

nsqadmin correctly displays that in-flight and messages for those nodes are all 0, so I'm confused about it not letting me remove the topic.

Truncated screen:
screen shot 2014-05-08 at 12 31 31 pm
@mreiferson
Copy link
Member

hi @visionmedia - looks like you have consumers still connected on the ingestion channel of the events topic - both producers and consumers will cause a topic (and channel) to be created if they connect.

@tj
Copy link
Contributor Author

tj commented May 8, 2014

ohh shit, we're using nsqlookupd to locate the nodes and reconnect when disconnected, I'll try pausing our ingestion queue and shut those down before removing the topic

@tj
Copy link
Contributor Author

tj commented May 8, 2014

nevermind that won't work too well for now haha way too many services connected from before, we'd have to shut down everything to get rid of it

@mreiferson
Copy link
Member

right, the typical order to completely delete a topic would be:

  1. stop publishing to it
  2. wait for queues to empty
  3. stop consuming it
  4. delete topic

@tj
Copy link
Contributor Author

tj commented May 8, 2014

just going to launch some new nodes that'll probably be faster, thanks!

@tj tj closed this as completed May 8, 2014
@tj
Copy link
Contributor Author

tj commented Aug 21, 2014

hrrm going to re-open, this is becoming a big issue for us, some how they keep sneaking in there, our own fault of course but we everything in our cluster uses "events" so we can't really shut them all down easily just to delete a topic from a node

@tj tj reopened this Aug 21, 2014
@mreiferson
Copy link
Member

can you paste the nsqadmin and nsqlookupd logs for the time period around when you delete the topic (presumably for a single node)?

@tj
Copy link
Contributor Author

tj commented Aug 21, 2014

it's by-design that a consumer also creates the topic isn't it? Problem is right now that we've got into a bit of a mess there with 3 or 4 programs, on 100s of machines connecting to these nsqds that accidentally had the "events" topic. Seems like the only way to really fix that problem is to shut them all down, remove the topic, and boot them all back up

@mreiferson
Copy link
Member

it is by design, but you should still be able to delete a topic for a single node (the topic should be tombstoned for a short interval where it isn't returned via lookupd, connections disconnected, and then purged fully from lookupd).

I was curious to see logs to see if those things were happening...

@tj
Copy link
Contributor Author

tj commented Aug 21, 2014

ah ok I don't remember trying individual nodes, I'll try that out

@mreiferson
Copy link
Member

That would explain things... if you weren't using the topic tombstone then you would experience these symptoms. Specifically, it's the little red-x next to the nodes on a topic page:

screenshot 2014-08-21 23 39 51

For more info see: http://nsq.io/components/nsqlookupd.html#deletion_tombstones

(closing)

@tj
Copy link
Contributor Author

tj commented Aug 22, 2014

hmm, I'm just using the default tombstone value, but they're definitely re-registering immediately. Here's with the X on the node's page:

2014/08/22 05:26:15 TOPIC(events): deleting
2014/08/22 05:26:15 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:26:15 TOPIC(events): closing ... messagePump
2014/08/22 05:26:15 TOPIC(events): closing ... router
2014/08/22 05:26:15 LOOKUPD(10.0.1.171:4160): topic UNREGISTER events
2014/08/22 05:26:15 CHANNEL(archiver): deleting
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.1.123:40148] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.1.123:40148) - read tcp 10.0.1.123:40148: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.1.121:35497] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.1.121:35497) - read tcp 10.0.1.121:35497: use of closed network connection
2014/08/22 05:26:15 CHANNEL(archiver): closing ... messagePump
2014/08/22 05:26:15 CHANNEL(archiver): closing ... pqueue worker
2014/08/22 05:26:15 CHANNEL(archiver): closing ... pqueue worker
2014/08/22 05:26:15 CHANNEL(archiver): closing ... router
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.1.123:40148] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.1.121:35497] exiting messagePump
2014/08/22 05:26:15 DISKQUEUE(events:archiver): emptying
2014/08/22 05:26:15 DISKQUEUE(events:archiver): deleting
2014/08/22 05:26:15 DISKQUEUE(events:archiver): closing ... ioLoop
2014/08/22 05:26:15 CHANNEL(debugger-persistence): deleting
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.1.61:51284] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.1.61:51284) - read tcp 10.0.1.61:51284: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.1.61:51285] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.1.61:51285) - read tcp 10.0.1.61:51285: use of closed network connection
2014/08/22 05:26:15 CHANNEL(debugger-persistence): closing ... messagePump
2014/08/22 05:26:15 CHANNEL(debugger-persistence): closing ... pqueue worker
2014/08/22 05:26:15 CHANNEL(debugger-persistence): closing ... pqueue worker
2014/08/22 05:26:15 CHANNEL(debugger-persistence): closing ... router
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.1.61:51284] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.1.61:51285] exiting messagePump
2014/08/22 05:26:15 DISKQUEUE(events:debugger-persistence): emptying
2014/08/22 05:26:15 DISKQUEUE(events:debugger-persistence): deleting
2014/08/22 05:26:15 DISKQUEUE(events:debugger-persistence): closing ... ioLoop
2014/08/22 05:26:15 CHANNEL(ingestion): deleting
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.86:52940] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.86:52940) - read tcp 10.0.0.86:52940: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.32:59001] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.32:59001) - read tcp 10.0.0.32:59001: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.188:37251] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.188:37251) - read tcp 10.0.0.188:37251: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.35:59604] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.35:59604) - read tcp 10.0.0.35:59604: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.34:56435] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.34:56435) - read tcp 10.0.0.34:56435: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.69:52153] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.69:52153) - read tcp 10.0.0.69:52153: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.71:46486] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.71:46486) - read tcp 10.0.0.71:46486: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.69:52155] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.69:52155) - read tcp 10.0.0.69:52155: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.32:59005] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.32:59005) - read tcp 10.0.0.32:59005: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.36:54273] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.36:54273) - read tcp 10.0.0.36:54273: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.190:51141] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.190:51141) - read tcp 10.0.0.190:51141: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.36:54279] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.36:54279) - read tcp 10.0.0.36:54279: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.70:60736] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.70:60736) - read tcp 10.0.0.70:60736: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.34:56428] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.34:56428) - read tcp 10.0.0.34:56428: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.36:54275] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.36:54275) - read tcp 10.0.0.36:54275: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.66:58623] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.66:58623) - read tcp 10.0.0.66:58623: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.70:60735] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.70:60735) - read tcp 10.0.0.70:60735: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.219:51622] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.219:51622) - read tcp 10.0.0.219:51622: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.190:51220] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.190:51220) - read tcp 10.0.0.190:51220: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.108:57401] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.108:57401) - read tcp 10.0.0.108:57401: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.85:44239] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.85:44239) - read tcp 10.0.0.85:44239: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.68:59101] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.68:59101) - read tcp 10.0.0.68:59101: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.188:37250] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.188:37250) - read tcp 10.0.0.188:37250: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.136:49417] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.136:49417) - read tcp 10.0.0.136:49417: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.188:37244] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.188:37244) - read tcp 10.0.0.188:37244: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.219:51619] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.219:51619) - read tcp 10.0.0.219:51619: use of closed network connection
2014/08/22 05:26:15 LOOKUPD(10.0.1.172:4160): topic UNREGISTER events
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.108:57409] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.108:57409) - read tcp 10.0.0.108:57409: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.66:58625] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.66:58625) - read tcp 10.0.0.66:58625: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.35:59600] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.35:59600) - read tcp 10.0.0.35:59600: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.32:59000] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.32:59000) - read tcp 10.0.0.32:59000: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.85:44241] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.85:44241) - read tcp 10.0.0.85:44241: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.85:44242] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.85:44242) - read tcp 10.0.0.85:44242: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.86:52943] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.86:52943) - read tcp 10.0.0.86:52943: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.34:56427] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.34:56427) - read tcp 10.0.0.34:56427: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.219:51618] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.219:51618) - read tcp 10.0.0.219:51618: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.82:58966] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.82:58966) - read tcp 10.0.0.82:58966: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.67:55115] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.67:55115) - read tcp 10.0.0.67:55115: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.108:57407] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.108:57407) - read tcp 10.0.0.108:57407: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.32:59002] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.32:59002) - read tcp 10.0.0.32:59002: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.85:44247] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.85:44247) - read tcp 10.0.0.85:44247: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.67:55113] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.67:55113) - read tcp 10.0.0.67:55113: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.108:57404] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.108:57404) - read tcp 10.0.0.108:57404: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.36:54278] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.36:54278) - read tcp 10.0.0.36:54278: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.82:58968] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.82:58968) - read tcp 10.0.0.82:58968: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.190:51142] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.190:51142) - read tcp 10.0.0.190:51142: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.190:51700] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.190:51700) - read tcp 10.0.0.190:51700: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.86:52941] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.86:52941) - read tcp 10.0.0.86:52941: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.188:37249] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.188:37249) - read tcp 10.0.0.188:37249: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.136:49418] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.136:49418) - read tcp 10.0.0.136:49418: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.35:59605] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.35:59605) - read tcp 10.0.0.35:59605: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.86:52944] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.86:52944) - read tcp 10.0.0.86:52944: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.71:46515] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.71:46515) - read tcp 10.0.0.71:46515: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.219:51617] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.219:51617) - read tcp 10.0.0.219:51617: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.68:59102] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.68:59102) - read tcp 10.0.0.68:59102: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.35:59603] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.35:59603) - read tcp 10.0.0.35:59603: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.34:56434] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.34:56434) - read tcp 10.0.0.34:56434: use of closed network connection
2014/08/22 05:26:15 CHANNEL(ingestion): closing ... messagePump
2014/08/22 05:26:15 CHANNEL(ingestion): closing ... pqueue worker
2014/08/22 05:26:15 CHANNEL(ingestion): closing ... pqueue worker
2014/08/22 05:26:15 CHANNEL(ingestion): closing ... router
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.86:52940] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.32:59001] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.188:37251] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.35:59604] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.34:56435] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.69:52153] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.71:46486] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.69:52155] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.32:59005] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.36:54273] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.190:51141] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.36:54279] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.70:60736] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.34:56428] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.36:54275] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.66:58623] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.70:60735] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.219:51622] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.190:51220] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.108:57401] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.85:44239] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.68:59101] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.188:37250] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.136:49417] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.188:37244] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.219:51619] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.108:57409] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.66:58625] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.35:59600] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.32:59000] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.85:44241] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.85:44242] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.86:52943] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.34:56427] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.219:51618] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.82:58966] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.67:55115] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.108:57407] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.32:59002] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.85:44247] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.67:55113] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.108:57404] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.36:54278] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.82:58968] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.190:51142] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.190:51700] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.86:52941] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.188:37249] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.136:49418] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.35:59605] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.86:52944] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.71:46515] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.219:51617] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.68:59102] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.35:59603] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.34:56434] exiting messagePump
2014/08/22 05:26:15 DISKQUEUE(events:ingestion): emptying
2014/08/22 05:26:15 DISKQUEUE(events:ingestion): deleting
2014/08/22 05:26:15 DISKQUEUE(events:ingestion): closing ... ioLoop
2014/08/22 05:26:15 CHANNEL(nsq-router-v3#ephemeral): deleting
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.240:58632] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.240:58632) - read tcp 10.0.0.240:58632: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.182:42639] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.182:42639) - read tcp 10.0.0.182:42639: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.182:42640] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.182:42640) - read tcp 10.0.0.182:42640: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.240:58633] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.240:58633) - read tcp 10.0.0.240:58633: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.38:54559] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.38:54559) - read tcp 10.0.0.38:54559: use of closed network connection
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.38:54560] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.0.38:54560) - read tcp 10.0.0.38:54560: use of closed network connection
2014/08/22 05:26:15 CHANNEL(nsq-router-v3#ephemeral): closing ... messagePump
2014/08/22 05:26:15 CHANNEL(nsq-router-v3#ephemeral): closing ... pqueue worker
2014/08/22 05:26:15 CHANNEL(nsq-router-v3#ephemeral): closing ... pqueue worker
2014/08/22 05:26:15 CHANNEL(nsq-router-v3#ephemeral): closing ... router
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.240:58632] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.182:42639] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.182:42640] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.240:58633] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.38:54559] exiting messagePump
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.0.38:54560] exiting messagePump
2014/08/22 05:26:15 CHANNEL(queue-stats#ephemeral): deleting
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.1.248:51311] exiting ioloop
2014/08/22 05:26:15 ERROR: client(10.0.1.248:51311) - read tcp 10.0.1.248:51311: use of closed network connection
2014/08/22 05:26:15 CHANNEL(queue-stats#ephemeral): closing ... messagePump
2014/08/22 05:26:15 CHANNEL(queue-stats#ephemeral): closing ... pqueue worker
2014/08/22 05:26:15 CHANNEL(queue-stats#ephemeral): closing ... pqueue worker
2014/08/22 05:26:15 CHANNEL(queue-stats#ephemeral): closing ... router
2014/08/22 05:26:15 PROTOCOL(V2): [10.0.1.248:51311] exiting messagePump
2014/08/22 05:26:15 DISKQUEUE(events): emptying
2014/08/22 05:26:15 DISKQUEUE(events): deleting
2014/08/22 05:26:15 DISKQUEUE(events): closing ... ioLoop
2014/08/22 05:26:15 LOOKUPD(10.0.1.173:4160): topic UNREGISTER events
2014/08/22 05:26:15 LOOKUPD(10.0.1.171:4160): channel UNREGISTER events archiver
2014/08/22 05:26:15 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:26:15 LOOKUPD(10.0.1.172:4160): channel UNREGISTER events archiver
2014/08/22 05:26:15 LOOKUPD(10.0.1.173:4160): channel UNREGISTER events archiver
2014/08/22 05:26:15 LOOKUPD(10.0.1.171:4160): channel UNREGISTER events debugger-persistence
2014/08/22 05:26:15 LOOKUPD(10.0.1.172:4160): channel UNREGISTER events debugger-persistence
2014/08/22 05:26:15 LOOKUPD(10.0.1.173:4160): channel UNREGISTER events debugger-persistence
2014/08/22 05:26:15 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:26:15 LOOKUPD(10.0.1.171:4160): channel UNREGISTER events ingestion
2014/08/22 05:26:15 LOOKUPD(10.0.1.172:4160): channel UNREGISTER events ingestion
2014/08/22 05:26:15 LOOKUPD(10.0.1.173:4160): channel UNREGISTER events ingestion
2014/08/22 05:26:15 LOOKUPD(10.0.1.171:4160): channel UNREGISTER events nsq-router-v3#ephemeral
2014/08/22 05:26:15 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:26:15 LOOKUPD(10.0.1.172:4160): channel UNREGISTER events nsq-router-v3#ephemeral
2014/08/22 05:26:15 LOOKUPD(10.0.1.173:4160): channel UNREGISTER events nsq-router-v3#ephemeral
2014/08/22 05:26:15 LOOKUPD(10.0.1.171:4160): channel UNREGISTER events queue-stats#ephemeral
2014/08/22 05:26:15 LOOKUPD(10.0.1.172:4160): channel UNREGISTER events queue-stats#ephemeral
2014/08/22 05:26:15 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:26:15 LOOKUPD(10.0.1.173:4160): channel UNREGISTER events queue-stats#ephemeral
2014/08/22 05:26:15 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:26:15 TCP: new client(10.0.1.61:51304)
2014/08/22 05:26:15 CLIENT(10.0.1.61:51304): desired protocol magic '  V2'
2014/08/22 05:26:15 TOPIC(events): created
2014/08/22 05:26:15 LOOKUPD: querying http://10.0.1.171:4161/channels?topic=events
2014/08/22 05:26:15 LOOKUPD: querying http://10.0.1.172:4161/channels?topic=events
2014/08/22 05:26:15 LOOKUPD: querying http://10.0.1.173:4161/channels?topic=events
2014/08/22 05:26:15 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:26:15 LOOKUPD(10.0.1.171:4160): topic REGISTER events
2014/08/22 05:26:15 LOOKUPD(10.0.1.172:4160): topic REGISTER events
2014/08/22 05:26:15 TOPIC(events): new channel(archiver)
2014/08/22 05:26:15 TOPIC(events): new channel(debugger-persistence)
2014/08/22 05:26:15 TOPIC(events): new channel(ingestion)
2014/08/22 05:26:15 TOPIC(events): new channel(nsq-router-v3#ephemeral)
2014/08/22 05:26:15 TOPIC(events): new channel(queue-stats#ephemeral)
2014/08/22 05:26:15 LOOKUPD(10.0.1.173:4160): topic REGISTER events
2014/08/22 05:26:15 LOOKUPD(10.0.1.171:4160): channel REGISTER events archiver
2014/08/22 05:26:15 LOOKUPD(10.0.1.172:4160): channel REGISTER events archiver
2014/08/22 05:26:15 LOOKUPD(10.0.1.173:4160): channel REGISTER events archiver
2014/08/22 05:26:15 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:26:15 LOOKUPD(10.0.1.171:4160): channel REGISTER events debugger-persistence
2014/08/22 05:26:15 LOOKUPD(10.0.1.172:4160): channel REGISTER events debugger-persistence
2014/08/22 05:26:15 LOOKUPD(10.0.1.173:4160): channel REGISTER events debugger-persistence
2014/08/22 05:26:15 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:26:15 LOOKUPD(10.0.1.171:4160): channel REGISTER events ingestion
2014/08/22 05:26:15 TCP: new client(10.0.0.86:54414)
2014/08/22 05:26:15 CLIENT(10.0.0.86:54414): desired protocol magic '  V2'
2014/08/22 05:26:15 LOOKUPD(10.0.1.172:4160): channel REGISTER events ingestion
2014/08/22 05:26:15 LOOKUPD(10.0.1.173:4160): channel REGISTER events ingestion
2014/08/22 05:26:15 LOOKUPD(10.0.1.171:4160): channel REGISTER events nsq-router-v3#ephemeral
2014/08/22 05:26:15 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:26:15 LOOKUPD(10.0.1.172:4160): channel REGISTER events nsq-router-v3#ephemeral
2014/08/22 05:26:15 LOOKUPD(10.0.1.173:4160): channel REGISTER events nsq-router-v3#ephemeral
2014/08/22 05:26:15 TCP: new client(10.0.0.34:57835)
2014/08/22 05:26:15 LOOKUPD(10.0.1.171:4160): channel REGISTER events queue-stats#ephemeral
2014/08/22 05:26:15 CLIENT(10.0.0.34:57835): desired protocol magic '  V2'
2014/08/22 05:26:15 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:26:15 LOOKUPD(10.0.1.172:4160): channel REGISTER events queue-stats#ephemeral
2014/08/22 05:26:15 LOOKUPD(10.0.1.173:4160): channel REGISTER events queue-stats#ephemeral
2014/08/22 05:26:15 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:26:15 TCP: new client(10.0.0.85:45611)
2014/08/22 05:26:15 CLIENT(10.0.0.85:45611): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.34:57836)
2014/08/22 05:26:15 CLIENT(10.0.0.34:57836): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.32:60459)
2014/08/22 05:26:15 CLIENT(10.0.0.32:60459): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.190:52537)
2014/08/22 05:26:15 CLIENT(10.0.0.190:52537): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.71:47210)
2014/08/22 05:26:15 CLIENT(10.0.0.71:47210): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.188:38664)
2014/08/22 05:26:15 CLIENT(10.0.0.188:38664): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.1.248:51332)
2014/08/22 05:26:15 CLIENT(10.0.1.248:51332): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.188:38665)
2014/08/22 05:26:15 CLIENT(10.0.0.188:38665): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.190:52541)
2014/08/22 05:26:15 CLIENT(10.0.0.190:52541): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.35:32780)
2014/08/22 05:26:15 CLIENT(10.0.0.35:32780): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.67:55801)
2014/08/22 05:26:15 CLIENT(10.0.0.67:55801): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.35:32781)
2014/08/22 05:26:15 CLIENT(10.0.0.35:32781): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.36:55722)
2014/08/22 05:26:15 TCP: new client(10.0.0.70:33168)
2014/08/22 05:26:15 CLIENT(10.0.0.70:33168): desired protocol magic '  V2'
2014/08/22 05:26:15 CLIENT(10.0.0.36:55722): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.188:38667)
2014/08/22 05:26:15 CLIENT(10.0.0.188:38667): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.85:45617)
2014/08/22 05:26:15 CLIENT(10.0.0.85:45617): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.82:59730)
2014/08/22 05:26:15 TCP: new client(10.0.0.219:53128)
2014/08/22 05:26:15 CLIENT(10.0.0.219:53128): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.70:33171)
2014/08/22 05:26:15 CLIENT(10.0.0.70:33171): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.67:55803)
2014/08/22 05:26:15 CLIENT(10.0.0.67:55803): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.35:32789)
2014/08/22 05:26:15 CLIENT(10.0.0.35:32789): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.85:45620)
2014/08/22 05:26:15 CLIENT(10.0.0.85:45620): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.86:54420)
2014/08/22 05:26:15 CLIENT(10.0.0.86:54420): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.66:59326)
2014/08/22 05:26:15 CLIENT(10.0.0.66:59326): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.34:57844)
2014/08/22 05:26:15 CLIENT(10.0.0.34:57844): desired protocol magic '  V2'
2014/08/22 05:26:15 CLIENT(10.0.0.82:59730): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.36:55729)
2014/08/22 05:26:15 CLIENT(10.0.0.36:55729): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.38:54667)
2014/08/22 05:26:15 CLIENT(10.0.0.38:54667): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.38:54668)
2014/08/22 05:26:15 TCP: new client(10.0.0.68:59839)
2014/08/22 05:26:15 CLIENT(10.0.0.68:59839): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.71:47212)
2014/08/22 05:26:15 CLIENT(10.0.0.71:47212): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.34:57846)
2014/08/22 05:26:15 CLIENT(10.0.0.34:57846): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.240:58672)
2014/08/22 05:26:15 CLIENT(10.0.0.240:58672): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.108:58877)
2014/08/22 05:26:15 CLIENT(10.0.0.108:58877): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.190:52542)
2014/08/22 05:26:15 CLIENT(10.0.0.190:52542): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.182:42744)
2014/08/22 05:26:15 CLIENT(10.0.0.182:42744): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.190:52543)
2014/08/22 05:26:15 CLIENT(10.0.0.190:52543): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.86:54421)
2014/08/22 05:26:15 CLIENT(10.0.0.86:54421): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.136:50179)
2014/08/22 05:26:15 CLIENT(10.0.0.136:50179): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.69:52951)
2014/08/22 05:26:15 CLIENT(10.0.0.69:52951): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.32:60465)
2014/08/22 05:26:15 CLIENT(10.0.0.32:60465): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.68:59840)
2014/08/22 05:26:15 CLIENT(10.0.0.68:59840): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.86:54422)
2014/08/22 05:26:15 CLIENT(10.0.0.86:54422): desired protocol magic '  V2'
2014/08/22 05:26:15 TCP: new client(10.0.0.219:53131)
2014/08/22 05:26:15 CLIENT(10.0.0.219:53131): desired protocol magic '  V2'
2014/08/22 05:26:16 TCP: new client(10.0.0.32:60466)
2014/08/22 05:26:16 CLIENT(10.0.0.32:60466): desired protocol magic '  V2'
2014/08/22 05:26:16 TCP: new client(10.0.0.219:53132)
2014/08/22 05:26:16 CLIENT(10.0.0.219:53132): desired protocol magic '  V2'
2014/08/22 05:26:16 TCP: new client(10.0.0.85:45622)
2014/08/22 05:26:16 CLIENT(10.0.0.85:45622): desired protocol magic '  V2'
2014/08/22 05:26:16 TCP: new client(10.0.0.108:58879)
2014/08/22 05:26:16 CLIENT(10.0.0.108:58879): desired protocol magic '  V2'
2014/08/22 05:26:16 TCP: new client(10.0.0.182:42745)
2014/08/22 05:26:16 CLIENT(10.0.0.182:42745): desired protocol magic '  V2'
2014/08/22 05:26:16 TCP: new client(10.0.0.35:32790)
2014/08/22 05:26:16 CLIENT(10.0.0.35:32790): desired protocol magic '  V2'
2014/08/22 05:26:16 TCP: new client(10.0.0.36:55732)
2014/08/22 05:26:16 CLIENT(10.0.0.36:55732): desired protocol magic '  V2'
2014/08/22 05:26:16 CLIENT(10.0.0.38:54668): desired protocol magic '  V2'
2014/08/22 05:26:16 TCP: new client(10.0.0.136:50181)
2014/08/22 05:26:16 CLIENT(10.0.0.136:50181): desired protocol magic '  V2'
2014/08/22 05:26:16 TCP: new client(10.0.0.240:58673)
2014/08/22 05:26:16 TCP: new client(10.0.1.61:51305)
2014/08/22 05:26:16 CLIENT(10.0.1.61:51305): desired protocol magic '  V2'
2014/08/22 05:26:16 CLIENT(10.0.0.240:58673): desired protocol magic '  V2'
2014/08/22 05:26:16 TCP: new client(10.0.0.219:53133)
2014/08/22 05:26:16 CLIENT(10.0.0.219:53133): desired protocol magic '  V2'
2014/08/22 05:26:16 TCP: new client(10.0.0.108:58880)
2014/08/22 05:26:16 CLIENT(10.0.0.108:58880): desired protocol magic '  V2'
2014/08/22 05:26:16 TCP: new client(10.0.0.32:60471)
2014/08/22 05:26:16 CLIENT(10.0.0.32:60471): desired protocol magic '  V2'
2014/08/22 05:26:16 TCP: new client(10.0.0.82:59732)
2014/08/22 05:26:16 CLIENT(10.0.0.82:59732): desired protocol magic '  V2'
2014/08/22 05:26:16 TCP: new client(10.0.0.108:58881)
2014/08/22 05:26:16 CLIENT(10.0.0.108:58881): desired protocol magic '  V2'
2014/08/22 05:26:16 TCP: new client(10.0.0.188:38674)
2014/08/22 05:26:16 CLIENT(10.0.0.188:38674): desired protocol magic '  V2'
2014/08/22 05:26:16 TCP: new client(10.0.0.66:59330)
2014/08/22 05:26:16 CLIENT(10.0.0.66:59330): desired protocol magic '  V2'
2014/08/22 05:26:16 TCP: new client(10.0.0.69:52953)
2014/08/22 05:26:16 CLIENT(10.0.0.69:52953): desired protocol magic '  V2'
2014/08/22 05:26:16 TCP: new client(10.0.0.36:55733)
2014/08/22 05:26:16 CLIENT(10.0.0.36:55733): desired protocol magic '  V2'

and here's the nsqd log with the X on the topic page:

2014/08/22 05:28:22 LOOKUPD(10.0.1.171:4160): sending heartbeat
2014/08/22 05:28:22 LOOKUPD(10.0.1.172:4160): sending heartbeat
2014/08/22 05:28:22 LOOKUPD(10.0.1.173:4160): sending heartbeat
2014/08/22 05:28:27 TOPIC(events): deleting
2014/08/22 05:28:27 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:28:27 TOPIC(events): closing ... messagePump
2014/08/22 05:28:27 TOPIC(events): closing ... router
2014/08/22 05:28:27 LOOKUPD(10.0.1.171:4160): topic UNREGISTER events
2014/08/22 05:28:27 CHANNEL(archiver): deleting
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.1.123:40170] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.1.123:40170) - read tcp 10.0.1.123:40170: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.1.121:35516] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.1.121:35516) - read tcp 10.0.1.121:35516: use of closed network connection
2014/08/22 05:28:27 CHANNEL(archiver): closing ... messagePump
2014/08/22 05:28:27 CHANNEL(archiver): closing ... pqueue worker
2014/08/22 05:28:27 CHANNEL(archiver): closing ... pqueue worker
2014/08/22 05:28:27 CHANNEL(archiver): closing ... router
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.1.123:40170] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.1.121:35516] exiting messagePump
2014/08/22 05:28:27 DISKQUEUE(events:archiver): emptying
2014/08/22 05:28:27 DISKQUEUE(events:archiver): deleting
2014/08/22 05:28:27 DISKQUEUE(events:archiver): closing ... ioLoop
2014/08/22 05:28:27 CHANNEL(debugger-persistence): deleting
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.1.61:51304] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.1.61:51304) - read tcp 10.0.1.61:51304: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.1.61:51305] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.1.61:51305) - read tcp 10.0.1.61:51305: use of closed network connection
2014/08/22 05:28:27 CHANNEL(debugger-persistence): closing ... messagePump
2014/08/22 05:28:27 CHANNEL(debugger-persistence): closing ... pqueue worker
2014/08/22 05:28:27 CHANNEL(debugger-persistence): closing ... pqueue worker
2014/08/22 05:28:27 CHANNEL(debugger-persistence): closing ... router
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.1.61:51304] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.1.61:51305] exiting messagePump
2014/08/22 05:28:27 DISKQUEUE(events:debugger-persistence): emptying
2014/08/22 05:28:27 DISKQUEUE(events:debugger-persistence): deleting
2014/08/22 05:28:27 DISKQUEUE(events:debugger-persistence): closing ... ioLoop
2014/08/22 05:28:27 CHANNEL(ingestion): deleting
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.85:45617] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.85:45617) - read tcp 10.0.0.85:45617: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.85:45620] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.85:45620) - read tcp 10.0.0.85:45620: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.108:58879] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.108:58879) - read tcp 10.0.0.108:58879: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.86:54414] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.86:54414) - read tcp 10.0.0.86:54414: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.190:52541] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.190:52541) - read tcp 10.0.0.190:52541: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.68:59840] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.68:59840) - read tcp 10.0.0.68:59840: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.32:60466] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.32:60466) - read tcp 10.0.0.32:60466: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.36:55732] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.36:55732) - read tcp 10.0.0.36:55732: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.136:50181] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.136:50181) - read tcp 10.0.0.136:50181: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.188:38665] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.188:38665) - read tcp 10.0.0.188:38665: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.70:33168] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.70:33168) - read tcp 10.0.0.70:33168: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.85:45622] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.85:45622) - read tcp 10.0.0.85:45622: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.71:47210] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.71:47210) - read tcp 10.0.0.71:47210: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.69:52951] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.69:52951) - read tcp 10.0.0.69:52951: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.219:53131] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.219:53131) - read tcp 10.0.0.219:53131: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.35:32780] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.35:32780) - read tcp 10.0.0.35:32780: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.35:32789] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.35:32789) - read tcp 10.0.0.35:32789: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.34:57844] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.34:57844) - read tcp 10.0.0.34:57844: use of closed network connection
2014/08/22 05:28:27 LOOKUPD(10.0.1.172:4160): topic UNREGISTER events
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.67:55803] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.67:55803) - read tcp 10.0.0.67:55803: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.108:58881] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.108:58881) - read tcp 10.0.0.108:58881: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.66:59330] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.66:59330) - read tcp 10.0.0.66:59330: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.188:38674] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.188:38674) - read tcp 10.0.0.188:38674: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.34:57836] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.34:57836) - read tcp 10.0.0.34:57836: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.86:54420] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.86:54420) - read tcp 10.0.0.86:54420: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.66:59326] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.66:59326) - read tcp 10.0.0.66:59326: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.34:57846] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.34:57846) - read tcp 10.0.0.34:57846: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.108:58877] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.108:58877) - read tcp 10.0.0.108:58877: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.188:38664] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.188:38664) - read tcp 10.0.0.188:38664: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.219:53128] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.219:53128) - read tcp 10.0.0.219:53128: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.70:33171] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.70:33171) - read tcp 10.0.0.70:33171: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.82:59730] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.82:59730) - read tcp 10.0.0.82:59730: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.69:52953] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.69:52953) - read tcp 10.0.0.69:52953: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.86:54421] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.86:54421) - read tcp 10.0.0.86:54421: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.219:53132] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.219:53132) - read tcp 10.0.0.219:53132: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.36:55729] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.36:55729) - read tcp 10.0.0.36:55729: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.35:32790] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.35:32790) - read tcp 10.0.0.35:32790: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.32:60459] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.32:60459) - read tcp 10.0.0.32:60459: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.67:55801] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.67:55801) - read tcp 10.0.0.67:55801: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.32:60471] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.32:60471) - read tcp 10.0.0.32:60471: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.82:59732] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.82:59732) - read tcp 10.0.0.82:59732: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.35:32781] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.35:32781) - read tcp 10.0.0.35:32781: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.36:55722] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.36:55722) - read tcp 10.0.0.36:55722: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.71:47212] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.71:47212) - read tcp 10.0.0.71:47212: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.86:54422] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.86:54422) - read tcp 10.0.0.86:54422: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.85:45611] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.85:45611) - read tcp 10.0.0.85:45611: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.190:52537] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.190:52537) - read tcp 10.0.0.190:52537: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.188:38667] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.188:38667) - read tcp 10.0.0.188:38667: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.190:52542] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.190:52542) - read tcp 10.0.0.190:52542: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.136:50179] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.136:50179) - read tcp 10.0.0.136:50179: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.32:60465] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.32:60465) - read tcp 10.0.0.32:60465: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.219:53133] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.219:53133) - read tcp 10.0.0.219:53133: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.108:58880] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.108:58880) - read tcp 10.0.0.108:58880: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.34:57835] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.34:57835) - read tcp 10.0.0.34:57835: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.68:59839] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.68:59839) - read tcp 10.0.0.68:59839: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.190:52543] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.190:52543) - read tcp 10.0.0.190:52543: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.36:55733] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.36:55733) - read tcp 10.0.0.36:55733: use of closed network connection
2014/08/22 05:28:27 CHANNEL(ingestion): closing ... messagePump
2014/08/22 05:28:27 CHANNEL(ingestion): closing ... pqueue worker
2014/08/22 05:28:27 CHANNEL(ingestion): closing ... pqueue worker
2014/08/22 05:28:27 CHANNEL(ingestion): closing ... router
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.85:45617] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.85:45620] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.108:58879] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.86:54414] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.190:52541] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.68:59840] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.32:60466] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.36:55732] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.136:50181] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.188:38665] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.70:33168] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.85:45622] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.71:47210] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.69:52951] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.219:53131] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.35:32780] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.35:32789] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.34:57844] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.67:55803] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.108:58881] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.66:59330] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.188:38674] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.34:57836] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.86:54420] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.66:59326] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.34:57846] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.108:58877] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.188:38664] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.219:53128] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.70:33171] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.82:59730] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.69:52953] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.86:54421] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.219:53132] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.36:55729] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.35:32790] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.32:60459] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.67:55801] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.32:60471] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.82:59732] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.35:32781] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.36:55722] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.71:47212] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.86:54422] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.85:45611] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.190:52537] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.188:38667] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.190:52542] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.136:50179] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.32:60465] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.219:53133] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.108:58880] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.34:57835] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.68:59839] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.190:52543] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.36:55733] exiting messagePump
2014/08/22 05:28:27 DISKQUEUE(events:ingestion): emptying
2014/08/22 05:28:27 DISKQUEUE(events:ingestion): deleting
2014/08/22 05:28:27 DISKQUEUE(events:ingestion): closing ... ioLoop
2014/08/22 05:28:27 CHANNEL(nsq-router-v3#ephemeral): deleting
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.38:54667] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.38:54667) - read tcp 10.0.0.38:54667: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.240:58672] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.240:58672) - read tcp 10.0.0.240:58672: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.182:42744] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.182:42744) - read tcp 10.0.0.182:42744: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.182:42745] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.182:42745) - read tcp 10.0.0.182:42745: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.240:58673] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.240:58673) - read tcp 10.0.0.240:58673: use of closed network connection
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.38:54668] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.0.38:54668) - read tcp 10.0.0.38:54668: use of closed network connection
2014/08/22 05:28:27 CHANNEL(nsq-router-v3#ephemeral): closing ... messagePump
2014/08/22 05:28:27 CHANNEL(nsq-router-v3#ephemeral): closing ... pqueue worker
2014/08/22 05:28:27 CHANNEL(nsq-router-v3#ephemeral): closing ... pqueue worker
2014/08/22 05:28:27 CHANNEL(nsq-router-v3#ephemeral): closing ... router
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.38:54667] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.240:58672] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.182:42744] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.182:42745] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.240:58673] exiting messagePump
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.0.38:54668] exiting messagePump
2014/08/22 05:28:27 CHANNEL(queue-stats#ephemeral): deleting
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.1.248:51332] exiting ioloop
2014/08/22 05:28:27 ERROR: client(10.0.1.248:51332) - read tcp 10.0.1.248:51332: use of closed network connection
2014/08/22 05:28:27 CHANNEL(queue-stats#ephemeral): closing ... messagePump
2014/08/22 05:28:27 CHANNEL(queue-stats#ephemeral): closing ... pqueue worker
2014/08/22 05:28:27 CHANNEL(queue-stats#ephemeral): closing ... pqueue worker
2014/08/22 05:28:27 CHANNEL(queue-stats#ephemeral): closing ... router
2014/08/22 05:28:27 PROTOCOL(V2): [10.0.1.248:51332] exiting messagePump
2014/08/22 05:28:27 DISKQUEUE(events): emptying
2014/08/22 05:28:27 DISKQUEUE(events): deleting
2014/08/22 05:28:27 DISKQUEUE(events): closing ... ioLoop
2014/08/22 05:28:27 LOOKUPD(10.0.1.173:4160): topic UNREGISTER events
2014/08/22 05:28:27 LOOKUPD(10.0.1.171:4160): channel UNREGISTER events archiver
2014/08/22 05:28:27 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:28:27 LOOKUPD(10.0.1.172:4160): channel UNREGISTER events archiver
2014/08/22 05:28:27 LOOKUPD(10.0.1.173:4160): channel UNREGISTER events archiver
2014/08/22 05:28:27 LOOKUPD(10.0.1.171:4160): channel UNREGISTER events debugger-persistence
2014/08/22 05:28:27 LOOKUPD(10.0.1.172:4160): channel UNREGISTER events debugger-persistence
2014/08/22 05:28:27 LOOKUPD(10.0.1.173:4160): channel UNREGISTER events debugger-persistence
2014/08/22 05:28:27 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:28:27 LOOKUPD(10.0.1.171:4160): channel UNREGISTER events ingestion
2014/08/22 05:28:27 LOOKUPD(10.0.1.172:4160): channel UNREGISTER events ingestion
2014/08/22 05:28:27 LOOKUPD(10.0.1.173:4160): channel UNREGISTER events ingestion
2014/08/22 05:28:27 LOOKUPD(10.0.1.171:4160): channel UNREGISTER events nsq-router-v3#ephemeral
2014/08/22 05:28:27 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:28:27 LOOKUPD(10.0.1.172:4160): channel UNREGISTER events nsq-router-v3#ephemeral
2014/08/22 05:28:27 LOOKUPD(10.0.1.173:4160): channel UNREGISTER events nsq-router-v3#ephemeral
2014/08/22 05:28:27 LOOKUPD(10.0.1.171:4160): channel UNREGISTER events queue-stats#ephemeral
2014/08/22 05:28:27 LOOKUPD(10.0.1.172:4160): channel UNREGISTER events queue-stats#ephemeral
2014/08/22 05:28:27 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:28:27 LOOKUPD(10.0.1.173:4160): channel UNREGISTER events queue-stats#ephemeral
2014/08/22 05:28:27 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:28:27 TCP: new client(10.0.0.219:58658)
2014/08/22 05:28:27 CLIENT(10.0.0.219:58658): desired protocol magic '  V2'
2014/08/22 05:28:27 TOPIC(events): created
2014/08/22 05:28:27 LOOKUPD: querying http://10.0.1.171:4161/channels?topic=events
2014/08/22 05:28:27 LOOKUPD: querying http://10.0.1.172:4161/channels?topic=events
2014/08/22 05:28:27 LOOKUPD: querying http://10.0.1.173:4161/channels?topic=events
2014/08/22 05:28:27 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:28:27 LOOKUPD(10.0.1.171:4160): topic REGISTER events
2014/08/22 05:28:27 LOOKUPD(10.0.1.172:4160): topic REGISTER events
2014/08/22 05:28:27 TOPIC(events): new channel(archiver)
2014/08/22 05:28:27 TOPIC(events): new channel(debugger-persistence)
2014/08/22 05:28:27 TOPIC(events): new channel(ingestion)
2014/08/22 05:28:27 TOPIC(events): new channel(nsq-router-v3#ephemeral)
2014/08/22 05:28:27 TOPIC(events): new channel(queue-stats#ephemeral)
2014/08/22 05:28:27 LOOKUPD(10.0.1.173:4160): topic REGISTER events
2014/08/22 05:28:27 LOOKUPD(10.0.1.171:4160): channel REGISTER events archiver
2014/08/22 05:28:27 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:28:27 TCP: new client(10.0.0.32:37954)
2014/08/22 05:28:27 CLIENT(10.0.0.32:37954): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.86:60557)
2014/08/22 05:28:27 CLIENT(10.0.0.86:60557): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.85:51685)
2014/08/22 05:28:27 CLIENT(10.0.0.85:51685): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.66:34083)
2014/08/22 05:28:27 CLIENT(10.0.0.66:34083): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.38:54947)
2014/08/22 05:28:27 CLIENT(10.0.0.38:54947): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.188:44183)
2014/08/22 05:28:27 CLIENT(10.0.0.188:44183): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.108:36710)
2014/08/22 05:28:27 CLIENT(10.0.0.108:36710): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.85:51687)
2014/08/22 05:28:27 CLIENT(10.0.0.85:51687): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.86:60558)
2014/08/22 05:28:27 CLIENT(10.0.0.86:60558): desired protocol magic '  V2'
2014/08/22 05:28:27 LOOKUPD(10.0.1.172:4160): channel REGISTER events archiver
2014/08/22 05:28:27 LOOKUPD(10.0.1.173:4160): channel REGISTER events archiver
2014/08/22 05:28:27 TCP: new client(10.0.0.70:36085)
2014/08/22 05:28:27 CLIENT(10.0.0.70:36085): desired protocol magic '  V2'
2014/08/22 05:28:27 LOOKUPD(10.0.1.171:4160): channel REGISTER events debugger-persistence
2014/08/22 05:28:27 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:28:27 LOOKUPD(10.0.1.172:4160): channel REGISTER events debugger-persistence
2014/08/22 05:28:27 LOOKUPD(10.0.1.173:4160): channel REGISTER events debugger-persistence
2014/08/22 05:28:27 LOOKUPD(10.0.1.171:4160): channel REGISTER events ingestion
2014/08/22 05:28:27 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:28:27 LOOKUPD(10.0.1.172:4160): channel REGISTER events ingestion
2014/08/22 05:28:27 LOOKUPD(10.0.1.173:4160): channel REGISTER events ingestion
2014/08/22 05:28:27 LOOKUPD(10.0.1.171:4160): channel REGISTER events nsq-router-v3#ephemeral
2014/08/22 05:28:27 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:28:27 LOOKUPD(10.0.1.172:4160): channel REGISTER events nsq-router-v3#ephemeral
2014/08/22 05:28:27 LOOKUPD(10.0.1.173:4160): channel REGISTER events nsq-router-v3#ephemeral
2014/08/22 05:28:27 LOOKUPD(10.0.1.171:4160): channel REGISTER events queue-stats#ephemeral
2014/08/22 05:28:27 NSQ: persisting topic/channel metadata to /data/nsqd/nsqd.665.dat
2014/08/22 05:28:27 LOOKUPD(10.0.1.172:4160): channel REGISTER events queue-stats#ephemeral
2014/08/22 05:28:27 TCP: new client(10.0.0.67:58834)
2014/08/22 05:28:27 CLIENT(10.0.0.67:58834): desired protocol magic '  V2'
2014/08/22 05:28:27 LOOKUPD(10.0.1.173:4160): channel REGISTER events queue-stats#ephemeral
2014/08/22 05:28:27 TCP: new client(10.0.0.36:33548)
2014/08/22 05:28:27 CLIENT(10.0.0.36:33548): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.70:36086)
2014/08/22 05:28:27 CLIENT(10.0.0.70:36086): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.36:33549)
2014/08/22 05:28:27 CLIENT(10.0.0.36:33549): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.86:60561)
2014/08/22 05:28:27 CLIENT(10.0.0.86:60561): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.35:38633)
2014/08/22 05:28:27 CLIENT(10.0.0.35:38633): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.68:34710)
2014/08/22 05:28:27 TCP: new client(10.0.0.35:38634)
2014/08/22 05:28:27 CLIENT(10.0.0.35:38634): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.34:35661)
2014/08/22 05:28:27 CLIENT(10.0.0.34:35661): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.190:58823)
2014/08/22 05:28:27 CLIENT(10.0.0.190:58823): desired protocol magic '  V2'
2014/08/22 05:28:27 CLIENT(10.0.0.68:34710): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.34:35663)
2014/08/22 05:28:27 CLIENT(10.0.0.34:35663): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.35:38636)
2014/08/22 05:28:27 CLIENT(10.0.0.35:38636): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.85:51689)
2014/08/22 05:28:27 CLIENT(10.0.0.85:51689): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.136:53211)
2014/08/22 05:28:27 CLIENT(10.0.0.136:53211): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.108:36718)
2014/08/22 05:28:27 TCP: new client(10.0.0.69:56039)
2014/08/22 05:28:27 CLIENT(10.0.0.69:56039): desired protocol magic '  V2'
2014/08/22 05:28:27 CLIENT(10.0.0.108:36718): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.32:37958)
2014/08/22 05:28:27 CLIENT(10.0.0.32:37958): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.188:44187)
2014/08/22 05:28:27 CLIENT(10.0.0.188:44187): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.219:58661)
2014/08/22 05:28:27 CLIENT(10.0.0.219:58661): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.190:58824)
2014/08/22 05:28:27 CLIENT(10.0.0.190:58824): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.190:58825)
2014/08/22 05:28:27 CLIENT(10.0.0.190:58825): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.36:33551)
2014/08/22 05:28:27 CLIENT(10.0.0.36:33551): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.66:34088)
2014/08/22 05:28:27 CLIENT(10.0.0.66:34088): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.136:53212)
2014/08/22 05:28:27 CLIENT(10.0.0.136:53212): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.190:58826)
2014/08/22 05:28:27 CLIENT(10.0.0.190:58826): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.86:60562)
2014/08/22 05:28:27 CLIENT(10.0.0.86:60562): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.32:37959)
2014/08/22 05:28:27 CLIENT(10.0.0.32:37959): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.71:50294)
2014/08/22 05:28:27 CLIENT(10.0.0.71:50294): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.69:56040)
2014/08/22 05:28:27 CLIENT(10.0.0.69:56040): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.108:36722)
2014/08/22 05:28:27 CLIENT(10.0.0.108:36722): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.85:51690)
2014/08/22 05:28:27 CLIENT(10.0.0.85:51690): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.1.61:51395)
2014/08/22 05:28:27 CLIENT(10.0.1.61:51395): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.219:58662)
2014/08/22 05:28:27 CLIENT(10.0.0.219:58662): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.219:58663)
2014/08/22 05:28:27 CLIENT(10.0.0.219:58663): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.71:50295)
2014/08/22 05:28:27 CLIENT(10.0.0.71:50295): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.82:33988)
2014/08/22 05:28:27 CLIENT(10.0.0.82:33988): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.34:35667)
2014/08/22 05:28:27 CLIENT(10.0.0.34:35667): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.188:44191)
2014/08/22 05:28:27 CLIENT(10.0.0.188:44191): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.35:38640)
2014/08/22 05:28:27 CLIENT(10.0.0.35:38640): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.38:54948)
2014/08/22 05:28:27 CLIENT(10.0.0.38:54948): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.108:36727)
2014/08/22 05:28:27 CLIENT(10.0.0.108:36727): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.68:34712)
2014/08/22 05:28:27 CLIENT(10.0.0.68:34712): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.188:44192)
2014/08/22 05:28:27 CLIENT(10.0.0.188:44192): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.1.61:51396)
2014/08/22 05:28:27 CLIENT(10.0.1.61:51396): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.82:33989)
2014/08/22 05:28:27 CLIENT(10.0.0.82:33989): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.34:35672)
2014/08/22 05:28:27 CLIENT(10.0.0.34:35672): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.182:43024)
2014/08/22 05:28:27 TCP: new client(10.0.0.182:43025)
2014/08/22 05:28:27 TCP: new client(10.0.0.67:58835)
2014/08/22 05:28:27 CLIENT(10.0.0.67:58835): desired protocol magic '  V2'
2014/08/22 05:28:27 CLIENT(10.0.0.182:43024): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.32:37963)
2014/08/22 05:28:27 CLIENT(10.0.0.32:37963): desired protocol magic '  V2'
2014/08/22 05:28:27 CLIENT(10.0.0.182:43025): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.240:58950)
2014/08/22 05:28:27 CLIENT(10.0.0.240:58950): desired protocol magic '  V2'
2014/08/22 05:28:27 TCP: new client(10.0.0.240:58951)
2014/08/22 05:28:27 TCP: new client(10.0.0.36:33558)
2014/08/22 05:28:27 CLIENT(10.0.0.240:58951): desired protocol magic '  V2'
2014/08/22 05:28:28 CLIENT(10.0.0.36:33558): desired protocol magic '  V2'
2014/08/22 05:28:28 TCP: new client(10.0.1.248:51434)
2014/08/22 05:28:28 CLIENT(10.0.1.248:51434): desired protocol magic '  V2'
2014/08/22 05:28:37 LOOKUPD(10.0.1.171:4160): sending heartbeat
2014/08/22 05:28:37 LOOKUPD(10.0.1.172:4160): sending heartbeat
2014/08/22 05:28:37 LOOKUPD(10.0.1.173:4160): sending heartbeat
2014/08/22 05:28:50 TCP: new client(10.0.1.123:40211)
2014/08/22 05:28:50 CLIENT(10.0.1.123:40211): desired protocol magic '  V2'
2014/08/22 05:28:52 STATSD: pushing stats to 10.0.0.214:5000
2014/08/22 05:28:52 LOOKUPD(10.0.1.171:4160): sending heartbeat
2014/08/22 05:28:52 LOOKUPD(10.0.1.172:4160): sending heartbeat
2014/08/22 05:28:52 LOOKUPD(10.0.1.173:4160): sending heartbeat
2014/08/22 05:29:05 TCP: new client(10.0.1.121:35556)
2014/08/22 05:29:05 CLIENT(10.0.1.121:35556): desired protocol magic '  V2'
2014/08/22 05:29:07 LOOKUPD(10.0.1.171:4160): sending heartbeat
2014/08/22 05:29:07 LOOKUPD(10.0.1.172:4160): sending heartbeat
2014/08/22 05:29:07 LOOKUPD(10.0.1.173:4160): sending heartbeat

and one of the nsqlookupd's:

2014/08/22 05:33:14 DB: setting tombstone for producer@10.0.1.121:5000 of topic(events)
2014/08/22 05:33:14 WARNING: client(10.0.1.121:53992) unexpected UNREGISTER category:channel key:events subkey:queue-stats#ephemeral
2014/08/22 05:33:14 WARNING: client(10.0.1.121:53992) unexpected UNREGISTER category:channel key:events subkey:ingestion
2014/08/22 05:33:14 WARNING: client(10.0.1.121:53992) unexpected UNREGISTER category:channel key:events subkey:nsq-router-v3#ephemeral
2014/08/22 05:33:14 WARNING: client(10.0.1.121:53992) unexpected UNREGISTER category:channel key:events subkey:debugger-persistence
2014/08/22 05:33:14 WARNING: client(10.0.1.121:53992) unexpected UNREGISTER category:channel key:events subkey:archiver
2014/08/22 05:33:14 DB: client(10.0.1.121:53992) UNREGISTER category:topic key:events subkey:
2014/08/22 05:33:14 CLIENT(10.0.1.91:36113): pinged (last ping 14.99989172s)
2014/08/22 05:33:14 DB: client(10.0.1.121:53992) REGISTER category:topic key:events subkey:
2014/08/22 05:33:14 DB: client(10.0.1.121:53992) REGISTER category:channel key:events subkey:archiver
2014/08/22 05:33:14 DB: client(10.0.1.121:53992) REGISTER category:channel key:events subkey:debugger-persistence
2014/08/22 05:33:14 DB: client(10.0.1.121:53992) REGISTER category:channel key:events subkey:ingestion
2014/08/22 05:33:14 DB: client(10.0.1.121:53992) REGISTER category:channel key:events subkey:nsq-router-v3#ephemeral
2014/08/22 05:33:14 DB: client(10.0.1.121:53992) REGISTER category:channel key:events subkey:queue-stats#ephemeral

@mreiferson mreiferson reopened this Aug 22, 2014
@mreiferson
Copy link
Member

@visionmedia these clients are using nsq.js, right?

What is their behavior when clients are configured to use nsqlookupd and you receive an un-expected disconnect from an nsqd?

According to these logs, it looks like it reconnects immediately. That's not quite correct, it should wait for the next discovery cycle and only if it finds that node again should it reconnect.

That would also cause these symptoms.

@tj
Copy link
Contributor Author

tj commented Aug 22, 2014

Yeah they just attempt to reconnect with backoff right now, I guess that's where I found the behaviour of a consumer creating a topic a little weird. I'll get that updated thanks!

@tj tj closed this as completed Aug 22, 2014
@tj tj mentioned this issue Aug 22, 2014
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tj @mreiferson