NewProducer and nslookupd

[gravis]

Hi,

It's a bit unclear to me why NewProducer needs a nsqd address, while the consumer has a ConnectToNSQLookupd. Why nsqlookupd doesn't return the list of available nsqd nodes, and the producer use that, like the consumer?
We have to keep a list of nodes for the producer, it's completely redundant with nsqlookupd.
(btw: https://godoc.org/github.com/nsqio/go-nsq#NewProducer is unclear about what is addr, the explanation is elsewhere https://godoc.org/github.com/nsqio/go-nsq#Producer).

Thanks

[mreiferson]

Hi @gravis - this is by design, because there's a chicken an egg problem for producers. The /lookup endpoint will only return nsqd nodes that have registered that topic, but if you haven't yet published that topic to a node it can't register without manual intervention

In our recommended deployment topology, publishing is directed at very specific nsqd nodes (usually localhost).

[gravis]

I can understand that, but still don't see why nslookup can't return the list of available nodes for that.
If nodes have the topic I want to produce on, nslookupd would return them (and the producer can have a fallback solution if the first node is failing).
If nodes don't have the topic, all nodes are good candidates to start.
I probably don't get the whole picture yet.

[judwhite]

@gravis You can get the publishing nsqds for a given topic in JSON format, for example http://localhost:4161/lookup?topic=abc. You can get all nodes from http://localhost:4161/nodes.

You may have your nsqds and nsqlookupds segregated so that all nodes are good candidates (we do this also), but this isn't a requirement. nsqlookupd can handle several nsqd "clusters" with distinct responsibilities, like a service discovery service.

When you get the nodes back you'll have to make a decision about which node you want to publish to. In our setup we either publish to localhost if nsqd is on the same box or a VIP.