New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NewProducer and nslookupd #170

Closed
gravis opened this Issue Mar 11, 2016 · 3 comments

Comments

Projects
None yet
3 participants
@gravis
Copy link

gravis commented Mar 11, 2016

Hi,

It's a bit unclear to me why NewProducer needs a nsqd address, while the consumer has a ConnectToNSQLookupd. Why nsqlookupd doesn't return the list of available nsqd nodes, and the producer use that, like the consumer?
We have to keep a list of nodes for the producer, it's completely redundant with nsqlookupd.
(btw: https://godoc.org/github.com/nsqio/go-nsq#NewProducer is unclear about what is addr, the explanation is elsewhere https://godoc.org/github.com/nsqio/go-nsq#Producer).

Thanks

@mreiferson

This comment has been minimized.

Copy link
Member

mreiferson commented Mar 11, 2016

Hi @gravis - this is by design, because there's a chicken an egg problem for producers. The /lookup endpoint will only return nsqd nodes that have registered that topic, but if you haven't yet published that topic to a node it can't register without manual intervention

In our recommended deployment topology, publishing is directed at very specific nsqd nodes (usually localhost).

@gravis

This comment has been minimized.

Copy link
Author

gravis commented Mar 15, 2016

I can understand that, but still don't see why nslookup can't return the list of available nodes for that.
If nodes have the topic I want to produce on, nslookupd would return them (and the producer can have a fallback solution if the first node is failing).
If nodes don't have the topic, all nodes are good candidates to start.
I probably don't get the whole picture yet.

@judwhite

This comment has been minimized.

Copy link
Contributor

judwhite commented Mar 15, 2016

@gravis You can get the publishing nsqds for a given topic in JSON format, for example http://localhost:4161/lookup?topic=abc. You can get all nodes from http://localhost:4161/nodes.

You may have your nsqds and nsqlookupds segregated so that all nodes are good candidates (we do this also), but this isn't a requirement. nsqlookupd can handle several nsqd "clusters" with distinct responsibilities, like a service discovery service.

When you get the nodes back you'll have to make a decision about which node you want to publish to. In our setup we either publish to localhost if nsqd is on the same box or a VIP.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment