too many open files

[mindscratch]

I ran into an issue where publishing messages to NSQ would eventually result in too many open files. I had code in a function that would publish a message as so:

func enqueue(queue string, message []byte) error {
  producer, err := nsq.NewProducer("...", config)
  if err != nil {
     return err
  }
  defer producer.Stop()
  err = producer.Publish(queue, message)
  return err
}

Everytime I called that function a new file descriptor (type = sock when using lsof to monitor things) was being created, that would not go away.

I moved the creation of a the producer outside the function, so the same instance would be used each time a message is published, which has "fixed" the issue. My question, I thought defer producer.Stop() would "cleanup" and thus not cause too many open files. Thoughts?

[mreiferson]

Correct, you should re-use the publisher so you don't continually open new connections.

It's possible you were calling enqueue faster than the OS was reclaiming fds.

[mindscratch]

Thanks.