Servname not supported for ai_socktype (48)

[eandersson]

class 'socket.gaierror' [Errno -8] Servname not supported for ai_socktype

I started seeing this message in the logs since I upgraded to 0.9.13. It is unfortunately a difficult bug to reproduce, as it happens about once per every couple of hundred connections opened and closed.

My guess is that this it could be related to the commit(s) introducing IPv6 support 61ecac445bf5a71a5f88381585b76246d50558a5We

I downgraded to 0.9.8 a couple of hours ago, and it hasn't happened since.

Running CentOS 6.4 with Python 2.7.3 and RabbitMQ 3.1 on localhost.

def connect(self, host):
    self.connection = pika.BlockingConnection(pika.ConnectionParameters(......))

def __open_channel(self, queue):
    self.queue = queue
    self.channel = self.connection.channel()
    self.channel.queue_declare(queue=queue, durable=True)
    self.channel.queue_bind(exchange='amq.direct',
           queue=self.queue,
           routing_key=self.queue)

def publish(self, message):
    self.channel.basic_publish(exchange='',
                               routing_key=self.queue,
                               body=message,
                               properties=pika.BasicProperties(
                                   delivery_mode=2  # make message persistent)
                               ))

[eandersson]

Not sure how to reproduce this in a testable manner, but might be worth double checking the getaddrinfo related changes.

[gmr]

Anything custom in your CentOS 6.4 configuration with regard to network stack? Do you have an IPv6 addr enabled?

Can I get a stacktrace with a little more context around the exception so i can catch it?

[gmr]

Hmm apparently this is the report port being parsed in as a string from googling a bit and reading blogposts on this exception. The only odd thing is Pika should be checking to make sure the port is an int. Are you changing this via attribute in your code by any chance?

[eandersson]

IPv6 should is disabled, and nothing out of the ordinary.

Yea, I have port=5672 set in the ConnectionParameters. It is odd as I have multiple connections running with identical parameters at any given time, and I have only seen it happen to one of those connections, never multiple at the same time.

[eandersson]

I am now able to reproduce it on my VM running CentOS 6.4. Here is the stacktrace.

*** print_exception:
Traceback (most recent call last):
  File "test.py", line 37, in process_task
    r.connect(str('localhost'))
  File "/home/eandersson/test_project/lib/connectors/rabbitmqbase.py", line 22, in connect
    host=host, port=5672))
  File "build/bdist.linux-x86_64/egg/pika/adapters/blocking_connection.py", line 107, in __init__
    super(BlockingConnection, self).__init__(parameters, None, False)
  File "build/bdist.linux-x86_64/egg/pika/adapters/base_connection.py", line 62, in __init__
    on_close_callback)
  File "build/bdist.linux-x86_64/egg/pika/connection.py", line 590, in __init__
    self.connect()
  File "build/bdist.linux-x86_64/egg/pika/adapters/blocking_connection.py", line 206, in connect
    if not self._adapter_connect():
  File "build/bdist.linux-x86_64/egg/pika/adapters/blocking_connection.py", line 274, in _adapter_connect
    if not super(BlockingConnection, self)._adapter_connect():
  File "build/bdist.linux-x86_64/egg/pika/adapters/base_connection.py", line 105, in _adapter_connect
    0, 0, socket.getprotobyname("tcp"))
gaierror: [Errno -8] Servname not supported for ai_socktype
*** format_exception:
['Traceback (most recent call last):\n', '  File "test.py", line 37, in process_task\n    r.connect(str(\'localhost\'))\n', '  File "/home/eandersson/test_project/lib/connectors/rabbitmqbase.py", line 22, in connect\n    host=host, port=5672))\n', '  File "build/bdist.linux-x86_64/egg/pika/adapters/blocking_connection.py", line 107, in __init__\n    super(BlockingConnection, self).__init__(parameters, None, False)\n', '  File "build/bdist.linux-x86_64/egg/pika/adapters/base_connection.py", line 62, in __init__\n    on_close_callback)\n', '  File "build/bdist.linux-x86_64/egg/pika/connection.py", line 590, in __init__\n    self.connect()\n', '  File "build/bdist.linux-x86_64/egg/pika/adapters/blocking_connection.py", line 206, in connect\n    if not self._adapter_connect():\n', '  File "build/bdist.linux-x86_64/egg/pika/adapters/blocking_connection.py", line 274, in _adapter_connect\n    if not super(BlockingConnection, self)._adapter_connect():\n', '  File "build/bdist.linux-x86_64/egg/pika/adapters/base_connection.py", line 105, in _adapter_connect\n    0, 0, socket.getprotobyname("tcp"))\n', 'gaierror: [Errno -8] Servname not supported for ai_socktype\n']

All I do to reproduce this is open and close a large amount of connections. Eventually I will run into this exception on my CentOS boxes.

[gmr]

This is pretty odd, few things:

1. Is IPv6 Enabled?
2. What are the entries from /etc/hosts for localhost for IPv4 and IPv6?

In theory it may be a problem with IPv6 being disabled but the first entry in hosts for localhost. I'll add an exception handler for this at the right spot, but it'd be good to know what the issue is that is causing it.

[eandersson]

1. I double checked and IPv6 is enabled on my VM.

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

It is indeed not a major issue, since I can simply catch it and open a new connection, but I have reproduced this on multiple servers. The real odd thing is that it isn't happening on earlier versions.

It does also not make any difference if I use localhost, 127.0.0.1 or the actual IP address of the blade.

[gmr]

The fact that we're using this when socket.SOCK_STREAM and the documentation at http://docs.python.org/2/library/socket.html#socket.getprotobyname indicates it is probably not needed. I'll dig into some of my IP books to see if this is the case.

[gmr]

So in my testing it appears this happens when there is a DNS resolution failure, oddly enough. I've refactored the code a small bit and added logging when this happens and why.

If you can test with this latest patch, it should detail why the problem is happening in pika.adapters.base_connection at the CRITICAL logging level.

[eandersson]

It does indeed look like removing getprotobyname resolved this.