Python Api-hour - Verification Failed

[LadyMozzarella]

After running Round 10, Preview 2 3 this test failed and appears to be passing locally:

• aiohttp.web (multiple queries, fortunes, and data updates):

  gunicorn.error:ERROR 2015-02-26 14:17:59,868 glogging.py:224 => Error handling request
  Traceback (most recent call last):
  File "/home/techempower/FrameworkBenchmarks/installs/py3/lib/python3.4/site-packages/aiohttp/server.py", line 240, in start
    yield from handler
  File "/home/techempower/FrameworkBenchmarks/installs/py3/lib/python3.4/site-packages/aiohttp/web.py", line 1209, in handle_request
    resp = yield from handler(request)
  File "/home/techempower/FrameworkBenchmarks/frameworks/Python/API-Hour/hello/hello/endpoints/world.py", line 23, in db
    return JSON((yield from get_random_record(container)))
  File "/home/techempower/FrameworkBenchmarks/frameworks/Python/API-Hour/hello/hello/services/world.py", line 7, in get_random_record
    pg = yield from container.engines['pg']
  File "/home/techempower/FrameworkBenchmarks/installs/py3/lib/python3.4/asyncio/futures.py", line 390, in __iter__
    return self.result()  # May raise too.
  File "/home/techempower/FrameworkBenchmarks/installs/py3/lib/python3.4/asyncio/futures.py", line 277, in result
    raise self._exception
  File "/home/techempower/FrameworkBenchmarks/installs/py3/lib/python3.4/asyncio/tasks.py", line 236, in _step
    result = next(coro)
  File "/home/techempower/FrameworkBenchmarks/installs/py3/lib/python3.4/site-packages/aiopg/pool.py", line 26, in create_pool
    yield from pool._fill_free_pool(False)
  File "/home/techempower/FrameworkBenchmarks/installs/py3/lib/python3.4/site-packages/aiopg/pool.py", line 159, in _fill_free_pool
    **self._conn_kwargs)
  File "/home/techempower/FrameworkBenchmarks/installs/py3/lib/python3.4/site-packages/aiopg/connection.py", line 51, in connect
    yield from conn._poll(waiter, timeout)
  File "/home/techempower/FrameworkBenchmarks/installs/py3/lib/python3.4/site-packages/aiopg/connection.py", line 155, in _poll
    yield from asyncio.wait_for(self._waiter, timeout, loop=self._loop)
  File "/home/techempower/FrameworkBenchmarks/installs/py3/lib/python3.4/asyncio/tasks.py", line 371, in wait_for
    return fut.result()
  File "/home/techempower/FrameworkBenchmarks/installs/py3/lib/python3.4/asyncio/futures.py", line 277, in result
    raise self._exception
  File "/home/techempower/FrameworkBenchmarks/installs/py3/lib/python3.4/site-packages/aiopg/connection.py", line 90, in _ready
    state = self._conn.poll()
  psycopg2.OperationalError: FATAL:  remaining connection slots are reserved for non-replication superuser connections
  
  gunicorn.error:ERROR 2015-02-26 14:17:59,946 glogging.py:224 => Error handling request
  Traceback (most recent call last):
  File "/home/techempower/FrameworkBenchmarks/installs/py3/lib/python3.4/site-packages/aiohttp/server.py", line 240, in start
    yield from handler
  File "/home/techempower/FrameworkBenchmarks/installs/py3/lib/python3.4/site-packages/aiohttp/web.py", line 1209, in handle_request
    resp = yield from handler(request)
  File "/home/techempower/FrameworkBenchmarks/frameworks/Python/API-Hour/hello/hello/endpoints/world.py", line 31, in queries
    return JSON((yield from get_random_records(container, limit)))
  File "/home/techempower/FrameworkBenchmarks/frameworks/Python/API-Hour/hello/hello/services/world.py", line 23, in get_random_records
    results.append(task.result())
  
  [ 2.13 GB of this ]
  
  asyncio:WARNING 2015-02-26 14:24:06,654 selector_events.py:613 => socket.send() raised exception.
  asyncio:WARNING 2015-02-26 14:24:06,655 selector_events.py:613 => socket.send() raised exception.
  asyncio:WARNING 2015-02-26 14:24:06,655 selector_events.py:613 => socket.send() raised exception.
  asyncio:WARNING 2015-02-26 14:24:06,656 selector_events.py:613 => socket.send() raised exception.
  asyncio:WARNING 2015-02-26 14:24:06,656 selector_events.py:613 => socket.send() raised exception.
  asyncio:WARNING 2015-02-26 14:24:06,656 selector_events.py:613 => socket.send() raised exception.
  asyncio:WARNING 2015-02-26 14:24:06,656 selector_events.py:613 => socket.send() raised exception.
  asyncio:WARNING 2015-02-26 14:24:06,657 selector_events.py:613 => socket.send() raised exception.
  asyncio:WARNING 2015-02-26 14:24:06,657 selector_events.py:613 => socket.send() raised exception.
  asyncio:WARNING 2015-02-26 14:24:06,658 selector_events.py:613 => socket.send() raised exception.
  asyncio:WARNING 2015-02-26 14:24:06,658 selector_events.py:613 => socket.send() raised exception.
  asyncio:WARNING 2015-02-26 14:24:06,658 selector_events.py:613 => socket.send() raised exception.
  asyncio:WARNING 2015-02-26 14:24:06,659 selector_events.py:613 => socket.send() raised exception.
  asyncio:WARNING 2015-02-26 14:24:06,659 selector_events.py:613 => socket.send() raised exception.
  asyncio:WARNING 2015-02-26 14:24:06,659 selector_events.py:613 => socket.send() raised exception.
  asyncio:WARNING 2015-02-26 14:24:06,660 selector_events.py:613 => socket.send() raised exception.
  asyncio:WARNING 2015-02-26 14:24:06,660 selector_events.py:613 => socket.send() raised exception.
  asyncio:WARNING 2015-02-26 14:24:06,661 selector_events.py:613 => socket.send() raised exception.
  asyncio:WARNING 2015-02-26 14:24:06,661 selector_events.py:613 => socket.send() raised exception.
  asyncio:WARNING 2015-02-26 14:24:06,661 selector_events.py:613 => socket.send() raised exception.
  asyncio:WARNING 2015-02-26 14:24:06,661 selector_events.py:613 => socket.send() raised exception.
  asyncio:WARNING 2015-02-26 14:24:06,662 selector_events.py:613 => socket.send() raised exception.
  asyncio:WARNING 2015-02-26 14:24:06,662 selector_events.py:613 => socket.send() raised exception.
  BENCHMARKING PLAINTEXT ... Complete

[ludovic-gasc]

Hi,

How many visible CPUs you use to benchmark on the server ?
FYI, with this configuration, API-Hour starts CPUs*2 workers, each worker starts a connection pool with 40 connections.
In https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/config/postgresql.conf#L64, I see you have 2000 connections max.
It means two different things:

1. You have 32 visible CPUs on the server for the tests.
2. A previous test doesn't release all DB connections.

Regards.

[msmith-techempower]

We have 40 cores on the server we are running the preview tests on.

Additionally, preview tests are truly nuked from orbit and a minimum of 60s is given for the OS to reclaim any abandoned sockets etc before the next test is run; we can consider this definitely not the problem (anymore)

[ludovic-gasc]

Ok...
In postgresql.conf, could you increase max_connections to 3500 ?
I can push a PR for that.

[msmith-techempower]

I find it odd that this is the only test seemingly running into this issue; are you sure it's the postgresql setup that needs altering?

[ludovic-gasc]

I have some issues with aiopg/psycopg2 when I have too much remaining queries in the pool, it's why I use a big pool.
You can also reduce the pool size in API-Hour, but not sure of the results: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Python/API-Hour/hello/etc/hello/main/main.yaml#L9
Personally, if it's possible to increase the max connection in pgsql, I prefer.

[msmith-techempower]

Pinging @hamiltont @bhauer to see if we have an opinion on this.

[hamiltont]

Modifying the postgres setup to increase max connections seems OK to me - there's probably no noticeable difference in postgres performance from just changing that value, and we restart the DB between each framework anyway.