Investigate number of open connections for DAS/Mongo interaction

[vkuznet]

Follow up from #290.

Regarding open connections, they are connected sockets, i.e sockets between DAS and MongoDB. Just ssh to cmsweb@… and run netstat -tanlp | grep ESTABLISHED | grep 27017 to see them. We have currently: {{{
$ netstat -tanlp | grep ESTABLISHED | grep 27017 | awk '{print $NF}' | sort | uniq -c
212 4500/mongod
138 4860/python
74 4875/python
}}}
Why there are that many I can't answer. Maybe every DAS thread creates some number of connections? Note that half of the sockets are for python side, the other half is the mongod side, as shown above.
in reply to: ↑ 69

[vkuznet]

valya: I investigated this problem. The pymongo driver creates one connection per thread. I used several connection objects in a code, e.g. mongocache, analytics, mapping, etc. DAS cache server has N workers (where N=4 currently and is defined in DAS configuration file). The DAS servers are configured with CherryPy thread pool whose size is set to 30. Finally, I look at number of threads used by cache/web servers and it is 39/34 respectively. So, it is not surprise to see 138 connections for DAS cache server, who has 30 CherryPy threads, where each thread can use several connections. The number of mongod connections reflects the number of connections in DAS cache/web servers.

The remedy for this is to use singleton Connection object in DAS code which will reduce total number of ESTABLISHED connections. Please note that Connection object still uses connection pool, so I think this approach will be safe.

[vkuznet]

valya: (In 08c9af2) Work on connection pooling, fixes #529

Signed-off-by: Valentin Kuznetsov vkuznet@gmail.com