xmlrpccpp spins forever when no more file handles

[trainman419]

When a ROS node runs out of available file handles, it can go into an infinite spin loop if it gets an incoming xmlrpc request.

We found this when executing the rosprofiler nodes against our system, so this is partially blocking us from running rosprofiler.

I've put a small test program that demonstrates this: https://github.com/trainman419/ros_infinite_loop ; compile and run the test.launch and top to see the loop node spin at 100%

I've done some very informal profiling and it looks like the program is spending all of its time in ros::CallbackQueue::callAvailable.

It also looks like the socket close here should not be commented out, and this may be contributing to leaked sockets:

ros_comm/utilities/xmlrpcpp/src/XmlRpcServer.cpp

Line 157 in ea13207

//this->close();

cc @Knio

[trainman419]

This is preventing my company from using the ROS profiling tools, and it's rather annoying. Does someone have time to take a look at this?

[wjwwood]

I'm sorry to say, but I don't have time right now, and I wouldn't count on any of the maintainers looking into it before the end of the year, but I could be wrong about that. Maybe someone out in the community could dig into it, especially since you set up a reproducible example (thanks for that btw).

Since you spent the time to make the reproducible example and make a suggestion of where the issue might be, I guess you guys spent some time trying to fix it and are stumped? Unfortunately, I'm not very familiar with the XMLRPC code that is embedded (😞) in ros_comm, so I probably can't give any suggestions or clues without spending a huge amount of time researching the issue. I think @ruffsl spent some time in that part of the code while he was looking at SROS recently. But he might not have time or insight into it either.

I assume it's not an option for your company to spend more time to dig into the issue and find the solution? If so that's a shame, because it would help your company and everyone else in the ROS community! Alternatively, you could raise the visibility of this by posting it on http://discourse.ros.org/ and seeing if anyone else finds it annoying enough to help try and fix it.

[ruffsl]

Well my exploration of ROS's xmlrpc for SROS has been mainly been confined to rospy, but I just tested example, and was able to reproduce the issue on my local machine:

$ apt-cache policy ros-kinetic-ros-comm 
ros-kinetic-ros-comm:
  Installed: 1.12.6-0xenial-20161026-193907-0700
  Candidate: 1.12.6-0xenial-20161026-193907-0700
  Version table:
 *** 1.12.6-0xenial-20161026-193907-0700 500
        500 http://packages.ros.org/ros/ubuntu xenial/main amd64 Packages
        100 /var/lib/dpkg/status

Needed to make a small change to the cmake to compile:
trainman419/ros_infinite_loop#1

@trainman419 , I tried uncommenting out that line you pointed to in this dockerfile.
I don't see the process pegging out anymore, but I do see the connection was refused.

ERROR: connection refused to [http://0e01582fb6e7:34856/]

Here was my build steps:

git clone -b docker https://github.com/ruffsl/ros_infinite_loop.git
cd ros_infinite_loop/docker
docker build -t ros:loop .
docker run --rm -it ros:loop /ros_infinite_loop.sh

[trainman419]

We don't have a lot of time to work on this either. I may try to fix this on my company's copy of ros_comm, but the last time I submitted a pull request to ros_comm was not a pleasant experience, so I'm not motivated to submit that fix upstream.

[wjwwood]

I may try to fix this on my company's copy of ros_comm, but the last time I submitted a pull request to ros_comm was not a pleasant experience, so I'm not motivated to submit that fix upstream.

I'm sorry to hear that. If there's anything I can do to motivate you (or your company) to do so when and if you come up with a fix, let me know.

[mgrrx]

I reproduced the problem today (Thanks @trainman419 for the reproducible example!). Basically I see here two different issues:

• running out of available file descriptors and
• not handling the problem when running out of available file descriptors.

The first problem is another instance of #610, the second is not yet addressed and I'll look into this issue in the coming days. I've an open pull request for #610 in #834, but it's not yet ready to be merged as discussed with @dirk-thomas. Any contribution on #834 is highly welcome and I can confirm that it fixes the first of the two problems (i.e. you are no longer running out of available file descriptors with this fix).