Intermittent Failure to Load Library Under High Thread Contention

[Jmeyer1292]

This issue is related to the one I opened up against pluginlib, found here. For context, I am running Ubuntu 14.04 and ROS-Indigo. I am writing an application that (for better or worse) ends up loading dozens of plugins per thread across up to several hundred threads. In the process of this development, the loading of plugins has become an issue. I can fix my issue by serializing the construction of plugins in my code, but I'd like to get it at the source. Pluginlib has its own issues, but it appears that (at least some of) the problem extends to this library.

I can produce occasional crashes under high contention with the following code:

#include <iostream>
#include <boost/shared_ptr.hpp>
#include <boost/thread.hpp>
#include <boost/ref.hpp>

#include <class_loader/class_loader.h>
#include <pluginlib_tutorials/polygon_base.h>

typedef boost::shared_ptr<polygon_base::RegularPolygon> RegularPolygonPtr;

// Experiment 1: Each worker has its own class loader
void workerThread1(const std::string& library_path, const std::string& class_name, int n)
{
  class_loader::ClassLoader loader (library_path);
  std::vector<RegularPolygonPtr> plugins;

  for (int i = 0; i < n; ++i)
  {
    RegularPolygonPtr plugin = loader.createInstance<polygon_base::RegularPolygon>(class_name);
    if (plugin)
      plugin->initialize(10.0);
    else
      std::cerr << "ERROR\n";
    plugins.push_back(plugin);
  }

  boost::this_thread::sleep(boost::posix_time::milliseconds(1000));
}

void launchExperiment1(const std::string& lib_path, const std::string& class_name, int n_threads, int n)
{
  std::vector<boost::thread*> threads;
  for (int i = 0; i < n_threads; ++i)
  {
    threads.push_back( new boost::thread(workerThread1, lib_path, class_name, n) );
  }

  for (int i = 0; i < n_threads; ++i)
  {
    threads[i]->join();
    delete threads[i];
  }
}

int main(int argc, char** argv)
{
  if (argc < 4)
  {
    std::cerr << "Usage: rosrun polygon_class_loader <path_to_library> <n_threads> <n_items_per_thread>\n";
    return 1;
  }

  const std::string library_path (argv[1]);
  class_loader::ClassLoader loader(library_path);
  std::vector<std::string> classes = loader.getAvailableClasses<polygon_base::RegularPolygon>();

  if (classes.empty())
  {
    std::cerr << "No available plugins in this library\n";
    return 2;
  }

  const std::string& class_to_load = classes.front();
  std::cout << "Choosing to load many instances of " << class_to_load << "\n";

  boost::this_thread::sleep(boost::posix_time::milliseconds(500));

  int n_threads = std::atoi(argv[2]);
  int n_items_per_thread = std::atoi(argv[3]);

  std::cout << "Starting multi-threaded loading experiment w/ " << n_threads << " threads " <<
            "and " << n_items_per_thread << " plugins per thread" << std::endl;

  std::cout << "Experiment 1: All threads have own class loader" << std::endl;
  launchExperiment1(library_path, class_to_load, n_threads, n_items_per_thread);
}

Drop this into the pluginlib tutorial package and build. Run with roscore and run with the following extra arguments 4000 100 which indicates 4000 threads loading 100 plugins a piece. It doesn't always trip an error, so try it a few times.

I frequently observe the following error in the middle of the run:

terminate called after throwing an instance of 'class_loader::CreateClassException'
  what():  Could not create instance of type polygon_plugins::Square
Aborted (core dumped)

This exception is coming from here. Some testing shows that a factory does exist when the plugin is created, but it's not owned by either requesting class loader or NULL.

Digging around, sure enough, it would appear that the underlying vector that keeps a list of the "owners" of class loader is not protected. Concurrent access can happen here and here for example.

Putting a lock in part of loadLibrary appears to make the crash go away. Ala:

  //If it's already open, just update existing metaobjects to have an additional owner.
  if(isLibraryLoadedByAnybody(library_path))
  {
    boost::recursive_mutex::scoped_lock lock(getPluginBaseToFactoryMapMapMutex());
    logDebug("class_loader.class_loader_private: Library already in memory, but binding existing MetaObjects to loader if necesesary.\n");
    addClassLoaderOwnerForAllExistingMetaObjectsForLibrary(library_path, loader);
    return;
  }

I'll submit a PR unless there's an objection. @mikaelarguedas.

[scpeters]

I've been seeing intermittent exceptions from MultiLibraryClassLoader::createInstance, which is thrown if getClassLoaderForClass returns NULL. Thanks for submitting a fix! Hope it works!

[mikaelarguedas]

@Jmeyer1292 thanks for the PR, this should be fixed by #40 .
Closing this for now, feel free to reopen it if crashes still occur in some cases