The recent plugins kwarg refactored the code to use a list of plugin
objects rather than plugin settings from the config. The plugin objects
were then being serialized and stored in each node's userdata when
This worked, however, if the plugin stored anything that's not
serializable after its first run, such as the list of nodes, this leads
to errors when later adding nodes in the same process. The loadbalancer
in particular is affected by this bug.
Fixed this by storing the __init__ kwargs and fully qualified class path
used to create the plugin in a __plugin_metadata__ attribute which gets
dynamically created via ClusterSetup.__new__. Updated Cluster class to
serialize a list of __plugin_metadata__ attributes (stored in
Cluster._plugins) rather than the plugin object list itself (stored in
Cluster.plugins). Updated Node.get_plugins to iterate through the
__plugin_metadata__ attributes and recreate the plugin objects.