Loading collector twice causes failure of task creation

[ghost]

Description

The issue appears when one accidentally attempts to load already loaded collector plugin. Framework informs that plugin has been already loaded, but after that creating a task fails. snapd needs to be restarted and all the plugins loaded properly from the beginning in order to run a task. This is quite confusing as it seems that something is wrong with the collector.

Steps to reproduce this error:

1. Run Snap:
   snapd -l 1 -t 0
2. Load collector plugin (tested with iostat, ethtool)
   snapctl plugin load snap-plugin-collector-iostat/build/rootfs/snap-plugin-collector-iostat
3. 'Accidentally' load collector plugin once again
   snapctl plugin load snap-plugin-collector-iostat/build/rootfs/snap-plugin-collector-iostat
4. Load mock-file-publisher and processor-passthru
5. Create a task:
   iostat.txt

Result

Task is created but does not collect metrics.

[IRCody]

I was able to reproduce this on master so labeling as a bug. Haven't been able to identify the exact cause yet but looking into it.

[lynxbat]

Nice catch @mkuculyma. This may be the best bug this quarter.

[IRCody]

Looking at it I think it's related to this line. A reference to the loaded plugin is added to the metric catalog before we verify if a plugin with a matching key is already loaded. For metrics it allows multiple writes with the same metrics, but for plugins it does not allow it so it happily overwrites the metrics in the catalog (and the plugin associated with them). The plugin it saves there ends up erroring because it is already loaded, and it's executable is removed.

When the task is created we look at the metric catalog and gather the removed plugin's info from it, which gives the error: Error creating task:open /tmp/786791258/snap-plugin-collector-iostat: no such file or directory.