CA-136041: Multipathd forgets faulty paths after an SR operation

[geosharath]

The operations pbd-plug and -unplug cause 'multipathd -k "reconfigure"'.
During reconfigure all(!) maps are deleted and re-created. Multipathd
re-creates maps without paths that are temporarily down when doing this.
These paths will only restore if they timeout beyond their dev_loss_tmo
and get re-added later, as this triggers the multipathd behaviour for new
devices. Hence, any SR operation done before dev_loss_tmo the faulty paths
are lost and never added. The fix to this problem is two fold -

a) multipathd reconfigure has been modified to include even faulty paths
while recreating maps

b) devices have to be deleted from system when SR operations are performed
before dev_loss_tmo

Signed-off-by: Sharath Babu sharath.babu@citrix.com

[germanop]

We cannot force this behaviour ignoring what is passed to the function

[germanop]

Even though this is mitigating the problem with multipath reconfigure this will force us to rescan the bus
every time we need those paths.
This is an expensive operation that slows down our control path significantly.

[germanop]

I added a sleep after deleting the nodes to give time to the system to settle.
In fact, after sr-create, sr_attach is issued immediately by xapi and, in some cases, it finds devices
that are in the middle of a remove operation but still visible.
Because of that, we think we do not need to rescan the bus, finding ourselves without valid devices for multipath few lines later

[germanop]

@geosharath, I am going to close this PR for the following reasons:

• Device removal could not be as quick as expected by later calls, as explained above
• Removing the devices explicitly is something that goes against vendor specific settings specified
  in multipath configuration (on one hand we allow them to specify their favourite settings and on the other hand we device removal)
• Rescanning the bus is too expensive to be done without a valid reason
• The counterpart of this patch in multipath is dangerous, because we do not let "reconfigure" undo
  the paths that are down but if those paths are in the system while doing the mapping, the mapping
  will fail. This is a corner case less likely than the one we are trying to address but still likely.
• We do have an alternative approach