Handle error on disk.dump Salt call

[ftalbart]

Component:

operator

What happened:

Sometimes, Volume creation fails with:

Message:               call to GetVolumeSize failed: disk.dump failed (target=worker-3, path=/dev/disk/by-uuid/567c960e-dcba-4cef-b5b2-4d8bb0a3446e): Salt API failed with code 500: {"status": 500, "return": "An unexpected error occurred"}

If we enable debug log on Salt API side, we get:

[DEBUG   ] Trying to connect to: tcp://192.168.1.40:4506
[DEBUG   ] Error while processing request for: /
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/netapi/rest_cherrypy/app.py", line 858, in hypermedia_handler
    ret = cherrypy.serving.request._hypermedia_inner_handler(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 60, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/salt/netapi/rest_cherrypy/app.py", line 1300, in POST
    token=cherrypy.session.get('token')))
  File "/usr/lib/python2.7/site-packages/salt/netapi/rest_cherrypy/app.py", line 1192, in exec_lowstate
    ret = self.api.run(chunk)
  File "/usr/lib/python2.7/site-packages/salt/netapi/__init__.py", line 75, in run
    return l_fun(*f_call.get('args', ()), **f_call.get('kwargs', {}))
  File "/usr/lib/python2.7/site-packages/salt/netapi/__init__.py", line 103, in local
    return local.cmd(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/client/__init__.py", line 754, in cmd
    **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/client/__init__.py", line 358, in run_job
    raise SaltClientError(general_exception)
SaltClientError: Salt request timed out. The master is not responding. You may need to run your command with `--async` in order to bypass the congested event bus. With `--async`, the CLI tool will print the job id (jid) and exit immediately without listening for responses. You can then use `salt-run jobs.lookup_jid` to look up the results of the job in the job cache later.
[INFO    ] 10.233.132.74 - - [27/Apr/2020:15:56:05] "POST /?timeout=1 HTTP/1.1" 500 57 "" "Go-http-client/1.1"

The problem may be related to a busy Salt master and/or more than one state trying to run at once.

What was expected:

The Operator should retry this operation and eventually the Volume should be created.

Steps to reproduce

@gdemonet managed to reproduce it by creating 10 sparseloop volumes at once on the same node

Resolution proposal:

Use async call for disk.dump (it's currently sync), that way we will benefit from the auto-retry logic that we've already implemented

[slaperche-scality]

I've updated the original message using the information we got during the troubleshooting.