Properly wait on returns in saltnado

[jacksontj]

This was broken because the behavior was to simply check the ckminions
and wait for only those returns to complete. This works assuming
ckminions is accurate (which there are many cases where it isn't, such
as syndics).

_disbatch_local's waiting on returns needs to match LocalClient's
behavior (namely that in get_iter_returns). This means we are allowed to
return when (1) we have waitged the min_wait_time (0 if not a syndic)
(2) no minions are running the job (3) all minions we saw running it are
done running the job. The only method allowed for earlier termination is
if the gather_job_timeout is exceeded.

Fixes #42659

What does this PR do?

What issues does this PR fix or reference?

Previous Behavior

Remove this section if not relevant

New Behavior

Remove this section if not relevant

Tests written?

Yes/No

Commits signed with GPG?

Yes/No

Please review Salt's Contributing Guide for best practices.

See GitHub's page on GPG signing for more information about signing commits with GPG.

[gtmanfred]

Thanks for the help with this issues @jacksontj

[gtmanfred]

we can just do

minions = {minion: False for minion in pub_data['minions']}

[jacksontj]

@gtmanfred oh, did we drop python 2.6 support? IIRC dict comprehension isn't in 2.6

[gtmanfred]

yup, we dropped 2.6 starting with 2017.7.0

[jacksontj]

Updated :)

[DmitryKuzmenko]

Awesome!

[jacksontj]

Squashed those commits, and rebased on current develop (in hopes of fixing some of the CI failures ;) )

[gtmanfred]

@jacksontj looks like you definitely fixed the syndic.

First differing element 0:
{'minion': True, 'localhost': True}
{'minion': True, 'localhost': True, 'sub_minion': True}

- [{'localhost': True, 'minion': True}]
+ [{'localhost': True, 'minion': True, 'sub_minion': True}]
?                                ++++++++++++++++++++

We are getting the subminion returning on the saltnado syndic tests, can you add that to the test case?

Thanks,
Daniel

[jacksontj]

When I run these locally I get a bunch of failures in the API tests with the localhost minion not responding. Actually, they fail with or without this patch, are these tests broken?

[gtmanfred]

The localhost minion won't be there if you don't pass --ssh to start the extra sshd process that is used to test salt-ssh, and in this case testing enable_ssh_minions: True

[gtmanfred]

The tests look good to me now.