Implement TCP transport reqchannel retry logic

[lukasraska]

What does this PR do?

Implements retry mechanism for TCP transports ReqChannel as it's currently done for ZeroMQ.

What issues does this PR fix or reference?

Fixes #59162

Previous Behavior

Once first publish fails, event is lost forever

New Behavior

Event publish is retried multiple times

Merge requirements satisfied?

[NOTICE] Bug fixes or features added to Salt require tests.

• Docs
• Changelog - https://docs.saltstack.com/en/master/topics/development/changelog.html
• Tests written/updated

Commits signed with GPG?

Yes

[lukasraska]

FYI @waynew.
I would love to get your insight into the test implementation, the currently implemented integration test requires setting fixed return_retry_timer in default conf files in order to have it at least a bit deterministic and I haven't found an easy option how to set specific opts for single test (except creating new fixture with own minion factory).

Thanks

[lukasraska]

re-run pr-ubuntu1604-py3-pytest

[mattp-]

nice 👍

[lukasraska]

Should probably be merged after #59172 , so there is no conflict with tests

[s0undt3ch]

Also, fyi, #59172 has been merged.

[lukasraska]

re-run pr-amazon2-py3-pytest

[mattp-]

bump. any reason not to merge this?

[lukasraska]

I expect due to the upcoming CVE release the merging has been postponed.

Anyway, I've rebased it, so it's up-to-date with master, updated the test with larger return_retry_tries (as the PR for that has been merged, so it could work much better without such large sleep) and changed the slowTest decorator to pytest mark, FYI @s0undt3ch

[lukasraska]

re-run pr-centos7-py3-pytest

[lukasraska]

re-run pr-freebsd122-py3-pytest

[lukasraska]

re-run pr-centos7-py3-pytest