fix(run_nodetool): fix coredump on timeout

[soyacz]

There are 2 reasons of coredump not creating on timeout error when running nodetool (e.g. in upgrade test when we run: node.run_nodetool("drain", timeout=3600, coredump_on_timeout=True):

1. kind of race condition when SSHReaderThread got end event (due timeout) but reader didn't acknowledge that it occured yet. This even lead to sometimes freeze of whole ssh command until CriticalError killed ssh connection.
2. wrong exception handling when running nodetool

This commit fixes both problems.

PR pre-checks (self review)

• I followed KISS principle and best practices
• I didn't leave commented-out/debugging code
• I added the relevant backport labels
• New configuration option are added and documented (in sdcm/sct_config.py)
• I have added tests to cover my changes (Infrastructure only - under unit-test/ folder)
• All new and existing unit tests passed (CI)
• I have updated the Readme/doc folder accordingly (if needed)

[soyacz]

@roydahan this is fix for missing coredump when nodetool drain freezes: scylladb/scylladb#11468

btw. Is there a reason we wait 1h before timing out on node drain?

[fgelcer]

LGTM

[roydahan]

@roydahan this is fix for missing coredump when nodetool drain freezes: scylladb/scylladb#11468

btw. Is there a reason we wait 1h before timing out on node drain?

I don't think so, it sounds way too much to me.
Please check if any previous commit message explains it, and if not lower it to something more reasonable. (15 mins?)

[roydahan]

This PR introduced issue #5418.
The parameter of "coredump_on_timeout" is now completely ignored.