HDDS-2918. Increase timeout of safe-mode exit in acceptance tests

[adoroszlai]

What changes were proposed in this pull request?

1. Increase timeout for exit from safe-mode in acceptance tests
2. Fix lack of logs in case exit from safe-mode times out

https://issues.apache.org/jira/browse/HDDS-2918

How was this patch tested?

Introduced fake error in wait_for_safemode_exit locally, then ran test.sh in ozone env. Verified that containers are stopped and container logs saved after the "error".

https://github.com/adoroszlai/hadoop-ozone/runs/401101722

[elek]

Thanks the patch @adoroszlai Do you have any example when 90s was not enough?

(I am fine with increasing the timeout anyway, but interested about the build speed and why the initialization takes so long time. I found that the keytab generation is slow but that one will be fixed by #455, I am interested if there are any other problems...)

[adoroszlai]

The Jira contains an example and I've seen it quite a few times in various CI runs. But I have no information on the cause due to the log collection problem, nor on how long it would take for SCM to exit safe mode in these cases.

[elek]

Ok. thanks the answer. hadoop31 should be a new, unknown problem. There is no keytab generation file there.

Let me commit this patch anyway...

[elek]

+1. Thanks the fix @adoroszlai

[adoroszlai]

Thanks @elek for reviewing and committing it.