[AutoTuner]Improve ETCD fault tolerance

[Caozhou1995]

PR types

Others

PR changes

Others

Description

PCard-76733
Improve ETCD fault tolerance.
In the AutoTuner scenario, multiple tasks will be launched. In multi machine scenarios, there may be instability in ETCD communication, such as poor connectivity. This PR add retry operation to avoid program crashes due to instability.
At the same time, this PR has fixed some bugs and upgrade autotuner as follows：

1. fix memory grep error due to rocm memory upgrade.
2. fix metric none error due to the first error
3. remove mp degree > 8 prune rule.
4. change the memory monitoring interval to 5 second.
5. add run best control filed.

[paddle-bot]

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[gongweibao]

The explanation needs to be more detailed?

1. What's the problem?
2. How to resolve them?

[Caozhou1995]

The explanation needs to be more detailed?

1. What's the problem?
2. How to resolve them?

1. In the AutoTuner scenario, multiple tasks will be launched. In multi machine scenarios, there may be instability in ETCD communication, such as poor connectivity. 2. Add retry operation to avoid program crush. @gongweibao

[zhiqiu]

LGTM