[FLINK-6160] [flip-6] Retry JobManager/ResourceManager connection in …

[rice668]

What is the purpose of the change

When timeout comes, retry JobManager/ResourceManager connection in case of timeout

Brief change log

When timeout, invoke requestHeartbeat in HeartbeatMonitor thread. Not directly invoke notifyHeartbeatTimeout and close the connection.

Verifying this change

This change is already covered by existing tests, but did minor changes. in the TaskExecutorTest.java, change testHeartbeatTimeoutWithResourceManager behavior to while timeout, does not invoke disconnectTaskManager.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): ( no)
• The public API, i.e., is any changed class annotated with @Public(Evolving): ( no)
• The serializers: (no)
• The runtime per-record code paths (performance sensitive): ( don't know)
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no )

Documentation

• Does this pull request introduce a new feature? (no)
• If yes, how is the feature documented? (not documented)

[rice668]

@tillrohrmann Could you please take a look on this when available ? Thanks

[tillrohrmann]

Thanks for the contribution @zhangminglei. I think this is not how we can solve the problem. Instead I would suggest to simply create a new ResourceManagerConnection in ResourceManagerHeartbeatListener#notifyHeartbeatTimeout in the JobMaster.