[FLINK-938] Automatically configure the jobmanager address when using start-cluster.sh

[qmlmoon]

No description provided.

[rmetzger]

I guess the rewrite of the yaml file works well with this pull request. But I think this approach is not always working.
This approach assumes that the configuration file is shared among all cluster nodes (for example via NFS).
I would implement this by passing (via the cli invocation) the Jobmanager's hostname (or ip address) to the TaskManagers when starting them and leave the yaml file untouched.

[qmlmoon]

I would like to rewrite at least the yaml file on the master node. Because when you run stratosphere, it also reads config file to get the jobmanager address. And CliFrontend already has an option -m to pass jobmanager address, which requires host:port. I think it won't be nice to add another option like default jobmanager address.

[qmlmoon]

The newest commit using cli invocation to pass jobmanager's hostname to JobManager and TaskManager. It still try to rewrite the config file on master node since CliFrontend also reads config to get jobmanager address.

[rmetzger]

I agree. Rewriting the config for the CliFrontend is good.

[StephanEwen]

Was the & removed on purpose?

[rmetzger]

Yes: https://issues.apache.org/jira/browse/FLINK-938?focusedCommentId=14036049&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14036049
It suppresses remote error messages.

[StephanEwen]

The rewriting of the config by the script seems fragile. Why is that even needed, when the taskmanagers get the jobmanager host as a parameter?

Also: What happens if someone put an address there (like a specific address to bind to? Seems that it will be just overwritten. That we cannot do.

[qmlmoon]

The rewriting of the config is because CliFrontend also reads config to get jobmanager address. If someone wants to configure the address, it will be overwritten, this is the drawback of automatically configuring the address.

[StephanEwen]

Then I think we need a bit of a different approach.

Can we remove the jobmanager rpc address from the config file, and have the script check whether there is an entry for the jobmanager address. If not, it uses the local hostname.

I would actually prefer to have something like a .jobmanager file in yarn, rather than rewriting the config. Seems much cleaner.

[StephanEwen]

I am skeptical whether it is a good idea to modify the config file at all. I bet that many people make the implicit assumption that this file is constant. And modifying it when the host is "localhost" is hidden constraint that makes it tricky to understand why the script sometimes configures things for you and sometimes not.

I vote for an approach to add this as a command line parameter to the ssh-transmitted call to start the taskmanager. The command line parser should check for that argument similarly as it checks for the config directory.

[qmlmoon]

I agree. Modifying the config file not sounds like a good approach. But otherwise user has to type like ./start-cluster.sh -m default to tell flink don't read job manager address from config file, use my hostname. Also in the run command ./flink run -m default. And the motivation of this issue was to make it more convenient.

[StephanEwen]

Why don't we comment out the jm address in the config by default? Then no entry in the config would mean that the parameter appended to the cli command should be taken...

[qmlmoon]

Ok. I have this in mind, will do it in another PR since most of the commits here would be irrelevant. Then this one could be closed.

[rmetzger]

I think only the author of the pull request (you) can close it.

[qmlmoon]

Yeah. I meant it.

[uce]

Thanks for the PR. I've looked into the other approach (#248) and I think you can safely close this on now.