[SDK] Use Training Client without Kube Config

[andreyvelich]

This change will allow user to create TrainingClient without kube config using Client Configuration and Bearer Token.
It is useful for remote-like and enterprise usage of our SDK.

Similar to this example: https://github.com/kubernetes-client/python/blob/67f9c7a97081b4526470cad53576bc3b71fa6fcc/examples/remote_cluster.py#L31

/assign @kubeflow/wg-training-leads @tenzen-y @anencore94

[coveralls]

Pull Request Test Coverage Report for Build 3997111929

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 5 unchanged lines in 1 file lost coverage.
• Overall coverage decreased (-0.06%) to 38.906%

---

Files with Coverage Reduction	New Missed Lines	%
pkg/controller.v1/mpi/mpijob_controller.go	5	77.23%

Totals
Change from base Build 3985605283:	-0.06%
Covered Lines:	2674
Relevant Lines:	6873

---

💛 - Coveralls

[tenzen-y]

@andreyvelich Thanks for this work!
This is very useful!
/lgtm

[terrytangyuan]

This is not necessary? If client_configuration is not None, then it should take precedence over config_file

[tenzen-y]

If we remove no_config, I guess we can not inform about missing client_configuration, right?
WDYT?

[andreyvelich]

@tenzen-y is right . I did this since user can set config_file and client_configuration to add additional information about the config at the same time.

[andreyvelich]

We had discussion with @johnugeorge around this.

I guess kube-config supports all configuration that user can set via client.Configuration() object (proxy, token. etc.).
@terrytangyuan @tenzen-y Did you ever see the use-case when user wants to set custom client_configuration with existing kube-config ?

If not, we can do this:

• If user set config_file use load_kube_config().
• If user doesn't set config_file and doesn't set client_configuration use load_kube_config() or load_incluster_config().
• If user set client_configuration skip load_kube_config() and load_incluster_config().

no_config flag might look a bit hacky.

[tenzen-y]

No, I have not seen that use case.

[andreyvelich]

@tenzen-y @terrytangyuan @johnugeorge I modified the PR to ignore kube config when client_configuration is set.
Let's see if we get any user request to change it back.

[tenzen-y]

@andreyvelich Is it possible to set a persistent_config with a client_configuration?

[andreyvelich]

I think persistent_config is used when we send client_configuration to load_kube_config() for Kubernetes python client.

[tenzen-y]

It makes sense.

[tenzen-y]

Thanks to @andreyvelich!

/lgtm

Also, can we introduce the same improvement to Katib?

https://github.com/kubeflow/katib/blob/a44aaea7f2c60fc203ceca1f31403a5351911858/sdk/python/v1beta1/kubeflow/katib/api/katib_client.py#L31-L61

[andreyvelich]

Also, can we introduce the same improvement to Katib?

Sure, I submit PR before the release once @terrytangyuan @johnugeorge agree on that change.

[johnugeorge]

LGTM
/assign @terrytangyuan

[terrytangyuan]

Thanks!

/lgtm
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich, terrytangyuan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• sdk/python/OWNERS [terrytangyuan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment