Add possibility to use client cert. with tracking API.

[PhilipMay]

see #2574

TODO

• tests
• documentation - https://mlflow.org/docs/latest/tracking.html#logging-to-a-tracking-server
• python docstring of MlflowHostCreds

[PhilipMay]

@dbczumar
Before I continue with this PR I would like to have some feedback.
In this PR I only plan to introduce client cert usage with environment variable.
I do not plan to open up the python tracking API.

Disregarding the missing documentation and tests: Is this implementation ok or do you see more needed changes on other places?

[dbczumar]

@PhilipMay Sorry for the delay here. This implementation looks great! Let me know if you can add some tests and documentation - more than happy to get this merged once those have been added! Thank you for your contribution!

[codecov-commenter]

Codecov Report

Merging #2843 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2843   +/-   ##
=======================================
  Coverage   85.04%   85.04%           
=======================================
  Files          20       20           
  Lines        1050     1050           
=======================================
  Hits          893      893           
  Misses        157      157           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 34e8174...9590c1e. Read the comment docs.

[PhilipMay]

I am finished with this PR.

• Tests written - positive and negative
• Docstring for python
• Documentation

CI Tests are all green...

@dbczumar
Can you please review this and give feedback?

Thanks,
Philip

[PhilipMay]

Hi. Any comments on this PR?

[dbczumar]

@PhilipMay This is a really minor point of feedback, but can we start this error message by indicating that ignore_tls_verification=True and server_cert_path should not both be specified and include the environment variable part afterwards as a hint? Users or some other piece of backend code may try to construct this object directly, without the use of environment variables.

Also, can we specify an error code of INVALID_PARAMETER_VALUE here? See https://github.com/mlflow/mlflow/blob/master/mlflow/utils/uri.py#L55 for an example.

[dbczumar]

@PhilipMay Sorry for the delay again - this looks awesome! I left one small comment about the error message behavior. I'm going to give this a manual test tomorrow and should be ready to approve / merge it by EOD tomorrow!

[PhilipMay]

The exception message has been improved. The error code is still missing.

2 lines above my exception is an other MlflowException without error code. Since I am not familiar with the proto stuff like in the example I am not sure how exactly to implement it...
Could you maybe add this after the PR?

@dbczumar What do you think?

[PhilipMay]

I'm going to give this a manual test tomorrow and should be ready to approve / merge it by EOD tomorrow!

If you have problems with the manual test I can provide an howto. I start the server with this command:

mlflow server --gunicorn-opts "--keyfile=/home/mlflow/mlflow-cert/server.key --certfile=/home/mlflow/mlflow-cert/server.crt --cert-reqs=2 --ca-certs=/home/mlflow/mlflow-cert/client.crt" --backend-store-uri postgresql://mlflow:<password>@127.0.0.1/mlflow?sslmode=require --default-artifact-root /home/mlflow/default-artifact-root -h 0.0.0.0

Here is the demo client code:

import mlflow
import os

os.environ["MLFLOW_TRACKING_CLIENT_CERT_PATH"] = "/<some_dir>/client.pem"
os.environ["MLFLOW_TRACKING_SERVER_CERT_PATH"] = "/<some_dir>/server.crt"

mlflow.set_tracking_uri("https://<URL>:5000")
mlflow.set_experiment("cert-test-01")

for i in range(4):
    with mlflow.start_run():
        mlflow.log_metric('x', 2*i)
        mlflow.log_metric('y', 100/((i+1)*2))
        mlflow.set_tag('t1', 'tag:{}'.format(i))
        mlflow.log_param('p1', 'param:{}'.format(i))

[dbczumar]

@PhilipMay I've pushed the tweaks you mentioned for the INVALID_PARAMETER_VALUE error code; I've tested this manually and found that it works as expected! I'll go ahead and merge this once the tests pass!

[dbczumar]

LGTM! Thanks so much @PhilipMay !