MLMD comparison with MLFlow/Tracking

[elgalu]

Hi, please forgive my ignorance,

Is this project somehow comparable to MLFlow/Tracking ?
https://mlflow.org/docs/latest/tracking.html#tensorflow-and-keras-experimental

[benmathes]

Hey @elgalu,

(for context, I'm a product manager at google working on a few things, including MLMD. Views here are mine, not through any official google spokesperson)

Great question! There are some fascinating HCI and product strategy opinions behind the answer.

There are multiple efforts in the ML industry to solve the problem of keeping track of all your ML artifacts: Models, datasets, features, etc. The same way classical software engineering iterated through multiple ways of keeping track of code (central repos like subversion, decentralized repos like git, etc.), ML engineering is evolving the best way to keep track of ML Engineering.

IMO, ML-Flow is a model-first view. Kind of like a star-schema in RDBMs, if you're familiar. Contrast to TFX's pipeline-first view. MLMD can be part of how TFX's pipeline records the DAG history of each run. MLFlow and MLMD are both under active development, so these opinions are moving targets :-)

Each view (model first or pipeline first) can be, in theory, recovered from the other. It is easier to recover the model-first view from the pipeline-first view because it only requires hiding information (rest of the pipeline). But a model-first view may not at first store extra information outside the model like its training data, analysis, comparisons with other models trained on the same pipeline, the model's full ineage, etc.

So MLFlow is kinda coming at the problem from a different angle than ML Metadata. Pipeline-first instead of Model-first. We at google have found making the pipeline a first-class citizen more useful.

[benmathes]

Some further reading and watching on this from the relevant google teams

https://medium.com/tensorflow/from-research-to-production-with-tfx-pipelines-and-ml-metadata-443a51dac188

https://www.youtube.com/watch?v=drYM04t57tU&feature=youtu.be