Remove the TFX dependency from the Model Card Toolkit

[hanneshapke]

The TFX Addons SIG wants to move the TFX component to the TFX Addons repo. This work allows us to remove the entire TFX dependencies from this repository.

The new dependencies could look like

REQUIRED_PACKAGES = [
    'absl-py>=0.9,<1.1',
    'jinja2>=3.1,<3.2',
    'matplotlib>=3.2.0,<4',
    'jsonschema>=3.2.0,<4',
    'tensorflow-data-validation>=1.5.0,<1.11.0',
    'tensorflow-model-analysis>=1.5.0,<1.11.0',
    'tensorflow-metadata>=1.5.0,<1.11.0',
    'ml-metadata>=1.5.0,<1.11.0',
    'dataclasses;python_version<"3.7"',
]

Furthermore, we should investigate if python version requirement still persists.

'dataclasses;python_version<"3.7"',

[codesue]

Summarizing discussions that happened elsewhere:

• We'll start by removing the plain TFX dependency and upgrading other libraries to unblock using MCT with newer versions of TFX. This is being done in Remove tfx dependencies #237.
• Later, we can look into whether it's feasible to move functionality specific to MLMD, e.g. in tfx_util.py, to the tfx-addons MLMD client library. If so, we could make TFX and MLMD utils available through a [tfx] extra, for example.

[adrinjalali]

That still leaves a few tensorflow dependencies:

    'tensorflow-data-validation>=1.5.0,<1.11.0',
    'tensorflow-model-analysis>=1.5.0,<1.11.0',
    'tensorflow-metadata>=1.5.0,<1.11.0',

which I think are rather large. I suppose this library is only useful for people who use TF, and I personally don't, and don't necessarily want my envs to be so large.

Although one can produce a model card for a scikit-learn based model using this lib, having TF deps makes it not a viable option; which is fine, this is a TF centered lib, and one needs to know that. It'd be nice to make it more explicit in the main pages of the docs.

[codesue]

For what it's worth, MCT doesn't generate metrics (i.e. evaluating models and outputting metrics). It (optionally) reads and parses precomputed metrics and adds them to a model card's list of PerformanceMetrics. See source code. You can also do this manually, as in the standalone example or read metrics from, e.g., a regular Python dictionary as in this notebook. Pretty much anything standalone/manual is framework agnostic but most automations are specific to TensorFlow. I haven't seen documentation on how to write your own utils to automate annotating metrics for other frameworks, though.

I really like the idea of having different sub-modules which handle different libraries. Elsewhere, someone also brought up the idea of limiting the main model-card-toolkit module to the bare minimum needed to define and export a model card.

[adrinjalali]

I remember trying to remove these dependencies from the repo when it came out, the issue was they were using TF data structures to pass things around.

If we change that to internal containers to store metric data, then it'd be easy to decouple these dependencies. There can be a module which reads info from TF files, and returns an internal container. Would need quite a bit of refactoring though.

[codesue]

Do you remember where TF data structures where being used to pass things around? If it was custom protos, these use protobuf and don't require TensorFlow and friends. I took a closer look, and in my understanding, now, TF data structures are only used in ModelCardToolkit if a source or mlmd_source is provided. Annotation functions are still called when these are None, but they're effectively no-ops.

[codesue]

I did some testing, and I think the minimal requirements to support core functionality with little refactoring are:

• absl-py>=0.9,<1.1: primarily used for testing and building docs, used for logging in one instance -- depending on how TensorFlow docs are built, it might be possible to move this to an extra
• jinja2>=3.1,<3.2: used for rendering the model card
• jsonschema>=3.2.0,<4: used for validating JSON schema
• protobuf>=3.19.0,<3.20.0: used for building, serializing, and parsing protos -- could be made optional if exporting a model card as a proto is optional

Here, core functionality means creating ModelCard and ModelCardToolkit objects, only manually annotating graphics and performance metrics, and rendering and exporting the model card. In other words, only the functionality necessary for running the Standalone Model Card Toolkit Demo.

I made the following changes in this quick, proof-of-concept branch to try this out:

• updated required deps in setup.py and introduced a tensorflow extra
• removed from model_card_toolkit.utils.source import * from the main __init__.py and the utils/__init__.py
• used optional imports in core.py
• moved tests in core_test.py that needed TensorFlow dependencies into a new test file

I tried it out by installing the minimal deps, running the demo notebook, and testing relevant files with pytest using:

pytest model_card_toolkit \              
--ignore model_card_toolkit/core_tensorflow_test.py \
--ignore model_card_toolkit/documentation/examples/cats_vs_dogs_test.py \
--ignore model_card_toolkit/utils/graphics_test.py \
--ignore model_card_toolkit/utils/tfx_util_test.py \
--ignore model_card_toolkit/utils/testdata/testdata_utils_test.py

Then I verified that all tests passed in another virtual environment that had TensorFlow deps.

Our next release will be a major one, so we have more wiggle room with making API changes that'll make it easier to support other frameworks. It's be great to have help with design ideas, feedback and code reviews. We're in the process of updating the contributing guide, so hopefully it'll become easier for people to help with implementation. 😄

[adrinjalali]

That looks perfect!

[codesue]

Furthermore, we should investigate if python version requirement still persists.

'dataclasses;python_version<"3.7"',

Let's bump the min Python version in setup.py to 3.7 and drop this requirement for the next release.

[codesue]

The tfx dependency was removed in #248 and there's an issue for making tensorflow-xyz dependencies optional.