which trained models can be used?

[statgirl101]

Hi There,
I am new to using the what-if tool. I would like to use it to see if my ML model is fair or not; I already have a trained XGBoost model (booster object, saved model). How can I use this model with the what-if tool and is this even possible? I notice in your code that can be modified by users the WIT-from scratch.ipynb that you use classifier = tf.estimator.LinearClassifier
What if I already have a saved model that is of the form I mentioned above (xgboost)? Can I still use the what-if tool or not. I am concerned that tensorflow does not support these types of booster object type models. Any help would be appreciated!!!! Thanks!

[jameswex]

Non-TF models can now be used in the What-If Tool through its colab/jupyter notebook mode. See https://github.com/PAIR-code/what-if-tool/blob/master/WIT_Toxicity_Text_Model_Comparison.ipynb (usable in colab through https://colab.research.google.com/github/PAIR-code/what-if-tool/blob/master/WIT_Toxicity_Text_Model_Comparison.ipynb) to see how to provide WIT with a custom prediction function so you can do the model prediction in xgboost.

Documentation on WIT in notebook mode can be found here: https://github.com/tensorflow/tensorboard/blob/master/tensorboard/plugins/interactive_inference/README.md#notebook-mode-details

[henningsway]

These links are really helpful, and the tool seems awesome!

It seems, that in general the datastructure for the test data used by tensorflow/What-If Tool (list of dictionaries), has to be converted to be accepted by keras_model.predict() (or basically any scoring function).

So far, my models accept and score data frame input, which I have transformed to the tensorflow example format using the helper function df_to_examples. Would I have to backtransform this data into dataframe format, before I can feed it into my custom prediction function (via set_custom_predict_fn)? Maybe https://www.tensorflow.org/api_docs/python/tf/estimator/inputs/pandas_input_fn is useful here?

It would be great to have an example for this usecase.

[tispratik]

Can the tool be used on an SSD based object detection model trained on Tensorflow 1.10?

[jameswex]

@tispratik what is the output of that detection model? Is it just a classification problem in that classes that can be detected in an image get scores based on the likelihood of those classes being in the input image? If so then WIT should be usable. If the model returns bounding boxes around objects, WIT would not work as WIT is only usable for regression and classification problems.

For WIT to recognize an input feature as an image and properly display it, the image must be stored in the input tf.Example proto as a base64 encoded string, in a feature with the name "image/encoded".

One example of using WIT to analyze an image-based model can be found in https://colab.sandbox.google.com/github/PAIR-code/what-if-tool/blob/master/WIT_Smile_Detector.ipynb

[tispratik]

Thanks @jameswex for the information. Yes the output of the detection model is bounding boxes and classes. I think I can do some pre-processing and extract the cropped images from bounding boxes and then feed it into what-if-tool as a classification problem. Do you think that will work?

[jameswex]

if you were able to create a version of the model that takes in images and outputs classification scores, then WIT should work, much like the smile demo shows.

[tispratik]

Awesome! Thats good to know.

[tispratik]

Btw, forgot to ask about one more thing. What would be feature labels be like in that case? In case of image classification or object detection, the features like edges, corners, etc are computed on the fly... would it work in the Datapoint editor?

[jameswex]

The datapoint editor should show whatever features are the input to the model itself, so if that is the image, it would be the image. If model also takes in some coordinates, those can be represented as floating point or int features.

We haven't used WIT with images except for a simple binary classification (like smile detection), so this would certainly be a new use case for WIT.