Friction log for TFX Chicago taxi cab example on minikube

[jlewi]

We'd love for some folks to try running the TFX Chicago Taxi Example
https://github.com/tensorflow/model-analysis/tree/master/examples/chicago_taxi

and providing a log of any pain points or problems they encounter trying to run it.

We are especially interested in the experience deploying and running on minikube as that's were folks will start.

The goal would be to start with deploying minikube and capture any information needed to properly setup minikube for kubeflow (e.g disk size). As noted in #502 we want to provide guidance on how to setup minikube for Kubeflow.

Then go through all the steps in the example for running locally and note any problems or difficulties you encounter.

It would also be fantastic to run it on other Kubeflow distributions.

[Maerville]

1. if while installing kubeflow person wants to install jupyter hub as well, default 20Gb disk-size of minikube is not enough due to large tf image
2. kubectl get svc -n=${NAMESPACE} -- this command shows outdated results in tutorial, now it also includes k8s-dashboard and ambassadors
3. ks param set kubeflow-core jupyterNotebookPVCMount /home/jovyan/work -- this is also funny

[jlewi]

@Maerville Thank you very much.

The jupyter notebook is always running as user jovyan which is why you have that path.

[Maerville]

1. Using tf-image version 1.7.0-cpu

2. Opened terminal in Jupyter hub and cloned repository https://github.com/tensorflow/model-analysis

3. Opened chicago_taxi_tfma_local_playground.ipynb
   Visualization: Plots -- kernel died.
   At first resources were cpu-1, memory-1Gi. With memory 2Gi, 3Gi kernel died as well. 4Gi finally worked out.
   Shutdown the notebook

4. Open terminal. Run steps in "Running the local example" section.
   bash ./preprocess_local.sh -- fails. Because uses "python" command, while python is pointing to /opt/conda/bin/python and jupyter kernels are pointing to /opt/conda/envs/ipykernel_py2/bin/python. Last one has all dependencies.
   So after changing path to python in preprocess_local.sh script worked.
   Had to make same changes in train_local.sh and process_tfma_local.sh

5. Ran all cells in chicago_taxi_tfma.ipynb successfully

6. bash ./start_model_server_local.sh
   ERROR: This script requires Docker
   [kinda obvious haha]

Conclusion:

1. minimum 4Gb memory for Jupyter hub (although even 2Gb was enough for training, problems started when plotting the results).
2. Need alias python to normal pythons. (normal means the ones who have all dependencies).

This test was performed on Minikube.

[Maerville]

Performed same test on Kubeadm (1 machine, provided 4 CPU and 16Gi to Jupyter hub).
Everything works successfully (but also had to change python link)

[jlewi]

Fantastic thank you.

[jlewi]

/close