To apply image classification and object detection on large scale of images, several libraries have been introduced to integrate deep learning frameworks such as Cognitive Toolkit (CNTK), TensorFlow, BigDL for purpose into Apache Spark. Among these Deep Learning Pipeline has become one popular choice.
Deep Learning Pipelines provides easy-to-use high-level APIs for scalable deep learning in Python with Apache Spark. It is an open source library created by Databricks, and supporting ensorFlow and TensorFlow-backed Keras workflows. In particular, it can be used with pre-trained models as well as transfer learning.
https://github.com/databricks/spark-deep-learning
Software required:
- Spark 2.3,
- Python 3.6
- pyspark
- TensorFlow
- Deep-Learning-Pipeline from Databricks
$SPARK_HOME/bin/spark-shell --packages databricks:spark-deep-learning:1.2.0-spark2.3-s_2.11
Databricks runtime features XGBoost, scikit-learn, and numpy as well as popular Deep Learning frameworks such as TensorFlow, Keras, Horovod, and their dependencies.
- Databricks on AWS
- Azure Databricks
Applying Pre-trained Models for Scalable Prediction
Transfer Learning
Local Cloud with Azue/AWS
Using TensorFlow model
Spark Deep Learning
https://databricks.com/blog/2017/06/06/databricks-vision-simplify-large-scale-deep-learning.html
Making Image Classification Simple With Spark Deep Learning
Deep Learning With Apache Spark
Image Data Support in Apache Spark
Tutorial
https://towardsdatascience.com/deep-learning-with-apache-spark-part-1-6d397c16abd
https://towardsdatascience.com/deep-learning-with-apache-spark-part-2-2a2938a36d35
http://www.adaltas.com/en/2018/05/29/spark-tensorflow-2-3/
Databricks-Deep-Learning-Pipeline
https://databricks.com/blog/2016/01/25/deep-learning-with-apache-spark-and-tensorflow.html
https://databricks.com/blog/2016/01/25/deep-learning-with-apache-spark-and-tensorflow.html
https://docs.databricks.com/applications/deep-learning/spark-integration.html
Microsoft
https://blogs.technet.microsoft.com/machinelearning/2018/03/05/image-data-support-in-apache-spark/