Skip to content
A list of references for MLOps
Branch: master
Clone or download

Latest commit

Latest commit 6c39f27 Mar 25, 2020


Type Name Latest commit message Commit time
Failed to load latest commit information. mlops on azure Mar 25, 2020

MLOps References

  1. Book "Building Machine Learning Powered Applications" by Emmanuel Ameisen
  2. Book "Managing Data Science" by Kirill Dubovikov
  3. Book “Machine Learning Engineering” by Andriy Burkov
  4. Book "Accelerated DevOps with AI, ML & RPA: Non-Programmer's Guide to AIOPS & MLOPS" by Stephen Fleming
  5. Book "Evaluating Machine Learning Models" by Alice Zheng
  6. Continuous Delivery for Machine Learning (by Thoughtworks)
  7. Linux Foundation AI Foundation
  8. MLOps SIG Specification
  9. MLSpec: A project to standardize the intercomponent schemas for a multi-stage ML Pipeline.
  10. Oreilly learning resources: Operationalize ML
  11. Awesome production machine learning: State of MLOps
  12. State of Enterprise ML 2019: PDF
  13. State of Enterprise ML 2019: Interactive
  14. Udemy “Deployment of ML Models”
  15. MLOps NY conference 2019
  16. Gartner AI Trends 2019
  17. Organizing machine learning projects: project management guidelines.
  18. Rules for ML Project (Best practices)
  19. ML Pipeline Template
  20. Data Science Project Structure
  21. Reproducible ML
  22. ML project template facilitating both research and production phases.
  23. Machine learning requires a fundamentally different deployment approach. As organizations embrace machine learning, the need for new deployment tools and strategies grows.
  24. What are model governance and model operations? A look at the landscape of tools for building and deploying robust, production-ready machine learning models
  25. Specialized tools for machine learning development and model governance are becoming essential. Why companies are turning to specialized machine learning tools like MLflow.
  26. Efficient ML engineering: Tools and best practices
  27. Why is DevOps for Machine Learning so Different?
  28. Lessons learned turning machine learning models into real products and services – O’Reilly
  29. MLOps: Model management, deployment and monitoring with Azure Machine Learning
  30. Guide to File Formats for Machine Learning: Columnar, Training, Inferencing, and the Feature Store
  31. Architecting a Machine Learning Pipeline How to build scalable Machine Learning systems
  32. What are model governance and model operations? – O’Reilly
  33. Why Machine Learning Models Degrade In Production
  34. Concept Drift and Model Decay in Machine Learning
  35. Bringing ML to Production
  36. Global AI Survey: AI proves its worth, but few scale impact
  37. ML in Production
  38. A Tour of End-to-End Machine Learning Platforms
  39. Full Stack Deep Learning
  40. What Does it Mean to Deploy a Machine Learning Model?
  41. Software Interfaces for Machine Learning Deployment
  42. Batch Inference for Machine Learning Deployment
  43. MLOps: Continuous delivery and automation pipelines in machine learning
  44. AI meets operations
  45. What would machine learning look like if you mixed in DevOps? Wonder no more, we lift the lid on MLOps
  46. Forbes: The Emergence Of ML Ops
  47. Cognilytica Report "ML Model Management and Operations 2020 (MLOps)"
  48. Introducing Cloud AI Platform Pipelines
  49. A Guide to Production Level Deep Learning
  50. The 5 Components Towards Building Production-Ready Machine Learning Systems
  51. Spring 2019 Full Stack Deep Learning Bootcamp
  52. Deep Learning in Production (references about deploying deep learning-based models in production)
  53. Book "Building Machine Learning Pipelines" – O’Reilly (Release Date: August 2020)
  54. Machine Learning Experiment Tracking
  55. The Team Data Science Process (TDSP)
  56. So You Want Git for Data?
  57. MLOps Solutions (Azure based)


  1. Building a Reproducible Machine Learning Pipeline
  2. A Systems Perspective to Reproducibility in Production Machine Learning Domain
  3. Hidden Technical Debt in Machine Learning Systems
  4. Scaling Machine Learning as a Service (Uber)
  5. What’s your ML Test Score? A rubric for ML production systems
  6. Adversarial Machine Learning Reading List
  7. Workshop on MLOps Systems. 2020 Third Conference on Machine Learning and Systems (MLSys)
  8. sensAI: Fast ConvNets Serving on Live Data via Class Parallelism. Guanhua Wang, Zhuang Liu, Siyuan Zhuang, Brandon Hsieh, Joseph Gonzalez and Ion Stoica.
  9. Towards Automated ML Model Monitoring: Measure, Improve and Quantify Data Quality. Tammo Rukat, Dustin Lange, Sebastian Schelter and Felix Biessmann.
  10. Towards Automating the AI Operations Lifecycle. Matthew Arnold, Jeff Boston, Michael Desmond, Evelyn Duesterwald, Benjamin Elder, Anupama Murthi, Jiri Navratil and Darrell Reimer.
  11. Efficient Scheduling of DNN Training on Multitenant Clusters. Deepak Narayanan, Keshav Santhanam, Amar Phanishayee and Matei Zaharia.
  12. Towards Complaint-driven ML Workflow Debugging. Weiyuan Wu, Lampros Flokas, Eugene Wu and Jiannan Wang.
  13. PerfGuard: Deploying ML-for-Systems without Performance Regressions. H M Sajjad Hossain, Lucas Rosenblatt, Gilbert Antonius, Irene Shaffer, Remmelt Ammerlaan, Abhishek Roy, Markus Weimer, Hiren Patel, Marc Friedman, Shi Qiao, Peter Orenberg, Soundarajan Srinivasan and Alekh Jindal.
  14. Implicit Provenance for Machine Learning Artifacts. Alexandru A. Ormenisan, Mahmoud Ismail, Seif Haridi and Jim Dowling.
  15. Addressing the Memory Bottleneck in AI Model-Training. David Ojika, Bhavesh Patel, G Anthony Reina, Trent Boyer, Chad Martin and Prashant Shah.
  16. Simulating Performance of ML Systems with Offline Profiling. Hongming Huang, Peng Cheng, Hong Xu and Yongqiang Xiong.
  17. A Viz Recommendation System: ML Lifecycle at Tableau. Kazem Jahanbakhsh, Eric Borchu, Mya Warren, Xiang-Bo Mao and Yogesh Sood.
  18. CodeReef: an open portal for cross-platform MLOps and reproducible benchmarking. Grigori Fursin, Herve Guillou and Nicolas Essayan.
  19. Towards split learning at scale: System design. Iker Rodríguez, Eduardo Muñagorri, Alberto Roman, Abhishek Singh, Praneeth Vepakomma and Ramesh Raskar.
  20. MLBox: Towards Reproducible ML. Victor Bittorf, Xinyuan Huang, Peter Mattson, Debojyoti Dutta, David Aronchick, Emad Barsoum, Sarah Bird, Sergey Serebryakov, Natalia Vassilieva, Tom St. John, Grigori Fursin, Srini Bala, Sivanagaraju Yarramaneni, Alka Roy, David Kanter and Elvira Dzhuraeva.
  21. Conversational Applications and Natural Language Understanding Services at Scale. Minh Tue Vo Thanh and Vijay Ramakrishnan.
  22. Towards Distribution Transparency for Supervised ML With Oblivious Training Functions. Moritz Meister, Sina Sheikholeslami, Robin Andersson, Alexandru Ormenisan and Jim Dowling.
  23. Tools for machine learning experiment management. Vlad Velici and Adam Prügel-Bennett.
  24. MLPM: Machine Learning Package Manager. Xiaozhe Yao.
  25. Common Problems with Creating Machine Learning Pipelines from Existing Code. Katie O’Leary, Makoto Uchida.


  1. DeliveryConf 2020. "Continuous Delivery For Machine Learning: Patterns And Pains" by Emily Gorcenski
  2. MLOps Conference: Talks from 2019
  3. A CI/CD Framework for Production Machine Learning at Massive Scale (using Jenkins X and Seldon Core)

Existing ML Systems

  1. Introducing FBLearner Flow: Facebook’s AI backbone
  2. TFX: A TensorFlow-Based Production-Scale Machine Learning Platform
  3. Getting started with Kubeflow Pipelines
  4. Meet Michelangelo: Uber’s Machine Learning Platform
  5. Meson: Workflow Orchestration for Netflix Recommendations
  6. What are Azure Machine Learning pipelines?

Machine Learning

  1. Book, Aurélien Géron,"Hands-On Machine Learning with Scikit-Learn and TensorFlow"
  2. Foundations of Machine Learning
  3. Best Resources to Learn Machine Learning
  4. Zhi-Hua Zhou. 2012. Ensemble Methods: Foundations and Algorithms. Chapman & Hall/CRC.
  5. Feature Engineering for Machine Learning. Principles and Techniques for Data Scientists.By Alice Zheng, Amanda Casari
  6. Google Research: Looking Back at 2019, and Forward to 2020 and Beyond
  7. O’Reilly: The road to Software 2.0
  8. Machine Learning and Data Science Applications in Industry
  9. Deep Learning for Anomaly Detection
  10. Federated Learning for Mobile Keyboard Prediction
  11. Federated Learning. Building better products with on-device data and privacy on default
  12. Federated Learning: Collaborative Machine Learning without Centralized Training Data

Software Engineering

  1. The Twelve Factors
  2. Book "Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations", 2018 by Nicole Forsgren
  3. Book "The DevOps Handbook" by Gene Kim, et al.

The Economics of ML/AI

  1. Book: "Prediction Machines: The Simple Economics of Artificial Intelligence"
  2. A list of articles about AI and the economy
  3. Getting started with AI? Start here! Everything you need to know to dive into your project
  4. 11 questions to ask before starting a successful Machine Learning project
  5. What AI still can’t do
  6. Demystifying AI Part 4: What is an AI Canvas and how do you use it?
  7. A Data Science Workflow Canvas to Kickstart Your Projects
  8. Is your AI project a nonstarter? Here’s a reality check(list) to help you avoid the pain of learning the hard way
  9. What is THE main reason most ML projects fail?
  10. Designing great data products. The Drivetrain Approach: A four-step process for building data products.
  11. The New Business of AI (and How It’s Different From Traditional Software)
  12. The idea maze for AI startups
  13. The Enterprise AI Challenge: Common Misconceptions
  14. Misconception 1 (of 5): Enterprise AI Is Primarily About The Technology
  15. Misconception 2 (of 5): Automated Machine Learning Will Unlock Enterprise AI
You can’t perform that action at this time.