Skip to content
Best Practices, code samples, and documentation for Computer Vision.
Jupyter Notebook Python
Branch: master
Clone or download

Latest commit

PatrickBue Merge pull request #526 from microsoft/staging
Syncing master <-> staging
Latest commit e3b455a Mar 26, 2020

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github Updated pull request template to add a check for squash merging into … Feb 12, 2020
contrib vmss setup (#512) Feb 24, 2020
data/misc Syncing staging <> master branches (#476) Jan 23, 2020
docker Staging (#390) Nov 6, 2019
scenarios Video Dataset / Model refactor + framework for action tests (#477) Mar 26, 2020
tests Video Dataset / Model refactor + framework for action tests (#477) Mar 26, 2020
utils_cv Video Dataset / Model refactor + framework for action tests (#477) Mar 26, 2020
.flake8 [#171 & #172] dsvm tool (#222) Sep 2, 2019
.gitignore Video Dataset / Model refactor + framework for action tests (#477) Mar 26, 2020
.gitmodules Add back submodules using https:// Jan 23, 2020
.pre-commit-config.yaml Adding pre-commit package to yml, more rules to flake8 lint config an… Mar 13, 2019
AUTHORS.md Update master (#425) Nov 14, 2019
CONTRIBUTING.md reorganize coding guidelines (#279) Aug 19, 2019
LICENSE Initial commit Feb 11, 2019
MANIFEST.in Make utils_cv pip-installable Aug 13, 2019
NOTICE.txt Adding re-ranking for image retrieval (#515) Mar 18, 2020
README.md Added release annoucement (#524) Mar 26, 2020
SETUP.md Update SETUP.md (#506) Feb 19, 2020
_config.yml Set theme jekyll-theme-cayman Nov 17, 2019
environment.yml Video Dataset / Model refactor + framework for action tests (#477) Mar 26, 2020
pyproject.toml set black config to use 79 chars per line Feb 28, 2019
setup.py Revert environment.yml, update README, change Python version >=3.6 Aug 13, 2019

README.md

+ March 27: Released v1.1 with new and improved
+   functionality for image retrieval, object detection, 
+   keypoint detection and action recognition.  
+   For additional details, please refer to our releases page.

Computer Vision

In recent years, we've see an extra-ordinary growth in Computer Vision, with applications in face recognition, image understanding, search, drones, mapping, semi-autonomous and autonomous vehicles. A key part to many of these applications are visual recognition tasks such as image classification, object detection and image similarity.

This repository provides examples and best practice guidelines for building computer vision systems. The goal of this repository is to build a comprehensive set of tools and examples that leverage recent advances in Computer Vision algorithms, neural architectures, and operationalizing such systems. Rather than creating implementions from scratch, we draw from existing state-of-the-art libraries and build additional utility around loading image data, optimizing and evaluating models, and scaling up to the cloud. In addition, having worked in this space for many years, we aim to answer common questions, point out frequently observed pitfalls, and show how to use the cloud for training and deployment.

We hope that these examples and utilities can significantly reduce the “time to market” by simplifying the experience from defining the business problem to development of solution by orders of magnitude. In addition, the example notebooks would serve as guidelines and showcase best practices and usage of the tools in a wide variety of languages.

These examples are provided as Jupyter notebooks and common utility functions. All examples use PyTorch as the underlying deep learning library.

Examples

This repository supports various Computer Vision scenarios which either operate on a single image:

Some supported CV scenarios

As well as scenarios such as action recognition which take a video sequence as input:

Target Audience

Our target audience for this repository includes data scientists and machine learning engineers with varying levels of Computer Vision knowledge as our content is source-only and targets custom machine learning modelling. The utilities and examples provided are intended to be solution accelerators for real-world vision problems.

Getting Started

To get started, navigate to the Setup Guide, which lists instructions on how to setup the compute environment and dependencies needed to run the notebooks in this repo. Once your environment is setup, navigate to the Scenarios folder and start exploring the notebooks.

Alternatively, we support Binder Binder which makes it easy to try one of our notebooks in a web-browser simply by following this link. However, Binder is free, and as as result only comes with limited CPU compute power and without GPU support. Expect the notebook to run very slowly (this is somewhat improved by reducing image resolution to e.g. 60 pixels but at the cost of low accuracies).

Scenarios

The following is a summary of commonly used Computer Vision scenarios that are covered in this repository. For each of the main scenarios ("base"), we provide the tools to effectively build your own model. This includes simple tasks such as fine-tuning your own model on your own data, to more complex tasks such as hard-negative mining and even model deployment.

Scenario Support Description
Classification Base Image Classification is a supervised machine learning technique that allows you to learn and predict the category of a given image.
Similarity Base Image Similarity is a way to compute a similarity score given a pair of images. Given an image, it allows you to identify the most similar image in a given dataset.
Detection Base Object Detection is a technique that allows you to detect the bounding box of an object within an image.
Keypoints Base Keypoint detection can be used to detect specific points on an object. A pre-trained model is provided to detect body joints for human pose estimation.
Action recognition Contrib Action recognition to identify in video/webcam footage what actions are performed (e.g. "running", "opening a bottle") and at what respective start/end times.
Crowd counting Contrib Counting the number of people in low-crowd-density (e.g. less than 10 people) and high-crowd-density (e.g. thousands of people) scenarios.

We separate the supported CV scenarios into two locations: (i) base: code and notebooks within the "utils_cs" and "scenarios" folders which follow strict coding guidelines, are well tested and maintained; (ii) contrib: code and other assets within the "contrib" folder, mainly covering less common CV scenarios using bleeding edge state-of-the-art approaches. Code in "contrib" is not regularly tested or maintained.

Computer Vision on Azure

Note that for certain computer vision problems, you may not need to build your own models. Instead, pre-built or easily customizable solutions exist which do not require any custom coding or machine learning expertise. We strongly recommend evaluating if these can sufficiently solve your problem. If these solutions are not applicable, or the accuracy of these solutions is not sufficient, then resorting to more complex and time-consuming custom approaches may be necessary.

The following Microsoft services offer simple solutions to address common computer vision tasks:

  • Vision Services are a set of pre-trained REST APIs which can be called for image tagging, face recognition, OCR, video analytics, and more. These APIs work out of the box and require minimal expertise in machine learning, but have limited customization capabilities. See the various demos available to get a feel for the functionality (e.g. Computer Vision).

  • Custom Vision is a SaaS service to train and deploy a model as a REST API given a user-provided training set. All steps including image upload, annotation, and model deployment can be performed using either the UI or a Python SDK. Training image classification or object detection models can be achieved with minimal machine learning expertise. The Custom Vision offers more flexibility than using the pre-trained cognitive services APIs, but requires the user to bring and annotate their own data.

If you need to train your own model, the following services and links provide additional information that is likely useful.

  • Azure Machine Learning service (AzureML) is a service that helps users accelerate the training and deploying of machine learning models. While not specific for computer vision workloads, the AzureML Python SDK can be used for scalable and reliable training and deployment of machine learning solutions to the cloud. We leverage Azure Machine Learning in several of the notebooks within this repository (e.g. deployment to Azure Kubernetes Service)

  • Azure AI Reference architectures provide a set of examples (backed by code) of how to build common AI-oriented workloads that leverage multiple cloud components. While not computer vision specific, these reference architectures cover several machine learning workloads such as model deployment or batch scoring.

Build Status

VM Testing

Build Type Branch Status Branch Status
Linux GPU master Build Status staging Build Status
Linux CPU master Build Status staging Build Status
Windows GPU master Build Status staging Build Status
Windows CPU master Build Status staging Build Status

AzureML Testing

Build Type Branch Status Branch Status
Linux GPU master Build Status staging Build Status
Linux CPU master Build Status staging Build Status
Notebook unit GPU master Build Status staging Build Status

Contributing

This project welcomes contributions and suggestions. Please see our contribution guidelines.

You can’t perform that action at this time.