An autotraining tool for customvision.ai using Azure Blob Storage and Azure Pipelines
THIS PROJECT HAS MOVED TO MICROSOFT/CUSTOM-VISION-AUTOTRAINER
- Data Catalogue
- Train, Test, and Holdout sets
- CLI, Python and CI/CD friendly
- Get started quickly with local uploads
- Store labels with images
- Select data for training Custom Vision models
- Train and export your models
Backed by Azure Storage, Autotrainer helps you maintain a large collection of labelled images for machine learning.
Machine learning often requires the use of multiple datasets that must remain segregated. Autotrainer provides three containers for image datasets: train, test, and holdout.
- Train: Used to training the model.
- Test: Used to test the model during training, and potentially join the training set.
- Holdout: Validate your model using this unseen data.
Consume autotrainer via the CLI, in Python code, or run in Azure Pipelines.
Upload a set of images from a directory in a single command.
Labels are stored in special label files, right next to the image in blob storage.
Select images from your training set, and push them to a Custom Vision project.
Automate the training and exporting of models.
docker run -it -e "CV_ENDPOINT=https://southcentralus.api.cognitive.microsoft.com" -e "CV_TRAINING_KEY=your_key" -e "STORAGE_ACCOUNT_CONNECTION_STRING=your_connection_string" flanagan89/custom-vision-autotrainer -h
Autotrainer requires three environment variables:
CV_ENDPOINT
: The location of your custom vision service, e.g. https://southcentralus.api.cognitive.microsoft.comCV_TRAINING_KEY
: Your custom vision training keySTORAGE_ACCOUNT_CONNECTION_STRING
: Connection string to an Azure Storage Account
I recommend using Mini Conda to manage your python environment. Download and install miniconda, then in a shell:
- Create a conda environment:
conda create -n customvisionautotrainer python=3.6
- Activate the environment:
activate customvisionautotrainer
. - Install runtime dependencies:
pip install -r src/autotrainer/requirements.txt
- Install developer dependencies:
pip install -r src/autotrainer/requirements-dev.txt
- Configure environment variables (see above)
$ cd src/autotrainer
$ python ./autotrainer_cli.py -h
usage: autotrainer [cv, catalogue, select] <options>
Autotrainer tools
positional arguments:
command Subcommand to run
optional arguments:
-h, --help show this help message and exit
$ python ./autotrainer_cli.py catalogue -h
usage: autotrainer catalogue <options>
Data Catalogue tools
positional arguments:
{describe,upload} Catalogue options
optional arguments:
-h, --help show this help message and exit
Autotrainer uses nose
First, run Azurite for local blob storage testing:
$ cd src/
$ docker-compose up -d
Then you can run the tests.
$ cd src/autotrainer
$ nosetests
............
----------------------------------------------------------------------
Ran 12 tests in 7.088s
OK
NOTE: Some tests require access to a real Azure Storage account and Custom Vision service. See the environment variable section above.