How to set the path to allow for Azure Machine Learning integration?

[V4A001]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

I am running Yolov5 in Azure Machine Learning instead of Google Colab. Microsoft has a full featured integrated capability of creating docker containers, and run it on GPU's in the cloud straight from Visual Studio Code. 0.23 USD/ hr for a T80 Tesla on low priority infrastructure. It even has full github ML Ops integrated. See image below where I show it is working...but with some hacks in your code. It is much easier to use than Google CoLab, but getting it to work with this repository is a bit hard. I particular as I do not know how to avoid mixing up our code with this repository, but that is maybe because Python is not that OO as C# to have it perfectly separated.

I modified this in the requirements.txt:
opencv-python-headless>=4.1.1
#opencv-python>=4.1.1

My script is a py file in the Yolo folder. Basically this is all needed to fully create a container and sent it to the Azure ML cloud:
`from pathlib import Path

from azureml.core import (
Environment,
Experiment,
Run,
ScriptRunConfig,
Workspace,
Dataset,
Datastore,
)
from azureml.data import OutputFileDatasetConfig

from azureml.core.compute import AmlCompute, ComputeTarget
from azureml.core.authentication import AzureCliAuthentication

def find_or_create_compute_target(
workspace,
name,
vm_size="Standard_NC6", #"Standard_D8_v3",
min_nodes=0,
max_nodes=1,
idle_seconds_before_scaledown=900,
vm_priority="lowpriority",
):

if name in workspace.compute_targets:
    return ComputeTarget(workspace=workspace, name=name)
else:
    config = AmlCompute.provisioning_configuration(
        vm_size=vm_size,
        min_nodes=min_nodes,
        max_nodes=max_nodes,
        vm_priority=vm_priority,
        idle_seconds_before_scaledown=idle_seconds_before_scaledown,
    )
    target = ComputeTarget.create(workspace, name, config)
    target.wait_for_completion(show_output=True)

return target

workspace_config = 'config.json'
requirements_file = 'requirements.txt'
compute_target_name = 'LowPrioStdNC6'
experiment_name = 'train-yolo-7'
script_path = 'segment/train.py'
blobstorage = 'workspaceblobstore'
blobcontainerpath = 'UI/2023-01-05_###_UTC'

cli_auth = AzureCliAuthentication()

Authenticate with your AzureML Resource via its config.json file

ws = Workspace.from_config(path=workspace_config, auth=cli_auth)

ws = Workspace.from_config(path="v4a/config.json", auth=cli_auth)

The experiment in this workspace under which our runs will be grouped

If an experiment with the given name doesn't exist, it will be created

exp = Experiment(ws, experiment_name)

The compute cluster you want to run on and its settings.

If it doesn't exist, it'll be created.

compute_target = find_or_create_compute_target(ws, compute_target_name)

The Environment lets us define any dependencies needed to make our script run

env = Environment.from_pip_requirements("my-pip-env", requirements_file)

datastore = Datastore.get(ws, blobstorage)
dataset = Dataset.File.from_files(path=(datastore, blobcontainerpath))

output = OutputFileDatasetConfig(name='outputs', destination=(datastore, 'outputdataset'))

script_args = ['--device', '0',
'--img', '640',
'--cfg', 'v4a/data/yolov5s-seg.yaml',
'--hyp', 'data/hyps/hyp.scratch-high.yaml',
'--batch', '16',
'--epochs', '100',
'--data', 'v4a/data/###.yaml',
'--weights', 'v4a/data/yolov5s-seg.pt',
'--workers', '24',
'--name', '#####',
'--project', output.as_mount(),
'--inputs', dataset.as_named_input('inputs').as_mount()]
#'--outputs', output.as_mount()]

A run configuration is how you define what youd like to run

We give it the directory where our code is, the script we want to run, the environment, and the compute info

run_config = ScriptRunConfig(
source_directory=".",#Path(script_path).parent,
script="./segment/train.py", #Path(script_path).name,
arguments=script_args,
compute_target=compute_target,
environment=env,
)

Submit our configured run under our experiment

run = exp.submit(run_config)

Wait for the run to finish

run.wait_for_completion(show_output=True)

Delete our compute target so it doesn't cost us money

#compute_target.delete()`

The input and outputs are explained in this Jupyter notebook: https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/work-with-data/datasets-tutorial/scriptrun-with-data-input-output/how-to-use-scriptrun.ipynb

As our dataset is too large we need to mount the dataset from an azure blob storage. I can however not see how to change the path. I see in the yaml file a path can be set, but that is not working.

Is it possible to support this natively ? I am a C# programmer and do not get the idea of OO in Python. So I did some modifications which are not so nice:

Another not so nice:

It is currently processing, question is if I can see the outputs of the training somewhere:

The output is also saved to a azure blob:

Additional

Would be nice to support Microsoft Azure Machine Learning natively including azure blob storage for large files. It will save the environment by not having to pump data around.

[github-actions]

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

• Wiki – https://github.com/ultralytics/yolov5/wiki
• Tutorials – https://docs.ultralytics.com/yolov5
• Docs – https://docs.ultralytics.com

Access additional Ultralytics ⚡ resources:

• Ultralytics HUB – https://ultralytics.com/hub
• Vision API – https://ultralytics.com/yolov5
• About Us – https://ultralytics.com/about
• Join Our Team – https://ultralytics.com/work
• Contact Us – https://ultralytics.com/contact

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!

[glenn-jocher]

@V4A001 thanks for reaching out! It's great to see YOLOv5 being utilized in Azure Machine Learning. Your thorough explanation and the screenshots provide valuable insight into your integration process. As YOLOv5 is designed to be modular and customizable, we appreciate your patience in working through the integration.

The YOLOv5 repository, developed by the Ultralytics team, aims to support a wide range of environments and use cases. While we don't have native support for Microsoft Azure Machine Learning and its specific functionalities, we encourage community contributions to extend YOLOv5's compatibility with various platforms and workflows.

I recommend thoroughly reviewing the Azure Machine Learning documentation to explore any specific best practices for integrating with external libraries like YOLOv5. Additionally, the Ultralytics Docs at https://docs.ultralytics.com/yolov5/ provide extensive information on customizing YOLOv5 training and deployment processes that might be helpful for your integration.

Thank you for your feedback and contribution to the YOLOv5 community! If you have any further questions or insights, feel free to share them.