Skip to content
This repository was archived by the owner on Jun 3, 2025. It is now read-only.

Conversation

@rahul-tuli
Copy link
Member

The goal of this PR is to add a pipeline interface for YOLO image segmentation models. The idea is to wrap preprocessing and postprocessing steps into a callable pipeline for ease of use, and out of the box support with deepsparse.server. This also adds an installable for yolo, to install all dependencies invoke:
pip install --editable "./[yolo]"

Usage:

  • Option 1, without a class names mapping
from deepsparse.pipeline import Pipeline

model_stub = "zoo:cv/detection/yolov5-s/pytorch/ultralytics/coco/pruned-aggressive_96"
task = "yolo"
input_file = "yolo-test-image.JPEG"



pipeline = Pipeline.create(
    task=task,
    model_path=stub,
    class_names="coco",
)

print("Pipeline outputs: ", pipeline(images=input_file))

output:

DeepSparse Engine, Copyright 2021-present / Neuralmagic, Inc. version: 0.11.2 (ca6e29a5) (release) (optimized) (system=avx512, binary=avx512)
Pipeline outputs:  predictions=[[[132.24066162109375, 0.04663848876953125, 269.58807373046875, 46.56146240234375, 0.4553620517253876, 2.0], [181.0561065673828, 7.16241455078125, 447.66375732421875, 612.1524047851562, 0.444583535194397, 0.0]]] boxes=[[[132.24066162109375, 0.04663848876953125, 269.58807373046875, 46.56146240234375], [181.0561065673828, 7.16241455078125, 447.66375732421875, 612.1524047851562]]] scores=[[0.4553620517253876, 0.444583535194397]] labels=[['car', 'person']]

@rahul-tuli rahul-tuli self-assigned this Apr 21, 2022
@rahul-tuli rahul-tuli requested review from a team, bfineran, dbogunowicz and markurtz and removed request for a team April 21, 2022 18:27
@rahul-tuli rahul-tuli added the 0.13 release label for PRs targeting release/0.13 label Apr 21, 2022
@rahul-tuli rahul-tuli requested a review from KSGulin April 21, 2022 18:27
Copy link
Contributor

@dbogunowicz dbogunowicz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, good job. I would refactor some of the code, but I assume that those pieces which I do not like are copied from the original yolo codebase, so it does not make sense to beautify it - better to have it in the original form for reference, would you agree?

@rahul-tuli rahul-tuli requested a review from dbogunowicz April 26, 2022 14:56
Base automatically changed from ic-pipelines to pipeline-feature-main April 29, 2022 19:06
rahul-tuli and others added 5 commits April 29, 2022 17:49
Add an installable for yolo integration
Added a task for YOLO

To install run:
* `pip install --editable "./[yolo]"`
* Moved COCO classes to a file
* Adds support to annotate images using YOLO

* Makes `YOLOOutput` iterable
* Returns a named tuple of image outputs when `next` is called on `YOLOOutput`
* Adds an annotate function to yolo utils

* Adds an annotation script, testing + minor fixes remain

* Intermediate-commit

* Intermediate WIP

* Working State with required bugfixes

* style fixes

Co-authored-by: Benjamin <ben@neuralmagic.com>
@bfineran
Copy link
Contributor

rebased onto feature-pipeline-main

@bfineran bfineran merged commit 47e0074 into pipeline-feature-main Apr 29, 2022
@bfineran bfineran deleted the yolo-pipeline branch April 29, 2022 21:51
bfineran added a commit that referenced this pull request Apr 29, 2022
…P,IC,OD pipelines (#317)

* base commit - make pydantic a general req

* Pipeline base class implementation (#315)

* Pipeline base class implementation

* constructor default values

* __call__ inputs/outputs parsing + validation

* documentation

* pipeline 'alias' argument

* review fixes

* [feature/Pipeline] PipelineConfig (#318)

* PipelineConfig pydantic model + Pipeline.from_config

* Pipeline.to_config() function

* [feature/Pipeline] refactor deepsparse.server to use deepsparse.Pipeline (#319)

* PipelineConfig pydantic model + Pipeline.from_config

* Pipeline.to_config() function

* refactor deepsparse.server to use deepsparse.Pipeline

* review nit fix

remove files for separate feature

* Image Classification Pipeline Integration (#322)

* Create a command line installable for image classification pipeline

* Intermediate Commit

* Image Classification pipeline implementation

* Remove faulty entry point

* Apply suggestions from @bogunowicz


* Changed function  name from `_infer_input_shape` to `_infer_image_shape`

* Add validation script for Image Classification pipeline (#328)

* Add Validation Script for Image Classification Models

* Update pipelines and corresponding schemas to work with numpy arrays

* Bugfix if prediction to be converted to int if it's a string

* Update docstring

* Update src/deepsparse/image_classification/validation_script.py

* [feature/Pipeline] fixes for ic-pipelines implementation (#336)

* fixes for ic-pipelines implementation

* sparsezoo support

Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com>

* Update src/deepsparse/pipeline.py


* quality

* [feature/Pipeline] deepsparse.Pipeline implementations for transformers (#320)

* add parsing layer for deepsparse.Pipeline for implementation flexibility

* initial deepsparse.transformers.pipelines implementation base class + text_classification

* make tokenizer and config attributes 'public', but not properties

* decorator fix

* use kwargs for transforemrs pipeline parent class args

* token classification impl

* token classification output schema parsing

* quality

* question answering pipeline impl

* fixes for pipline impls - bs1 santity check inferences working

* [feature/Pipeline] deprecate and migrate existing transformers pipelines (#335)

* remove old pipeline pathway and files, add API for deprecated pathway

* migrate eval_downstream

* update readme

* server pipeline input fix

* hf license attribution

* `YOLO` pipeline integration for deepsparse (#327)

* Added YOLO pipeline
Add an installable for yolo integration
Added a task for YOLO

To install run:
* `pip install --editable "./[yolo]"`

* Changed function  name from `_infer_input_shape` to `_infer_image_shape`

* Update docstring

* Comments from @bogunowicz
* Moved COCO classes to a file

* Adds support to annotate images using YOLO (#332)

* Adds support to annotate images using YOLO

* Makes `YOLOOutput` iterable
* Returns a named tuple of image outputs when `next` is called on `YOLOOutput`
* Adds an annotate function to yolo utils

* Adds an annotation script, testing + minor fixes remain

* Intermediate-commit

* Intermediate WIP

* Working State with required bugfixes

* style fixes

Co-authored-by: Benjamin <ben@neuralmagic.com>

Co-authored-by: Benjamin <ben@neuralmagic.com>

* [feature/Pipeline] rename input/output _model to _schema (#340)

* rename input/output _model to _schema

* refactor yolo pipeline

* default model support for Pipeline.register (#339)

* default model support for Pipeline.register

* update default stubs for transformers and IC

* yolo default model

* minor fixes

* model->schema for server

Co-authored-by: Rahul Tuli <rahul@neuralmagic.com>
markurtz added a commit that referenced this pull request May 2, 2022
* SparseServer.UI example [WIP] (#308)

* add megasparse dir

* edited readme to support DS integration

* small edit to docstring

* edited settings.py module so only 2 models are loaded.

* added more context to the readme about adding more models.

* fixed image

* added different  host  to streamlit client as default

* quality check edits

* quality check commit

* passing copyright quality check

* content edits

* rename dir to sparseserver-ui

* added new config file for quick start

* edited multipipelineclient in settings.py

* changed name of config file

* edited model stubs

* edited readme

* added dependency pins

* changed server pin

* edited model choice logic

* altered front-end features

* style update

* renamed samples file

* added variant descriptions

* style changes

* edited samples.py module

* style changes

* added new pic

* edited README

* updated readme cmds

* SparseServer edit (#314)

* edit number of models

* edit settings.py

* edit readme

* Update label mapping for deepsparse.transformers.eval_downstream (#323)

* Update label mapping for deepsparse.transformers.eval_downstream

* Fix MNLI as well

* bump up main to 0.13.0 (#313)

Co-authored-by: dhuang <dhuang@dhuangs-MacBook-Pro.local>

* AWS Sagemaker example integration (#305)

* AWS Sagemaker example integration

* documentation, sample config, dockerfile fixes

* fix ecr repo name

* readme code changes from testing

* Update huggingface-transformers/README.md with new models (#329)

* Update README.md (#330)

* Update README.md

various grammatical edits
additional edits for section headline consistency

* Topology file for HB120rs_v3 (#334)

* Topology file for HB120rs_v3

Specifies core-per-CCX grouping for HB120rs_v3 VM's, used by multi-process script.

* Update README.md to reference Azure topo file

* Move all benchmarking within deepsparse/benchmark/ (#333)

* Move all benchmarking within deepsparse/benchmark/

* Update benchmark_model

* Expose results at benchmark base

* isort

* Skip flake8

* server integration check bug fix (#331)

* server integration check bug fix

need to verify integration is set before calling `integration.lower()`

* respond to review - click choice

* add default integration val to server config schema (#337)

* deepsparse.Pipeline - generic pipeline, deepsparse.server support, NLP,IC,OD pipelines (#317)

* base commit - make pydantic a general req

* Pipeline base class implementation (#315)

* Pipeline base class implementation

* constructor default values

* __call__ inputs/outputs parsing + validation

* documentation

* pipeline 'alias' argument

* review fixes

* [feature/Pipeline] PipelineConfig (#318)

* PipelineConfig pydantic model + Pipeline.from_config

* Pipeline.to_config() function

* [feature/Pipeline] refactor deepsparse.server to use deepsparse.Pipeline (#319)

* PipelineConfig pydantic model + Pipeline.from_config

* Pipeline.to_config() function

* refactor deepsparse.server to use deepsparse.Pipeline

* review nit fix

remove files for separate feature

* Image Classification Pipeline Integration (#322)

* Create a command line installable for image classification pipeline

* Intermediate Commit

* Image Classification pipeline implementation

* Remove faulty entry point

* Apply suggestions from @bogunowicz


* Changed function  name from `_infer_input_shape` to `_infer_image_shape`

* Add validation script for Image Classification pipeline (#328)

* Add Validation Script for Image Classification Models

* Update pipelines and corresponding schemas to work with numpy arrays

* Bugfix if prediction to be converted to int if it's a string

* Update docstring

* Update src/deepsparse/image_classification/validation_script.py

* [feature/Pipeline] fixes for ic-pipelines implementation (#336)

* fixes for ic-pipelines implementation

* sparsezoo support

Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com>

* Update src/deepsparse/pipeline.py


* quality

* [feature/Pipeline] deepsparse.Pipeline implementations for transformers (#320)

* add parsing layer for deepsparse.Pipeline for implementation flexibility

* initial deepsparse.transformers.pipelines implementation base class + text_classification

* make tokenizer and config attributes 'public', but not properties

* decorator fix

* use kwargs for transforemrs pipeline parent class args

* token classification impl

* token classification output schema parsing

* quality

* question answering pipeline impl

* fixes for pipline impls - bs1 santity check inferences working

* [feature/Pipeline] deprecate and migrate existing transformers pipelines (#335)

* remove old pipeline pathway and files, add API for deprecated pathway

* migrate eval_downstream

* update readme

* server pipeline input fix

* hf license attribution

* `YOLO` pipeline integration for deepsparse (#327)

* Added YOLO pipeline
Add an installable for yolo integration
Added a task for YOLO

To install run:
* `pip install --editable "./[yolo]"`

* Changed function  name from `_infer_input_shape` to `_infer_image_shape`

* Update docstring

* Comments from @bogunowicz
* Moved COCO classes to a file

* Adds support to annotate images using YOLO (#332)

* Adds support to annotate images using YOLO

* Makes `YOLOOutput` iterable
* Returns a named tuple of image outputs when `next` is called on `YOLOOutput`
* Adds an annotate function to yolo utils

* Adds an annotation script, testing + minor fixes remain

* Intermediate-commit

* Intermediate WIP

* Working State with required bugfixes

* style fixes

Co-authored-by: Benjamin <ben@neuralmagic.com>

Co-authored-by: Benjamin <ben@neuralmagic.com>

* [feature/Pipeline] rename input/output _model to _schema (#340)

* rename input/output _model to _schema

* refactor yolo pipeline

* default model support for Pipeline.register (#339)

* default model support for Pipeline.register

* update default stubs for transformers and IC

* yolo default model

* minor fixes

* model->schema for server

Co-authored-by: Rahul Tuli <rahul@neuralmagic.com>

* Remove: startlette dep (#338)

* Update src/deepsparse/version.py

Co-authored-by: Ricky Costa <79061523+InquestGeronimo@users.noreply.github.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>
Co-authored-by: dhuangnm <74931910+dhuangnm@users.noreply.github.com>
Co-authored-by: dhuang <dhuang@dhuangs-MacBook-Pro.local>
Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com>
Co-authored-by: Jeannie Finks <74554921+jeanniefinks@users.noreply.github.com>
Co-authored-by: Govind Ramnarayan <77341216+govindr-nm@users.noreply.github.com>
Co-authored-by: Rahul Tuli <rahul@neuralmagic.com>
Co-authored-by: Konstantin Gulin <66528950+KSGulin@users.noreply.github.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

0.13 release label for PRs targeting release/0.13

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants