Micro batching example #2210

mreso · 2023-03-31T23:48:59Z

Description

This PR introduces a new example demonstrating micro batching for a ResNet image classifier.

Type of change

Please delete options that are not relevant.

New feature (non-breaking change which adds functionality)

Feature/Issue validation/testing

Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

pytest test/pytest/test_micro_batching.py

============================================================================================================================================================================= test session starts ==============================================================================================================================================================================
platform linux -- Python 3.9.16, pytest-7.2.2, pluggy-1.0.0
rootdir: /home/ubuntu/serve
plugins: mock-3.10.0, cov-4.0.0
collected 14 items

test/pytest/test_micro_batching.py ..............                                                                                                                                                                                                                                                                                                                        [100%]

=============================================================================================================================================================================== warnings summary ===============================================================================================================================================================================
../../../opt/conda/envs/microbatching/lib/python3.9/site-packages/pkg_resources/__init__.py:121
  /opt/conda/envs/microbatching/lib/python3.9/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
    warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)

../../../opt/conda/envs/microbatching/lib/python3.9/site-packages/pkg_resources/__init__.py:2870
  /opt/conda/envs/microbatching/lib/python3.9/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../../../opt/conda/envs/microbatching/lib/python3.9/site-packages/pkg_resources/__init__.py:2870
../../../opt/conda/envs/microbatching/lib/python3.9/site-packages/pkg_resources/__init__.py:2870
  /opt/conda/envs/microbatching/lib/python3.9/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================================================================================================================================================= 14 passed, 4 warnings in 11.80s ========================================================================================================================================================================

pytest test/pytest/test_example_micro_batching.py

============================================================================================================================================================================= test session starts ==============================================================================================================================================================================
platform linux -- Python 3.9.16, pytest-7.2.2, pluggy-1.0.0
rootdir: /home/ubuntu/serve
plugins: mock-3.10.0, cov-4.0.0
collected 3 items

test/pytest/test_example_micro_batching.py ...                                                                                                                                                                                                                                                                                                                           [100%]

============================================================================================================================================================================== 3 passed in 49.75s ==============================================================================================================================================================================

Checklist:

Did you have fun?
Have you added tests that prove your fix is effective or that this feature works?
Has code been commented, particularly in hard-to-understand areas?
Have you made corresponding changes to the documentation?

codecov · 2023-04-01T00:12:37Z

Codecov Report

Merging #2210 (9022980) into master (4450287) will increase coverage by 1.24%.
The diff coverage is 95.26%.

❗ Current head 9022980 differs from pull request most recent head b3b487a. Consider uploading reports for the commit b3b487a to get more accurate results

@@            Coverage Diff             @@
##           master    #2210      +/-   ##
==========================================
+ Coverage   69.44%   70.69%   +1.24%     
==========================================
  Files          77       78       +1     
  Lines        3450     3638     +188     
  Branches       57       57              
==========================================
+ Hits         2396     2572     +176     
- Misses       1051     1063      +12     
  Partials        3        3

Impacted Files	Coverage Δ
ts/handler_utils/micro_batching.py	`90.90% <90.90%> (ø)`
ts/tests/unit_tests/test_micro_batching.py	`100.00% <100.00%> (ø)`

... and 3 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

HamidShojanazeri

Thanks @mreso , LGTM.

lxning

many users ignore TS good examples. Micro batching is a good feature to boost TPS. We need promote it to a feature announcement, not just an example.

can we move extract the core parts such as micro_batching.py and micro_batch_handler.py to handler utils so that users can easily plugged in customer handler?

lxning · 2023-04-04T18:07:22Z

examples/micro_batching/micro_batching_handler.py

+    def initialize(self, ctx):
+        super().initialize(ctx)
+
+        config_file = Path(ctx.system_properties["model_dir"], "micro_batching.json")


can we use the new feature model-config.yaml to set these parameters?

all of the parameters can be accessed via ctx. model_yaml_config

lxning · 2023-04-04T18:10:30Z

examples/micro_batching/micro_batching_handler.py

+                self.handle.parallelism = config["parallelism"]
+                self.handle.micro_batch_size = config["micro_batch_size"]


users may be thought "parallelism" and "micro_batch_size" are related to large model parallel. Can we have a good naming to differentiate?

Sure, can just call them mb_size and mb_parallelism

Went with ["micro_batching"]["parallelism"], that should be sufficient for users to differentiate.

…nfiguration

lxning

ts/handler_utils is handler utils dir. could you move utils/micro_batching.py to ts/handler_utils/microbatching?

lxning · 2023-05-24T19:19:09Z

ts/handler_utils/__init__.py

+from .micro_batching import MicroBatching
+
+__all__ = ["MicroBatching"]


this needs to be removed.

agunapal · 2023-05-25T17:47:29Z

@mreso I see the MacOS CI is failing because of Resource issue. Can you please check if its because of this PR?

mreso requested review from HamidShojanazeri, lxning, namannandan and agunapal April 3, 2023 20:33

HamidShojanazeri approved these changes Apr 4, 2023

View reviewed changes

lxning reviewed Apr 4, 2023

View reviewed changes

mreso added 21 commits April 4, 2023 23:12

Add unit test for micro batching

466526b

Adds simple implementation for microbatching with coroutines

c3c82a4

Added testing to microbatching

06b69ca

Created MBHandler

c39ae60

Added more tests to microbatching handler

9f5d17a

Adds configurable parallelism

07eecb5

Enables loading of micro batching parameters through config file

169099e

Moved microbatching into example

7373866

Moved micro batching test into test/pytest folder

26a7319

Rewrote micro batching to use threading

061abb6

Implemented method to update parallelism

3645c9a

Fix and test spin up spin down fo threads

9aaec3c

Clean up and comments

88022d2

More comments

6309ffb

Adds README to micro batching example

866b2ce

Refined readme + added config.yaml

565ccf1

Add config_file

6caf30d

Fix linting error

3d5d76d

Fix spell check error

c5fd370

Fix linting error

45aa9c2

Move micro_batching.py into ts.utils and use model_yaml_config for co…

9d06fe6

…nfiguration

mreso force-pushed the feature/microbatching branch from 00b3bb5 to 9d06fe6 Compare April 5, 2023 00:41

mreso requested a review from lxning April 5, 2023 00:43

mreso added 4 commits April 5, 2023 04:07

Fix links in README

91e523b

Merge branch 'master' into feature/microbatching

de05ebf

Merge branch 'master' into feature/microbatching

2c95274

Merge branch 'master' into feature/microbatching

5af65df

lxning reviewed May 24, 2023

View reviewed changes

Moved to handler_utils

15295bc

lxning reviewed May 24, 2023

View reviewed changes

remove __all__

b3b487a

mreso force-pushed the feature/microbatching branch from e6b97c1 to b3b487a Compare May 25, 2023 04:10

lxning approved these changes May 25, 2023

View reviewed changes

lxning merged commit 4f85477 into master May 25, 2023
18 of 19 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Micro batching example #2210

Micro batching example #2210

mreso commented Mar 31, 2023

codecov bot commented Apr 1, 2023 •

edited

HamidShojanazeri left a comment

lxning left a comment

lxning Apr 4, 2023

lxning Apr 4, 2023

mreso Apr 4, 2023

mreso Apr 5, 2023

lxning left a comment

lxning May 24, 2023

agunapal commented May 25, 2023

		self.handle.parallelism = config["parallelism"]
		self.handle.micro_batch_size = config["micro_batch_size"]

		from .micro_batching import MicroBatching

		__all__ = ["MicroBatching"]

Micro batching example #2210

Micro batching example #2210

Conversation

mreso commented Mar 31, 2023

Description

Type of change

Feature/Issue validation/testing

Checklist:

codecov bot commented Apr 1, 2023 • edited

Codecov Report

HamidShojanazeri left a comment

Choose a reason for hiding this comment

lxning left a comment

Choose a reason for hiding this comment

lxning Apr 4, 2023

Choose a reason for hiding this comment

lxning Apr 4, 2023

Choose a reason for hiding this comment

mreso Apr 4, 2023

Choose a reason for hiding this comment

mreso Apr 5, 2023

Choose a reason for hiding this comment

lxning left a comment

Choose a reason for hiding this comment

lxning May 24, 2023

Choose a reason for hiding this comment

agunapal commented May 25, 2023

codecov bot commented Apr 1, 2023 •

edited