[OpenCL][Texture] Improved texture memory planning #15058

srkreddy1238 · 2023-06-07T19:38:54Z

Motivated form the fact that textures can be allocated over a clBuffer object and the size of backing clBuffer can be computed based on hardware image pitch alignment.

This optimizes the overall memory allocation on device and helps greately the models with large memory requirements.

Improvised the graph memory planner to not differentiate buffer and texture storage tokens and reuse them across. The texture pool in OpenCL runtime is rebranded as memory pool that handles allocation for both buffer and image objects.

NDArray to DeviceAPI interface is extended with AllocDataSpaceView and FreeDataSpaceView. These new API's acommodates accessing same physical memory as clBuffer / clImage objects.

tvm-bot · 2023-06-07T19:38:57Z

Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from Reviewers by @-ing them in a comment.

cc @echuraev, @elvin-n _{See #10317 for details}

_{Generated by tvm-bot}

tvm-bot · 2023-06-07T19:38:57Z

Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from Reviewers by @-ing them in a comment.

cc @echuraev, @elvin-n _{See #10317 for details}

_{Generated by tvm-bot}

echuraev

Quickly took a look at this PR. It contains many changes, but no tests. I'd like to see unit tests for these changes.

srkreddy1238 · 2023-06-08T05:39:40Z

Quickly took a look at this PR. It contains many changes, but no tests. I'd like to see unit tests for these changes.

I agree. I am working on it.

Appreciate a review on the interface changes (DeviceAPI, NDArray) and over all design aspects mean while.

srkreddy1238 · 2023-06-12T02:23:24Z

@echuraev can you take a look now ?

echuraev

Sorry for the delay. It was a public holiday. Several comments.

src/runtime/graph_executor/graph_executor.cc

src/runtime/cuda/cuda_device_api.cc

src/runtime/metal/metal_device_api.mm

src/runtime/opencl/memory_pool.cc

tests/cpp-runtime/opencl/opencl_texture_pool_test.cc

tests/cpp-runtime/opencl/texture_copy_test.cc

tests/cpp/relay/backend/graph_plan_token_alloc.cc

src/runtime/ndarray.cc

tqchen · 2023-06-13T14:43:35Z

include/tvm/runtime/device_api.h

+   * \param dtype The type of elements.
+   * \param mem_scope The memory scope of allocated tensor.
+   * \return The allocated device pointer.
+   */


Looking at this set of changes. I feel that maybe we need something different that makes the memory allocator more explicit. Let me elaborate in reply

tqchen · 2023-06-13T14:50:28Z

Thanks @srkreddy1238 . Reading the set of changes, i feel that we need something that is more lifted out from the DeviceAPI since this is getting closer to the two stage concept.

Allocator allocates physical Storage object (in this case backed by clBuffer)
The storage object then allocates the NDArray(backed by the storage object).

We already have some of similar concept in memory manager, e.g. in relax, but of course the memory manager as it is does not yet reflect the need of supporting things like creating different memory scope. This would help to move some of the complexity in the texture part to a specialized allocator that creates the Storage with related behavior.

The main rationale is to keep the overall DeviceAPI minimum without behavior of things like pooling and move some of the logic to Storage, NDArray separation, that can hopefully also enable direct support in VM backed runtimes like unity. Maybe we can discuss a bit on how to make that part more generalized that can be reused.

srkreddy1238 · 2023-06-14T09:20:48Z

Thanks @echuraev and @tqchen for a quick review.

I see MemoryManager can meet the requirements of the pool concept we have here. I will improve the PR by

Initializing the MemoryManager (PooledAllocator) from graph runtime (StorageSetup).
NDArray->Empty => PooledAllocator->Empty (Assuming Pooled Allocator works for all runtimes or I cam limit this for texture scoped graphs).
NDArray->CreateView => DeviceAPI->AllocDataSpaceView : This will take care of various views over NDArray

Another situation I came across multiple is about runtime::GetDataSize(container->dl_tensor)

Can we move this to DeviceAPI ? In case of texture there is device specific attributes (image row pitch) that defines the underlaying memory size and accessing these attributes outside doesn't look to be a great idea.

srkreddy1238 · 2023-06-14T15:12:39Z

@tqchen later I realized that the MemoryManager interface is part of relax (unity branch, not mainline). Do I bring bring it into relay and tailor it ? Or Is there any other ideas ?

tqchen · 2023-06-14T15:17:51Z

One thing that I can probably do is do do a bit of refactoring and bring it to runtime folder so we can leverage it in most places, I may need a few weeks to get to this. There is also already one in relay that can be used as getting started. GetDataSize is also a logic that can be specific to the pool.

srkreddy1238 · 2023-07-17T06:42:10Z

@tqchen

I enhanced to reuse the existing VM runtime memory manager by graph executor at 31d7de0

Basically,

Promoted memory manager from runtime/vm to runtime level
Redirected graph executor to use Memory Manager interface for StorageSetup.
Corresponding changes in OpenCL Runtime to work with this MemoryManager.

Let me know your opinion on these.

Another recommendation I have is redirecting AllocWorkspace and FreeWorkspace to MemoryManager and drop workspace_pool. workspace_pool is slightly different from pooled_allocator (workspace pool looks for best fit where as pooled allocator look for exact fit) and we can amend this functionality.

…y allocation Motivated form the fact that textures can be allocated over a clBuffer object and the size of backing clBuffer can be computed based on hardware image pitch alignment. This optimizes the overall memory allocation on device and helps greately the models with large memory requirements. Improvised the graph memory planner to not differentiate buffer and texture storage tokens and reuse them across. The texture pool in OpenCL runtime is rebranded as memory pool that handles allocation for both buffer and image objects. NDArray to DeviceAPI interface is extended with AllocDataSpaceView and FreeDataSpaceView. These new API's acommodates accessing same physical memory as clBuffer / clImage objects. * MemoryPool test cases and lint errors. * test cases and fallback support. * bug fix and cpp-runtime tests cases for texture views. * various cl device info organized * fix graph plan memory bug and correct the testcase. * device attribute handling * Some fallback for texture plan on devices w/o cl_khr_image2d_from_buffer * Memory Manager Move the VM memory manager to the runtime level. Use this memory manager for graph runtime. * Resolve conflicts for VerifyDataType and Buffer * review comments

* Allocators need not aware of scope. They only do plaing memory allocations. * DeviceAPI can handle the scope.

srkreddy1238 · 2023-10-09T04:34:25Z

@tqchen @echuraev can you take another look on this ?
I have rebased and improvised based on the new memory manager interface.

echuraev

In general LGTM. Several comments.

apps/android_deploy/app/src/main/jni/tvm_runtime.h

src/runtime/memory/memory_manager.cc

src/runtime/opencl/opencl_device_api.cc

Co-authored-by: Egor Churaev <egor.churaev@gmail.com>

echuraev

LGTM. Thanks!

srkreddy1238 · 2023-10-20T03:36:43Z

@tqchen & @yongwww can you take a look on this PR ?

srkreddy1238 · 2023-11-22T05:28:26Z

@tqchen can you take a look on this PR ?

srkreddy1238 requested review from csullivan and echuraev June 7, 2023 19:38

srkreddy1238 changed the title ~~[OpenCL][Texture] Improved texture memmory planning and runtime memor…~~ [OpenCL][Texture] Improved texture memory planning Jun 7, 2023

srkreddy1238 requested a review from tqchen June 7, 2023 19:41

srkreddy1238 force-pushed the texture_enhancements branch from 35ca39f to fa689e5 Compare June 8, 2023 03:36

echuraev requested changes Jun 8, 2023

View reviewed changes

srkreddy1238 force-pushed the texture_enhancements branch from fa689e5 to 33e854b Compare June 8, 2023 05:35

srkreddy1238 force-pushed the texture_enhancements branch 2 times, most recently from 613d0f0 to 8351151 Compare June 9, 2023 03:12

srkreddy1238 requested a review from echuraev June 9, 2023 20:09

srkreddy1238 force-pushed the texture_enhancements branch 6 times, most recently from 59cca16 to ed8b70e Compare June 11, 2023 17:43

echuraev reviewed Jun 13, 2023

View reviewed changes

tqchen reviewed Jun 13, 2023

View reviewed changes

src/runtime/ndarray.cc Show resolved Hide resolved

tqchen requested changes Jun 13, 2023

View reviewed changes

srkreddy1238 force-pushed the texture_enhancements branch from ed8b70e to 31d7de0 Compare July 17, 2023 06:30

srkreddy1238 force-pushed the texture_enhancements branch from 31d7de0 to d3dee67 Compare July 17, 2023 09:58

srkreddy1238 force-pushed the texture_enhancements branch 4 times, most recently from 788ffe0 to ae78943 Compare August 23, 2023 05:58

srkreddy1238 mentioned this pull request Sep 27, 2023

[VM] Memory Manager moved up to runtime #15833

Merged

srkreddy1238 force-pushed the texture_enhancements branch from ae78943 to 537e87f Compare October 6, 2023 11:31

srkreddy1238 force-pushed the texture_enhancements branch from 537e87f to 2b48572 Compare October 6, 2023 13:13

srkreddy1238 added 4 commits October 7, 2023 15:41

* improve texture allocations.

22ca220

* Allocators need not aware of scope. They only do plaing memory allocations. * DeviceAPI can handle the scope.

* Now we support texture under pooled allocator.

70babe2

* improve OpenCL Texture cpp test cases.

d615a55

* Device API follows the executor Allocator type.

24610c8

srkreddy1238 force-pushed the texture_enhancements branch from ccc058b to 24610c8 Compare October 8, 2023 04:07

* More tests for allocators to cover various conflicts.

6ca6a53

srkreddy1238 force-pushed the texture_enhancements branch from 43db208 to 6ca6a53 Compare October 8, 2023 10:25

* Try to use existing allocator else create Naive.

4153c1b

srkreddy1238 requested a review from echuraev October 9, 2023 04:32

srkreddy1238 added 2 commits October 9, 2023 11:38

* Minor change for better usability.

4ab4656

* bug fix.

2da1716

srkreddy1238 requested a review from yongwww October 9, 2023 16:43

echuraev reviewed Oct 10, 2023

View reviewed changes

srkreddy1238 and others added 2 commits October 10, 2023 18:07

Update src/runtime/opencl/opencl_device_api.cc

40de788

Co-authored-by: Egor Churaev <egor.churaev@gmail.com>

* review comments

54d4dfd

srkreddy1238 requested a review from echuraev October 11, 2023 05:13

echuraev approved these changes Oct 11, 2023

View reviewed changes

Merge branch 'main' into texture_enhancements

4fb2646

Merge branch 'main' into texture_enhancements

cce9518

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[OpenCL][Texture] Improved texture memory planning #15058

[OpenCL][Texture] Improved texture memory planning #15058

srkreddy1238 commented Jun 7, 2023 •

edited

tvm-bot commented Jun 7, 2023

tvm-bot commented Jun 7, 2023

echuraev left a comment

srkreddy1238 commented Jun 8, 2023

srkreddy1238 commented Jun 12, 2023

echuraev left a comment •

edited

tqchen Jun 13, 2023

tqchen commented Jun 13, 2023

srkreddy1238 commented Jun 14, 2023

srkreddy1238 commented Jun 14, 2023

tqchen commented Jun 14, 2023

srkreddy1238 commented Jul 17, 2023

srkreddy1238 commented Oct 9, 2023

echuraev left a comment

echuraev left a comment

srkreddy1238 commented Oct 20, 2023

srkreddy1238 commented Nov 22, 2023

[OpenCL][Texture] Improved texture memory planning #15058

Are you sure you want to change the base?

[OpenCL][Texture] Improved texture memory planning #15058

Conversation

srkreddy1238 commented Jun 7, 2023 • edited

tvm-bot commented Jun 7, 2023

tvm-bot commented Jun 7, 2023

echuraev left a comment

Choose a reason for hiding this comment

srkreddy1238 commented Jun 8, 2023

srkreddy1238 commented Jun 12, 2023

echuraev left a comment • edited

Choose a reason for hiding this comment

tqchen Jun 13, 2023

Choose a reason for hiding this comment

tqchen commented Jun 13, 2023

srkreddy1238 commented Jun 14, 2023

srkreddy1238 commented Jun 14, 2023

tqchen commented Jun 14, 2023

srkreddy1238 commented Jul 17, 2023

srkreddy1238 commented Oct 9, 2023

echuraev left a comment

Choose a reason for hiding this comment

echuraev left a comment

Choose a reason for hiding this comment

srkreddy1238 commented Oct 20, 2023

srkreddy1238 commented Nov 22, 2023

srkreddy1238 commented Jun 7, 2023 •

edited

echuraev left a comment •

edited