Skip to content

Improving quantized matmul performance by devectorizing shader. #15274

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Improving quantized matmul performance by devectorizing shader. #15274

wants to merge 2 commits into from
+39 −26

Conversation

trivedivivek
Copy link
Contributor

Summary:
This diff improves the performance of quantized matrix multiplication by devectorizing the shader.

An example modification is shown below:

// Before
VEC4_T sums[TILE_ROWS][TILE_TXCOLS];

// After
T sums[TILE_ROWS * TILE_TXCOLS * 4];

// Before
sums[r][${c}] = VEC4_T(0.0);

// After
for (int j = 0; j < 4; j++) {
    sums[r * TILE_TXCOLS * 4 + ${c} * 4 + j] = T(0.0);
}

Differential Revision: D85023829

@trivedivivek trivedivivek requested a review from SS-JIA as a code owner October 20, 2025 15:46
@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Oct 20, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15274

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

❌ 1 New Failure, 124 Pending

As of commit 81e68f6 with merge base 5d71c9b (image):

NEW FAILURE - The following job has failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 20, 2025
@meta-codesync Meta CodeSync
Copy link

meta-codesync bot commented Oct 20, 2025

@trivedivivek has exported this pull request. If you are a Meta employee, you can view the originating Diff in D85023829.

@trivedivivek trivedivivek added the release notes: vulkan Changes to the Vulkan backend delegate label Oct 20, 2025
trivedivivek added a commit to trivedivivek/executorch that referenced this pull request Oct 20, 2025
@trivedivivek @facebook-github-bot
Improving quantized matmul performance by devectorizing shader. (pyto…
4913210 
…rch#15274)

Summary:

This diff improves the performance of quantized matrix multiplication by devectorizing the shader. 

An example modification is shown below:

```glsl
// Before
VEC4_T sums[TILE_ROWS][TILE_TXCOLS];

// After
T sums[TILE_ROWS * TILE_TXCOLS * 4];

// Before
sums[r][${c}] = VEC4_T(0.0);

// After
for (int j = 0; j < 4; j++) {
    sums[r * TILE_TXCOLS * 4 + ${c} * 4 + j] = T(0.0);
}
```

Differential Revision: D85023829
@trivedivivek trivedivivek force-pushed the export-D85023829 branch 2 times, most recently from 4913210 to b831b18 Compare October 20, 2025 22:09
trivedivivek added a commit to trivedivivek/executorch that referenced this pull request Oct 20, 2025
@trivedivivek @facebook-github-bot
Improving quantized matmul performance by devectorizing shader. (pyto…
b831b18 
…rch#15274)

Summary:

This diff improves the performance of quantized matrix multiplication by devectorizing the shader. 

An example modification is shown below:

```glsl
// Before
VEC4_T sums[TILE_ROWS][TILE_TXCOLS];

// After
T sums[TILE_ROWS * TILE_TXCOLS * 4];

// Before
sums[r][${c}] = VEC4_T(0.0);

// After
for (int j = 0; j < 4; j++) {
    sums[r * TILE_TXCOLS * 4 + ${c} * 4 + j] = T(0.0);
}
```

Differential Revision: D85023829
trivedivivek added a commit to trivedivivek/executorch that referenced this pull request Oct 21, 2025
@trivedivivek @facebook-github-bot
Improving quantized matmul performance by devectorizing shader. (pyto…
88f7aa0 
…rch#15274)

Summary:

This diff improves the performance of quantized matrix multiplication by devectorizing the shader. 

An example modification is shown below:

```glsl
// Before
VEC4_T sums[TILE_ROWS][TILE_TXCOLS];

// After
T sums[TILE_ROWS * TILE_TXCOLS * 4];

// Before
sums[r][${c}] = VEC4_T(0.0);

// After
for (int j = 0; j < 4; j++) {
    sums[r * TILE_TXCOLS * 4 + ${c} * 4 + j] = T(0.0);
}
```

Differential Revision: D85023829
trivedivivek added a commit to trivedivivek/executorch that referenced this pull request Oct 21, 2025
@trivedivivek @facebook-github-bot
Improving quantized matmul performance by devectorizing shader. (pyto…
1a04777 
…rch#15274)

Summary:

This diff improves the performance of quantized matrix multiplication by devectorizing the shader. 

An example modification is shown below:

```glsl
// Before
VEC4_T sums[TILE_ROWS][TILE_TXCOLS];

// After
T sums[TILE_ROWS * TILE_TXCOLS * 4];

// Before
sums[r][${c}] = VEC4_T(0.0);

// After
for (int j = 0; j < 4; j++) {
    sums[r * TILE_TXCOLS * 4 + ${c} * 4 + j] = T(0.0);
}
```

Differential Revision: D85023829
trivedivivek added a commit to trivedivivek/executorch that referenced this pull request Oct 21, 2025
@trivedivivek @facebook-github-bot
Improving quantized matmul performance by devectorizing shader. (pyto…
29a03da 
…rch#15274)

Summary:

This diff improves the performance of quantized matrix multiplication by devectorizing the shader. 

An example modification is shown below:

```glsl
// Before
VEC4_T sums[TILE_ROWS][TILE_TXCOLS];

// After
T sums[TILE_ROWS * TILE_TXCOLS * 4];

// Before
sums[r][${c}] = VEC4_T(0.0);

// After
for (int j = 0; j < 4; j++) {
    sums[r * TILE_TXCOLS * 4 + ${c} * 4 + j] = T(0.0);
}
```

Reviewed By: SS-JIA

Differential Revision: D85023829
@trivedivivek @facebook-github-bot
Minor perf improvements to quantized mat mul shader. (pytorch#15261)
c2efb80 
Summary:

The diff includes minor performance improvements to the quantized matrix multiplication shader.

Reviewed By: SS-JIA

Differential Revision: D84998542
@trivedivivek @facebook-github-bot
Improving quantized matmul performance by devectorizing shader. (pyto…
81e68f6 
…rch#15274)

Summary:

This diff improves the performance of quantized matrix multiplication by devectorizing the shader. 

An example modification is shown below:

```glsl
// Before
VEC4_T sums[TILE_ROWS][TILE_TXCOLS];

// After
T sums[TILE_ROWS * TILE_TXCOLS * 4];

// Before
sums[r][${c}] = VEC4_T(0.0);

// After
for (int j = 0; j < 4; j++) {
    sums[r * TILE_TXCOLS * 4 + ${c} * 4 + j] = T(0.0);
}
```

Reviewed By: SS-JIA

Differential Revision: D85023829
trivedivivek added a commit to trivedivivek/executorch that referenced this pull request Oct 21, 2025
@trivedivivek @facebook-github-bot
Improving quantized matmul performance by devectorizing shader. (pyto…
d4b474d 
…rch#15274)

Summary:

This diff improves the performance of quantized matrix multiplication by devectorizing the shader. 

An example modification is shown below:

```glsl
// Before
VEC4_T sums[TILE_ROWS][TILE_TXCOLS];

// After
T sums[TILE_ROWS * TILE_TXCOLS * 4];

// Before
sums[r][${c}] = VEC4_T(0.0);

// After
for (int j = 0; j < 4; j++) {
    sums[r * TILE_TXCOLS * 4 + ${c} * 4 + j] = T(0.0);
}
```

Reviewed By: SS-JIA

Differential Revision: D85023829
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@SS-JIA SS-JIA SS-JIA approved these changes

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. fb-exported meta-exported release notes: vulkan Changes to the Vulkan backend delegate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@trivedivivek @SS-JIA