Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a new x8-packq microkernel that packs and per-row dynamically quantizes fp32 to qp8. #6424

Merged
merged 1 commit into from
Jun 14, 2024

Conversation

copybara-service[bot]
Copy link
Contributor

Add a new x8-packq microkernel that packs and per-row dynamically quantizes fp32 to qp8.

The microkernels themselves are just wrappers for the corresponding KleidiAI kernels.

const size_t k_block_len = kr / sr;

for (size_t row_idx = 0; row_idx < num_rows; ++row_idx) {
float max0 = -FLT_MAX;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use params or initialize to the first value, like rmax

// Scale the values.
int32_t v0_s32 = (int32_t)(round(src0_0 * scale0));

v0_s32 = v0_s32 + nudged_zero_point0;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we make this adjustable for vnni to support +128?

const float src0_0 = *(src_ptr + k_idx + k_block_idx);

// Scale the values.
int32_t v0_s32 = (int32_t)(round(src0_0 * scale0));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

roundf?

v0_s32 = v0_s32 + nudged_zero_point0;
v0_s32 = fmaxf(v0_s32, INT8_MIN);
v0_s32 = fminf(v0_s32, INT8_MAX);
*((int8_t*)(dst_ptr)) = (int8_t)v0_s32;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can this be made to do 8 values from src0 and 8 values from src1 for i8mm?

size_t m_idx_start, // Starting index in `lhs_packed`.
const float* XNN_RESTRICT lhs, // Left-hand operator to pack.
size_t lhs_stride, // Stride in bytes between the rows of `lhs`.
void* XNN_RESTRICT lhs_packed // The quantized and packed output.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can lhs output be standard gemm qd8 int8 with stride and quantization params as (row sum) a different pointer?

@copybara-service copybara-service bot force-pushed the test_633914713 branch 7 times, most recently from ba3ec60 to 9d5e952 Compare May 27, 2024 12:03
@copybara-service copybara-service bot force-pushed the test_633914713 branch 8 times, most recently from 3d53157 to 46834e5 Compare June 14, 2024 12:07
…uantizes `fp32` to `qp8`.

The microkernels themselves are just wrappers for the corresponding KleidiAI kernels.

PiperOrigin-RevId: 643324022
@copybara-service copybara-service bot merged commit bffaa6f into master Jun 14, 2024
@copybara-service copybara-service bot deleted the test_633914713 branch June 14, 2024 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants