Quick question about cache alignment and INT8 memory overhead on ARM #35130

[AtulBoyal]

Hey everyone,

I’ve been going through the CPU plugin source while finishing up my GSoC proposal for the ARM quantized inference project, and I had a quick architecture question.

While looking at how INT8 operations (like quantized::linear_relu) map to SIMD intrinsics, I noticed what seems like some extra memory rearrangement happening just before execution.

Does the ARM backend expect tensor data to already be aligned to cache line boundaries before it reaches the compiled graph, or is that handled dynamically by the JIT at runtime? I’m trying to understand where it’s safe to remove redundant tensor copies without affecting performance.

Thanks,
Atul

[Ganesh-403]

@AtulBoyal From what I can tell, there isn't a documented guarantee that tensor data must already be cache-line aligned before reaching the compiled graph.

The CPU plugin already performs its own memory management and kernel selection internally, and for INT8 inference the execution path is driven by the low-precision transformation pipeline together with backend libraries such as oneDNN (and Arm Compute Library on Arm). Those libraries generally handle packing/reordering of tensors as needed for the selected kernel rather than requiring callers to provide pre-packed buffers. :contentReference[oaicite:0]{index=0}

So regarding your question:

"Where is it safe to remove redundant tensor copies?"

I'd be cautious about removing copies solely based on cache alignment assumptions. A copy that looks redundant may actually be producing:

• the memory layout expected by the backend kernel,
• packed weights,
• or a contiguous/aligned buffer required by oneDNN/ACL for efficient execution.

Without profiling, it can be difficult to distinguish a genuine redundant copy from one that's enabling a faster kernel.

A couple of questions that would help narrow this down:

1. Are you looking at the OpenVINO CPU plugin itself, or the Arm Compute Library backend used by the CPU plugin?
2. Is the copy happening:
   • during model compilation,
   • tensor creation,
   • or immediately before a specific INT8 primitive (e.g. quantized::linear_relu) executes?

If it's immediately before kernel execution, I'd also be interested to know whether the source tensor is already contiguous and in the expected layout. If not, the rearrangement may be intentional rather than just an alignment optimization.

Regarding your draft PR (#35003), I don't have review permissions, but it's great that you're contributing. Hopefully one of the maintainers can take a look when they have time.

If this solves your problem, feel free to mark it as the accepted answer so others can find it easily.