Integrating HAWX Framework with hls4ml for Approximate ViT Hardware Evaluation

[saeedalmasi]

Hello hls4ml team,

My name is Mohammad Saeed Almasi. I am a researcher working on a project titled “HAWX: A Hardware-Aware Framework for Fast and Scalable Approximation of DNNs” (https://arxiv.org/abs/2602.16336). The HAWX framework enables efficient design-space exploration of approximate operators (e.g., multipliers) under hardware constraints.

I am currently extending this work to Vision Transformer (ViT) architectures and have completed the algorithm-level evaluation. My next step is to obtain reliable FPGA-based hardware results, and I believe hls4ml is the ideal tool for this, given its robustness and standardization.

I have a few specific questions regarding this integration, particularly concerning the Vitis backend:

ViT Support Status: I understand from the documentation that Multi-head Attention (MHA) support is currently 'experimental'. Could you elaborate on the current capabilities and limitations of this experimental support? I am particularly interested in whether it has been tested with more complex Vision Transformer variants, for example, the Swin Transformer, which uses a shifted window attention mechanism.

Handling Approximate Operators: My work involves injecting approximate multipliers into the DNN dataflow. I noticed related work on integrating approximate multipliers with hls4ml. Are you aware of any ongoing efforts or best practices for seamlessly integrating custom approximate operators into the hls4ml workflow?

Efficient Path to Hardware Results: Since my algorithm-level evaluation is complete, I’m focused on the hardware synthesis stage. I’m considering using hls4ml’s Extension API to implement support for custom layers/operators. Could you offer any guidance or point me to examples that would help streamline this process for a ViT-like architecture?

From my perspective, this integration could be mutually beneficial:

For my research, it would provide a solid hardware evaluation framework for approximate ViTs.

For the hls4ml community, it could extend the tool’s use cases into approximate computing and design-space exploration for transformer architectures, which I believe aligns with the project's future directions.

I am very willing to align my implementation with hls4ml's internals and contribute back any useful extensions.

Thank you for your time and for maintaining this excellent framework.

Best regards,
Mohammad Saeed Almasi