Repository navigation
Q-ANT inference backend #5826
slobodaapl
started this conversation in
Ideas
Replies: 1 comment
|
Hey, thanks for checking in before initiating any work! We wouldn't be looking to upstream such a backend at this stage, our backend work is focused on CubeCL backends, with Flex as a light & portable alternative. A caveat if you're planning on exploring the integration: since Burn 0.22 removed backend generics from the user API, third-party backends cannot plug directly into Burn's API at this time. That would require separate design work to support with the dispatch layer. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I’m looking into adding support for the Q-ANT Native Computing Toolkit as an inference-only Burn backend.
Repository:
https://github.com/Q-ANT-GmbH/qant_native_computing_toolkit
The toolkit exposes its own native compute API and already has Rust bindings, so my current thinking is that this fits best as a direct Burn backend. The hardware is not yet avail. for testing but they released it early so integrators can start working on it early.
The intended architecture would roughly be:
The backend would intentionally be inference-only and would not implement autodiff, since it doesn't seem like the hardware will initially be built around this.
Initially I would focus on mapping the tensor operations required for practical inference onto the toolkit’s existing API, with unsupported operations left unimplemented fro now.
Before I start implementing this, I wanted to check whether this is the direction the Burn maintainers would consider appropriate. I feel like this might be a meaningful opportunity for Burn to differentiate itself uniquely and possible even get the hardware manufacturer's attention and co-advertising and co-exposure, perhaps even partnership down the line (just speculating here, but it's not insane to think this can be done)
I also considered implementing Q-ANT as a CubeCL target instead. That would obviously be a much deeper integration and potentially allow existing CubeCL/CubeK kernels to target Q-ANT, but it seems unnecessary unless the Q-ANT execution model maps well enough onto CubeCL to justify treating it as a general CubeCL runtime rather than simply exposing its existing compute primitives through Burn. At least until the hardware is more mature.
I'm just curious if I should start this undertaking since it will likely be a considerable chunk of development, and if the authors are interested in eventually upstreaming this once I am done implementing it, if the quality is fine and it works well? I'd work on getting hardware to test this on myself once it's avail., for now they expose CPU-simulated SDK for tests which should be enough to validate the impl.
All reactions