Allow cancelling ongoing async generation in LlmInference

[bringert]

MediaPipe Solution (you are using)

LlmInference for Android

Programming language

Android Java

Are you willing to contribute it

Yes

Describe the feature and the current behaviour/state

There should be a way to cancel the task created by generateResponseAsync(). Proposed API: add a cancel() method in the LlmInference class that cancels all ongoing requests, or provide a version of generateResponseAsync() that returns a Future

Will this change the current API? How?

No response

Who will benefit with this feature?

No response

Please specify the use cases for this feature

Applications that allow the user to cancel a query, either explicitly, or implicitly by submitting a new query before the last one has finished processing, would need this feature to avoid having to process a request that it no longer cares about.

Any Other info

No response

[kuaashish]

Hi @bringert,

Could you please provide more details about your query? This will help us understand it better and work on implementing the feature if possible.

Thank you!!

[bringert]

What additional information would you like? The idea is to be able to cancel generation before it completes.

[schmidt-sebastian]

Logged this as an internal feature request: b/335859045