Batch Prediction API

[sahinbatmaz]

Description of Problem:

I am pointing my feature request in NLU part. '/model/parse' api produces results one by one.

To shorten the response time, a batch prediction can be used. Especially, classifier layers take the most of the time and these models are capable of predicting batches of data.

Overview of the Solution:

An additional function can be added to 'Component' class as 'process_batch'. It can take a list of text as input, and in the response it produces results for each.

Or, if the input of 'process' function is not only one message but a list of messages, the process function can produce outputs.

For these things, /model/parse api should be updated or a new api should be introduced.

[sara-tagger]

Exalate commented:

sara-tagger commented:

Thanks for submitting this feature request

🚀
@akelad will get back to you about it soon!

✨

[sahinbatmaz]

Exalate commented:

sahinbatmaz commented:

Any updates on this issue?

[dakshvar22]

Exalate commented:

dakshvar22 commented:

@sahinbatmaz Thanks for suggesting this. Although, this makes sense from a technical perspective as it could speed up inference in some cases but not really for use cases Rasa Open Source focusses on. What's your exact use case for doing batch inferencing in Rasa Open Source?

[youmaa]

Exalate commented:

youmaa commented:

Any updates? For a use case, in my case, I needed this to apply an interpretability model based on prediction probabilities

[akelad]

Exalate commented:

akelad commented:

Could you elaborate a bit? As in, is it in the context of building a bot?

[youmaa]

Exalate commented:

youmaa commented:

@akelad sorry for the late reply it's actually after building a bot, I wanna apply an interpretability model (LIME for example) and it needs to make batch prediction from a performance perspective

[Ghostvv]

Exalate commented:

Ghostvv commented:

It is not a research issue, I moved it to Enable board

[sync-by-unito]

➤ Maxime Verger commented:

💡 Heads up! We're moving issues to Jira: https://rasa-open-source.atlassian.net/browse/OSS.

From now on, this Jira board is the place where you can browse (without an account) and create issues (you'll need a free Jira account for that). This GitHub issue has already been migrated to Jira and will be closed on January 9th, 2023. Do not forget to subscribe to the corresponding Jira issue!

➡️ More information in the forum: https://forum.rasa.com/t/migration-of-rasa-oss-issues-to-jira/56569.