[Feature Request]: Input validations before calling LLM

[prabatdev]

Do you need to file an issue?

• I have searched the existing issues and this feature is not already filed.
• My model is hosted on OpenAI or Azure. If not, please look at the "model providers" issue and don't file a new one here.
• I believe this is a legitimate feature request, not just a question. If this is a question, please use the Discussions area.

Is your feature request related to a problem? Please describe.

There are instances where we get the following error messages. This looks to be due to some chunks in the documents are empty.

2025-02-02 12:12:11 [ERROR] 2025-02-02 12:12:11,281 - Error Invoking LLM

2025-02-02 12:12:11 openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid value for 'content': expected a string, got null.", 'type': 'invalid_request_error', 'param': 'messages.[1].content', 'code': None}}

Describe the solution you'd like

1. Do validations to see if the chunk has a valid content (should not be empty or null) before calling the LLM
2. When there are invalid chunks,

• Publish an information message saying that a particular chunk Id is invalid
• Publish a metric on the number of invalid chunks per workflow.

This will help in two ways

1. Avoid unnecessary calls to LLM reduces the cost and time
2. Lots of such error messages are confusing to debug

Additional context

No response