Config option to specify the intermediate step types in workflow_output.json

[AnuradhaKaruppiah]

Currently, only TOOL_OUTPUT and LLM_OUTPUT are included in the published workflow_output.json. But there are usecases where other types such as TOOL_START may asl be needed. This PR provides a config option in config.yml to allow the user to specify the filter.

Sample config (examples/simple/configs/eval_config.yml):

    eval:
      general:
        output:
          dir: ./.tmp/aiq/examples/simple/
          workflow_output_step_filter: [LLM_END, TOOL_START, TOOL_END]

Description

Closes

By Submitting this PR I confirm:

• I am familiar with the Contributing Guidelines.
• We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
  • Any contribution which contains commits that are not Signed-Off will not be accepted.
• When the PR is ready for review, new or existing tests cover these changes.
• When the PR is ready for review, the documentation is up to date with these changes.

[Copilot]

Pull Request Overview

This PR introduces a configuration option in config.yml to allow users to specify which intermediate step types are included in the generated workflow_output.json. Key changes include:

• Adding a new configuration parameter (workflow_output_step_filter) to control intermediate step filtering.
• Updating the dataset_handler and evaluate modules to accept and apply the new filter.
• Enhancing the test suite and documentation with examples and tests for the new functionality.

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/aiq/eval/dataset_handler/test_dataset_handler.py	Added tests and fixtures to validate intermediate step filtering behavior.
src/aiq/eval/evaluate.py	Updated publish_eval_input call to pass the filter configuration.
src/aiq/eval/dataset_handler/dataset_handler.py	Extended filter_intermediate_steps and publish_eval_input to support the new filter option; added type hints.
src/aiq/data_models/evaluate.py	Updated the EvalOutputConfig model to include the new filter field.
docs/source/concepts/evaluate.md	Documented the new workflow_output_step_filter option with an example.

Comments suppressed due to low confidence (2)

src/aiq/eval/dataset_handler/dataset_handler.py:149

• Fix typo: replace 'orginal' with 'original' in the docstring.

Convert the EvalInput object to a JSON output for storing in a file. Use the orginal keys to

src/aiq/eval/dataset_handler/dataset_handler.py:150

• Fix typo: replace 'orignal' with 'original' in the docstring.

allow re-running evaluation using the orignal config file and '--skip_workflow' option.

[AnuradhaKaruppiah]

/merge

[ssling0817]

Hi @AnuradhaKaruppiah, does this update allow users to add for example "standard output" to the intermediate steps, so users can see the outputs in the UI? We can only see the following types in the image, and I am wondering if I can add standard output to the intermediate steps here. Thank you.