Task Generation and Reproduction for CISO Scenarios

[yihangyao]

Dear authors,

Thank you for your excellent and systematic work on ITBench, it has been both inspiring and informative. I am currently studying the CISO scenarios and have a couple of questions regarding task generation. In Figure 25 of the ITBench paper, it is mentioned that there are multiple tasks per scenario (e.g., 10 tasks for Kyverno and 20 tasks for RHEL-OPA). However, I was unable to locate the corresponding configurations that define these individual tasks (e.g., for indices 1–20). Could you kindly guide me on how to generate or access the tasks shown in Figure 25 in order to reproduce the reported results?

Additionally, I would greatly appreciate a clarification on the flexibility of scenario configuration. Specifically, is it possible to generate additional tasks within a given scenario by modifying its configurations? For instance, could we create another set of 20 tasks for the Kyverno scenario through configuration changes?

Thank you very much for your time and attention. I look forward to your guidance.

Best,

Yihang

[yuji-watanabe-jp]

Dear Yihang,

Thank you very much for your kind message and for your interest in ITBench. We truly appreciate your thoughtful feedback and insightful questions.

Let me address your inquiries regarding task generation and scenario flexibility.

As you correctly pointed out, Figure 25 in the ITBench paper illustrates a total of 50 tasks distributed across four different scenario types. For each scenario type, multiple tasks were prepared during the time of writing. Currently, one representative example task per scenario type - four in total - is publicly available in the following repository. You can try them by following the instructions in the README: https://github.com/IBM/ITBench-Scenarios/tree/main/ciso

The remaining 46 tasks are not fully released in the repository, while they will become available soon through the upcoming ITBench service, which will allow anyone to run benchmark evaluations using the complete set of tasks, making the full benchmark experience publicly accessible.

Regarding your question on extending scenarios: yes, it is absolutely possible to create additional tasks - what we internally refer to as task scenarios - within an existing scenario type. Moreover, users are encouraged to contribute entirely new scenario types along with their own custom tasks. In fact, this extensibility and collaborative growth are central to the vision of ITBench. The initial set of 50 tasks is just the beginning. Our goal is to build a richer and more comprehensive benchmark suite for IT automation through community-driven contributions.

The upcoming ITBench service is being designed with full support for custom scenario and task contributions, and we plan to provide detailed guides to help users get started with building and sharing their own benchmarks.

We look forward to your continued interest and engagement with ITBench. Please feel free to reach out to me and @yana1205 (author of the 50 tasks) if you have further questions or ideas we’d love to hear them!

Thanks a lot!

[yihangyao]

Dear Yuji,

Thank you very much for your prompt and detailed responses! They’ve been extremely helpful! I’m truly excited about your impressive work and am looking forward to the full release of the ITBench service.

If possible, may I kindly ask if there is an estimated timeline for the public release of the full dataset for SRE and CISO?

Thank you again for your time and support! I really appreciate your help.

Best wishes,

Yihang