FEAT add support for multimodal data from HarmBench

[romanlutz]

Is your feature request related to a problem? Please describe.

We have support for HarmBench text data as described in #270 but at the time decided to hold off on multimodal data. With prompt seed groups, we are able to capture this much better than we were a few months ago. A prompt seed group can show that all parts of a multimodal message (e.g., image + text) belong together (same group ID) and are sent in the same turn (same sequence number).

Describe the solution you'd like

We should make harmbench multimodal prompts available as a dataset of seed prompt groups.

Describe alternatives you've considered, if relevant

Additional context

[paulinek13]

Hi! I'd like to work on this if it's still up for grabs :)

[romanlutz]

Of course! It is a bit more complicated than the usual dataset issues but please feel free to ask questions to clarify.

[paulinek13]

@romanlutz thanks! yes, I need some guidance on how to approach this; it seems to me there are currently a few ways of making it work:

• (A): extending existing SeedPromptDataset class to support SeedPromptGroup as input (in the constructor), and then either flattening the prompts internally (there still is group_seed_prompts_by_prompt_group_id method to get the groups) or keeping them grouped (and updating all relevant methods to handle such case properly)

• (B): creating a new SeedPromptGroupDataset class that is specifically designed for multimodal datasets with grouped prompts

• (C): similar to (A), but instead of flattening the prompts in the constructor, doing that before (basically at the level of dataset fetch function, so that SeedPromptDataset remains unchanged), and again, user can just use group_seed_prompts_by_prompt_group_id to get the groups [this approach seems to be the simplest, but maybe not the most elegant?]

So, the question is, which of these approaches (or maybe some completely different one 😄) would be preferred?

Edit:

Also, I assume that I should create a new function, sth like fetch_harmbench_multimodal_dataset, rather than modifying the existing fetch_harmbench_dataset by adding a multimodal: bool = False argument. Is that correct?

[romanlutz]

It should be SeedPrompts per piece of text/image and they get the same group ID. Then these SeedPrompts become part of a dataset because the datset has a method to extract the groups. It's a bit convoluted right now, sorry.

I'm not sure I understand the difference between A and C.

Yes, new fetch function would be better 🙂