OpenAgentSafety validation errors

[juanmichelini]

Doesn't seem to affect results.

Observed on #102 (change to commit number once merged)

Steps to reproduce

 uv run openagentsafety-infer ~/v1/benchmarks/.llm_config/sonnet-4.json \
     --dataset mgulavani/openagentsafety_full_updated_v3 \
     --split train \
     --output-dir ./results \
     --num-workers 10 \
     --n-limit 10 \
     --critic pass --note "test-422-$(date +%Y%m%d%H%M%S)"

see

pydantic_core._pydantic_core.ValidationError: 1 validation error for ObservationEvent
observation.ThinkObservation.content
Input should be a valid list [type=list_type, input_value='Your thought has been logged.', input_type=str]
For further information visit https://errors.pydantic.dev/2.11/v/list_type

[neubig]

Hi @juanmichelini and @MadhaviSG , can I check on this issue. Why is it that this error is happening in the first place?

It seems that maybe this is because @MadhaviSG created a different version of the agent server that is not compatible with the most recent version of OpenHands?

If this is the case, there may be a temporary patch to fix this, but it may break again in the future. I think that a better solution would be to make sure that the server that we're using for OpenAgentSafety remains compatible. @MadhaviSG could you explain a little bit more about the design and we can consider options?