Tests are run where the test input is an empty string

[bzorn]

This was observed running llama3.3 as the PP model and in several samples, including extract-names and sentence rewrite. The tests in the tests_results.jon file look like this. The tests.csv file and the tests_results.json file are both attached.

test_results.json
tests.csv

  {
    "id": "1aa450e",
    "promptid": "1aa450e",
    "ruleid": null,
    "model": "gpt-4o-mini-2024-07-18",
    "input": "",
    "output": "[\"NA\"]",
    "compliance": "ok",
    "complianceText": "The given chatbot output is [\"NA\"]. According to the description provided between the delimiters <DESC> and </DESC>, if the chatbot does not find model names in the abstract or is unsure, it should return [\"NA\"]. Since the chatbot has returned [\"NA\"], this suggests that it is following the rules outlined for when model names are either not found or the chatbot is uncertain. There's no explicit violation of the format as described since [\"NA\"] does indeed represent an array with a single string value \"NA\" which aligns with the required response format.\n\nOK"
  },

[pelikhan]

what command do you use to run this?

[bzorn]

Haven't seen this issue lately.