Dealing with Content Policy Violation Errors

[hgoelen]

Hello,

For my task I am frequently confronted with content policy violation errors. This could certainly be expected, however I have not yet found a way to deal with these effectively.

My first idea was to first send a different prompt simply asking if the text I was to use in a dspy flow was funny or not (using the standard OpenAI library). If this prompt failed, I would then no longer consider this text in the dspy flow. To be clear, deciding whether a text is funny or not is not my end goal, this is merely a sample prompt to lure out content policy violation errors.

However, the prompt generated by dspy seems to trigger the content policy filter anyway for texts that passed my first test. My next idea was then to call the dspy module to be used in the compiler seperately from the compiler and see there, again, if the text triggers the content policy filter and remove it from the dataset. However, it seems that the module keeps sending the same text to ChatGPT over and over again, resulting in errors like this:

backoff - INFO - Backing off request(...) for 0.7s (openai.BadRequestError: Error code: 400 - {'error': {'message': "The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766", 'type': None, 'param': 'prompt', 'code': 'content_filter', 'status': 400, 'innererror': {'code': 'ResponsibleAIPolicyViolation', 'content_filter_result': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': True, 'severity': 'medium'}}}}}) Backing off 0.7 seconds after 3 tries calling function <function AzureOpenAI.request at 0x7f06521740d0> with kwargs {}

This does not stop after a single try, but rather keeps running, seemingly without a way to bypass it (this was already embedded in a try - except statement). The same thing happens if I run the module in the compiler.

Is there a way to deal with this, ideally without having to query ChatGPT twice for non-offending texts (i.e. directly in the compiler)?

Thanks in advance

[GautamVhavle]

The backoff you are seeing is DSPy retrying the same Azure content-filter 400. content_filter is not a transient network error, but the old AzureOpenAI.request (and current dspy.LM(..., num_retries=N)) still retries it with exponential backoff. Your try/except around the module never fires, because the retry loop is inside the LM client.

Stop the retries first:

lm = dspy.LM("azure/gpt-4o", num_retries=0, ...)
dspy.configure(lm=lm)

On current DSPy the failure becomes an LMError / wrapped BadRequestError you can catch. Filter the trainset by running the program once per example and dropping the ones that raise.

Inside a teleprompter, do not wrap the compiler. Teach the metric to treat a filter hit as a failed example (return False / 0) instead of raising. Bootstrap will skip that row instead of sitting in backoff.

A pre-filter prompt on the raw text will not match what DSPy actually sends. The compiled prompt is instruction + signature + demos + the example, so Azure's filter sees a different string than your "is this funny?" probe. That is why texts that passed the probe still die in the compiler.

There is no compiler flag that means "skip content-policy examples without calling the LM". The cheap version is num_retries=0 plus a metric/try that drops them. The expensive version (double LM call) is what you already tried, and it cannot see the compiled prompt anyway.