To run functional.py example locally, I need to escape """ in Python docstrings

[simonff]

Somehow, https://github.com/stanfordnlp/dspy/blob/main/examples/functional/functional.ipynb works for me on github, but when I try to run the

predictor = TypedPredictor(CodeSignature)
prediction = predictor(
    prompt=PythonCode(code=ds['test'][0]['prompt']),
    test=PythonCode(code=ds['test'][0]['test']),
    entry_point=ds['test'][0]['entry_point']
)

print(prediction)

section in a regular Python script, I keep getting JSON decoding errors.

To fix them, I had to change in dspy/functional/functional.py the return value of _unwrap_json():
return output.replace('"""', '\"\"\"')

Otherwise the """ in the generated code confuse JSON parser (you can replicate by just using json.loads() on the string below):

'{"code": "from typing import List\\n\\n\\ndef has_close_elements(numbers: List[float], threshold: float) -> bool:\\n    \\"\\"\\" Check if in given list of numbers, are any two numbers closer to each other than\\n    given threshold.\\n    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)\\n    False\\n    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)\\n    True\\n    \\"\\"\\"\\n    for i in range(len(numbers)):\\n        for j in range(i + 1, len(numbers)):\\n            if abs(numbers[i] - numbers[j]) < threshold:\\n                return True\\n    return False"}'

[okhat]

Thank you @simonff ! Could be of interest to @thomasahle but it sounds like you sort of have a PR in mind @simonff ?

[simonff]

I need to get approval from work for submitting PRs, and I'm not sure I have all the right context here anyway, so I'd rather wait a bit for someone more knowledgeable to comment.

[thomasahle]

This is super interesting. I'll have a look. What language model are you using? Gpt-3.5-turbo?

[simonff]

Yes, gpt-3.5-turbo and also gemini-1.0-pro

Sorry for the typo above, the replacement string should have double backslash escapes before each quote.

[thomasahle]

I think the issue was actually that the signature was mistakenly defined to be:

class CodeSignature(Signature):
    prompt: str = InputField()
    test: PythonCode = InputField()
    entry_point: str = InputField()
    solution: PythonCode = OutputField()

Instead of

class CodeSignature(Signature):
    prompt: PythonCode = InputField()
    test: PythonCode = InputField()
    entry_point: str = InputField()
    solution: PythonCode = OutputField()

I fixed that now, and also added some robustness via parsing through ujson instead of json.
Do you mind checking in my fork whether it works for you as well?

[simonff]

Thank you, Thomas! Can confirm the problem is gone if I use your fork.