Traceable Evals

[CyrusNuevoDia]

  ,-~~-.___.
 / |  '     \
(  )         0
 \_/-, ,----'
    ====           //
   /  \-'~;    /~~~(O)
  /  __/~|   /       |
=(  _____| (_________|

from dspy.evaluate import tracer

@tracer
def number_to_bits(number):
    if number:
        bits = []
        while number:
            number, remainder = divmod(number, 2)
            bits.insert(0, remainder)
        return bits
    else:
        return [0]

str(number_to_bits.source) # source code with line numbers
"""
    1 | @tracer
    2 | def number_to_bits(number):
    3 |     if number:
    4 |         bits = []
    5 |         while number:
    6 |             number, remainder = divmod(number, 2)
    7 |             bits.insert(0, remainder)
    8 |         return bits
    9 |     else:
   10 |         return [0]
"""

number_to_bits(6)
str(number_to_bits.trace) # last execution with a trace like the one below
"""
 >>> Call to number_to_bits in File "...", line N
 ...... number = 6
    2 | def number_to_bits(number):
    3 |     if number:
    4 |         bits = []
    5 |         while number:
    6 |             number, remainder = divmod(number, 2)
 .................. number = 3
 .................. remainder = 0
    7 |             bits.insert(0, remainder)
 .................. bits = [0]
 .................. len(bits) = 1
    5 |         while number:
    6 |             number, remainder = divmod(number, 2)
 .................. number = 1
 .................. remainder = 1
    7 |             bits.insert(0, remainder)
 .................. bits = [1, 0]
 .................. len(bits) = 2
    5 |         while number:
    6 |             number, remainder = divmod(number, 2)
 .................. number = 0
    7 |             bits.insert(0, remainder)
 .................. bits = [1, 1, 0]
 .................. len(bits) = 3
    5 |         while number:
    8 |         return bits
 <<< Return value from number_to_bits: [1, 1, 0]
"""

Kudos to the wonderful https://github.com/alexmojaki/snoop

[CyrusNuevoDia]

Weird, tests pass locally for me but fail in the CI/CD. Also, the tests have nothing to do with what I've changed.

[mikeedjones]

The version of pydantic? there was a release to 2.10 recently. Looks like tests are failing when trying to work with pydantic models.

Looks like there's been a change to the way model_fields is created https://github.com/pydantic/pydantic/pull/10493/files

I'll make a fix to replace.

[mikeedjones]

Can you try with: #1855 please?

[CyrusNuevoDia]

@mikeedjones it works now!