Skip to content

Conversation

@kiranandcode
Copy link
Contributor

This PR refactors the encoding/decoding pipeline to go through a unified encode/decode operation.

  • effectful/handlers/llm/encoding.py implements an associated type morally equivalent to:

    structure Encodable (A: Type) where
        U: Type
        encode: A -> U
        decode: U -> A

    it provides a function type_to_encodable_type which does a sort of type-class resolution and returns an instance of Encodable for a given type T.

  • effectful/handlers/llm/provider.py now uniformly constructs these Encodable[T] objects and uses their encode and decode operations when calling the LLM.

The high level pipeline to implement Template.__call__(template, args):

  • use type_to_encodable_type to obtain an Encodable[T] for each type in the signature of template
  • use the associated encode operators to convert the arguments args to pydantic types
  • use serialize to map pydantic types to OpenAIMessages
  • call provider with message
  • while tool_calls:
    • use decode to map llm provided pydantic types back to python user types
    • call tool with arguments
    • use encode to map result to a pydantic type
    • use serialize to map result to OpenAIMessage
    • send back to LLM
  • take final call, use type_to_encodable_type to get Encodable for return type
  • use decode to map pydantic type to user type
  • return final result

I expect some discussion about the interface around Tool - for this initial version, I kept most of the API and logic the same, but we might want to restructure it a bit.

Another aspect is the encoding type resolution might be a bit overengineered. Currently it descends into types such as tuple[Image.Image, Image.Image] to construct an Encodable that does the right thing for nested types as well.

Copy link
Contributor

@eb8680 eb8680 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great. Two high-level questions in addition to the comments below:

  1. I'm not seeing the type_to_encodable_type logic for the case when the user type T is a pydantic.BaseModel subtype, or any test cases. Is that case meant to be the object case where encode and decode are identity mappings? Can you add tests that exercise this logic?
  2. What happened to deserialize? Have we decided to identify it with pydantic.BaseModel.model_validate_json?

@kiranandcode
Copy link
Contributor Author

kiranandcode commented Dec 12, 2025

@eb8680

  1. I'm not seeing the type_to_encodable_type logic for the case when the user type T is a pydantic.BaseModel subtype, or any test cases. Is that case meant to be the object case where encode and decode are identity mappings? Can you add tests that exercise this logic?

Yes, will do.

  1. What happened to deserialize? Have we decided to identify it with pydantic.BaseModel.model_validate_json?

Yes, from implementing this, a slight refinement on the design. Pydantic gives us:

serialize: U -> json
deserialize: json -> U

it could be that the serialize defop I've defined is misnamed, what it provides is actually serialize: U -> OpenAIMessage, for which we don't need an inverse.

@kiranandcode
Copy link
Contributor Author

@eb8680 @jfeser after some iterations, whittled down type_to_encodable_type a bit more. Now we have an implementation, what I'm realising is that for most case type_to_encodable_type is identity (U is the same type as T).

The one case where we don't want identity is for images:

@type_to_encodable_type.register(Image.Image)
class EncodableImage(_Encodable[Image.Image, ChatCompletionImageUrlObject]):
    t = ChatCompletionImageUrlObject

    @classmethod
    def encode(cls, image: Image.Image) -> ChatCompletionImageUrlObject:
        return {
            "detail": "auto",
            "url": _pil_image_to_base64_data_uri(image),
        }

    @classmethod
    def decode(cls, image: ChatCompletionImageUrlObject) -> Image.Image:
        image_url = image["url"]
        if not image_url.startswith("data:image/"):
            raise RuntimeError(
                f"expected base64 encoded image as data uri, received {image_url}"
            )
        data = image_url.split(",")[1]
        return Image.open(fp=io.BytesIO(base64.b64decode(data)))

and on top of this, we want non-identity mappings for two more cases on top of this: list[Image.Image], and tuple[....,Image.Image,...].

We map these types to list[ChatCompletionImageUrlObject] and tuple[...,ChatCompletionImageUrlObject,...] so when we use pydantic to serialize user-input that contains PIL.Image.Image's, we don't get an exception.

That being said, maybe this is incorrect? Pydantic won't complain now, but currently serialize will just call str on the encoded U, which will provide a string input to the llm, and return a list with a single ChatCompletionTextObject but maybe we'd want serialize to return a list of ChatCompletionImageUrlObjects.

adding a test for this functionality (whether tools that return list[Image] work as expected)

@kiranandcode
Copy link
Contributor Author

Test fails, as expected:

I retrieved a set of 5 images. Each image appears to be a simple placeholder graphic encoded in Base64. Here's a general description:

1. **Image 1 to 5:** All images are identical, consisting of a very basic visual placeholder. They are tiny (8x8 pixels) representations that lack distinctive features or detailed content, likely used just as a stand-in graphic, possibly for testing purposes. They don't contain any meaningful scenes or identifiable objects.

These are minimalistic graphics and don't offer much to describe. Please provide more specific criteria if you need a different type of image or additional detail!

@kiranandcode kiranandcode requested a review from jfeser December 12, 2025 20:31
Copy link
Contributor

@eb8680 eb8680 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, but @jfeser and @datvo06 should probably also sign off before we merge.

@kiranandcode kiranandcode requested a review from datvo06 December 13, 2025 07:40
Copy link

@datvo06 datvo06 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! LGTM!

@eb8680
Copy link
Contributor

eb8680 commented Dec 15, 2025

@jfeser any further comments before this lands?

Copy link
Contributor

@jfeser jfeser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few nits, but otherwise looks good!

@kiranandcode kiranandcode requested a review from jfeser December 15, 2025 15:55
@eb8680 eb8680 added this to the LLM Infrastructure milestone Dec 15, 2025
@jfeser jfeser merged commit 44d7d12 into staging-llm Dec 15, 2025
6 checks passed
@jfeser jfeser deleted the kg-encodable branch December 15, 2025 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants