Fix RetryHandler not to use Template.replace() #483
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
eb8680
reviewed
Jan 13, 2026
Contributor
eb8680
left a comment
eb8680
left a comment
There was a problem hiding this comment.
Could we not just remove Template.replace in this PR? Where else is it used in staging-llm now that the synthesis functionality is being held out for the next release?
effectful/handlers/llm/providers.py
Outdated
| @implements(Template.__apply__) | ||
| def _retry_completion(self, template: Template, *args, **kwargs) -> Any: | ||
| """Retry template execution with error feedback injection.""" | ||
| self._error_feedback = "" |
Contributor
There was a problem hiding this comment.
It's not correct to have this state live on RetryLLMHandler - nested Template calls with failures will overwrite each other. It should be in a separate handler like InstructionsHandler in #484.
Contributor
Author
There was a problem hiding this comment.
I see, thanks! Let me look at #484 and remove Template.replace also.
Contributor
Author
There was a problem hiding this comment.
@eb8680 Thank you,
I see now. It might be the case that the error happens inside a nested template, so only the nested one would be corrected, but not the outer one get corrected.
In that case, we might want to install an error feedback handler (InstructionHandler) specifically designed for the current scope. Because these handlers are installed according to the calling, they'd handle the right failure for the right template.
I'll implement this now.
eb8680
approved these changes
Jan 14, 2026
eb8680
added a commit
that referenced
this pull request
Jan 15, 2026
* Staging branch for LLM module * Move LLM interface code from `robotl` (#358) * import llm code from robotl * make optional deps optional * replace template with Template.define * bring in tests * typing bullshit * disable incremental type checking to avoid crash * format * remove anthropic and cache modules * fix * remove dupe * rename * fix not handled * reorganize * wip * wip * wip * wip * restrict python version * specify python version * rename * rename * Implement basic tool calling (#366) * import llm code from robotl * make optional deps optional * replace template with Template.define * bring in tests * typing bullshit * disable incremental type checking to avoid crash * format * remove anthropic and cache modules * fix * remove dupe * rename * fix not handled * reorganize * wip * wip * wip * wip * restrict python version * specify python version * rename * work on a tool call interface * wip * wip * format * wip * revert whitespace changes * fix decorator calls * lint * lint * allow no arguments to decorator * work * enable strict mode for tool calling (#375) * add structured generation and remove unused `decode` operation (#376) * implemented support for class methods in `Template.define` (#377) * Revert "implemented support for class methods in `Template.define` (#377)" This reverts commit 02c4378. * Add support for methods in `Template.define` (#377) (#378) * Adding a lower-level event and a logger example (#382) * Adding a lower-level event and a logger example * Bring back the note * Linting * Add a logging handler that expose python logging * Lint * Miscs * Update LoggingHandler interface * Linting and fixing signatures * Add support for tools returning images (#385) * added support for images * updated conversion to be delegated through a dynamic pydantic model instead * Revert "updated conversion to be delegated through a dynamic pydantic model instead" This reverts commit d41e810. * updated implementation to use OpenAI types and unify conversion logic under Tool. * Implement Caching Handler for LLM (#392) * Adding caching handler * Adding caching handler * Linting and cleaner implementation of hashing keys * Minor linting * Slightly more precise type for cacher * Slightly more precise type for cacher * Addressing comments * Attempt to fix issue with pypandoc * Fixing CI * Linting * implement first to k-ahead sampler (#412) * Add inheritable class for stateful templates (#416) * refactor OpenAIAPIProvider Template.__call__ now desugars into more primitive operations * add an inheritable class that provides stateful behavior to templates * remove unnecessary unset * Support multiple providers (via `litellm`) (#418) * add litellm to the dependencies for effectful llm * switched internals to use litellm & fixed bug with dicts * updated schemas and imports * fixed encoding-decoding issue with shim * switched to completions and litellm * removed old compatibility shim * replaced openai type with litellm type * s/LLMProvider/LiteLLMProvider/ * modified LiteLLMProvider to inject params into llm_request * fixed ipynb imports * updated s/llm_request/completion and functools.wraps * s/llm_request/completion * dropped try-catch on litellm import * updated providers and dropped openai dep * updated completion to be more compositional * minor fix for single positional argument for completion * updated init to default to gpt-4o to preserve previous behaviour * store source of generated functions in `__src__` attribute (#403) * updates ProgramSynthesis to use linecache and integrate with inspect.getsource * Adds type-based encoding and support for legacy APIs (#411) * refactored encoding to be type directed * updated handling of str and bytes * Add LLM Integration tests to the workflows. (#420) * Merge master into llm-staging (#423) * Release v0.2.3 (#374) * Install prettyprinter for term when library is available (#386) * install prettyprinter for term when library is available * lint * move code into types.py * fix pypandoc issue (#397) * Convert evaluate to a singledispatch (#398) * convert evaluate to a singledispatch * lint * add jnp.pi and ArrayTerm.T (#394) * Deprecate defterm (#399) * deprecate defterm * remove defterm case * remove defterm * lint * evaluate distribution arguments * lint * remove interpreter * Revert "remove interpreter" This reverts commit 3044277. * wip * lint * Rework numpyro distribution handling to enable symbolic distributions and handling of distribution methods (#311) * refactor distribution operations * add a test for typeof of distributions * add tests for symbolic dists/arguments * introduce operations for distribution methods * comment * fix tests * work around #310 * replace hack with new hack * tweak repr for _BaseOperation * lint * work around #312 * clean up access to dist ops * wip * wip * add type annotations to get correct term conversion * lint * include distribution arguments as properties * fix distribution calls * try again * fixes * format * Box the output of `__type_rule__` (#387) * box the output of __type_rule__ * fix tests * fix tests * require callers of __type_rule__ to box arguments * fix * move Box out of ops.types * lint * fix test * fix syntactic_eq implementation for jax arrays (#405) * Fix recursion error in sizesof (#406) * fix recursion error in sizesof * format * Allow `_BaseOperation` subclasses to have an overrideable `apply` method (#414) * stash * fixes * initial * wip * lint * ensure each subclass has a fresh operation * wip * wip * lint * wip * wip * lint * refactor class method support * move defops * fix test * remove singledispatch case and add test * move definition * cleanup * simplify * cleanup * lint * fix failing test * fix classmethod * __isabstractmethod__ * revert --------- Co-authored-by: Eli <eli@basis.ai> * lint --------- Co-authored-by: Tim Cooijmans <cooijmans.tim@gmail.com> Co-authored-by: Eli <eli@basis.ai> * Fix `staging-llm` diff against `master` (#426) * Release v0.2.3 (#374) * Install prettyprinter for term when library is available (#386) * install prettyprinter for term when library is available * lint * move code into types.py * fix pypandoc issue (#397) * Convert evaluate to a singledispatch (#398) * convert evaluate to a singledispatch * lint * add jnp.pi and ArrayTerm.T (#394) * Deprecate defterm (#399) * deprecate defterm * remove defterm case * remove defterm * lint * evaluate distribution arguments * lint * remove interpreter * Revert "remove interpreter" This reverts commit 3044277. * wip * lint * Rework numpyro distribution handling to enable symbolic distributions and handling of distribution methods (#311) * refactor distribution operations * add a test for typeof of distributions * add tests for symbolic dists/arguments * introduce operations for distribution methods * comment * fix tests * work around #310 * replace hack with new hack * tweak repr for _BaseOperation * lint * work around #312 * clean up access to dist ops * wip * wip * add type annotations to get correct term conversion * lint * include distribution arguments as properties * fix distribution calls * try again * fixes * format * Box the output of `__type_rule__` (#387) * box the output of __type_rule__ * fix tests * fix tests * require callers of __type_rule__ to box arguments * fix * move Box out of ops.types * lint * fix test * fix syntactic_eq implementation for jax arrays (#405) * Fix recursion error in sizesof (#406) * fix recursion error in sizesof * format * Allow `_BaseOperation` subclasses to have an overrideable `apply` method (#414) * stash * fixes * initial * wip * lint * ensure each subclass has a fresh operation * wip * wip * lint * wip * wip * lint * refactor class method support * move defops * fix test * remove singledispatch case and add test * move definition * cleanup * simplify * cleanup * lint * fix failing test * fix classmethod * __isabstractmethod__ * revert --------- Co-authored-by: Eli <eli@basis.ai> * Try pulling in pyproject.toml from staging-llm to master (#425) --------- Co-authored-by: Jack Feser <jack.feser@gmail.com> Co-authored-by: Tim Cooijmans <cooijmans.tim@gmail.com> * Implement a RetryHandler for LLM module (#428) * Logging retries * Remove redundant assignment * Linting * Adding mypy check and test * SynthesizedFunction for constrained decoding, adapt the tests * Linting * Let LLM generate param names * Pydantic field annotation * Adding validation error example * Linting * Fix minor formatting for type context * Update llm dependencies to include mypy * More comprehensive error message * linting * More comprehensive import * Fix `staging-llm` diff against `master` (#426) * Release v0.2.3 (#374) * Install prettyprinter for term when library is available (#386) * install prettyprinter for term when library is available * lint * move code into types.py * fix pypandoc issue (#397) * Convert evaluate to a singledispatch (#398) * convert evaluate to a singledispatch * lint * add jnp.pi and ArrayTerm.T (#394) * Deprecate defterm (#399) * deprecate defterm * remove defterm case * remove defterm * lint * evaluate distribution arguments * lint * remove interpreter * Revert "remove interpreter" This reverts commit 3044277. * wip * lint * Rework numpyro distribution handling to enable symbolic distributions and handling of distribution methods (#311) * refactor distribution operations * add a test for typeof of distributions * add tests for symbolic dists/arguments * introduce operations for distribution methods * comment * fix tests * work around #310 * replace hack with new hack * tweak repr for _BaseOperation * lint * work around #312 * clean up access to dist ops * wip * wip * add type annotations to get correct term conversion * lint * include distribution arguments as properties * fix distribution calls * try again * fixes * format * Box the output of `__type_rule__` (#387) * box the output of __type_rule__ * fix tests * fix tests * require callers of __type_rule__ to box arguments * fix * move Box out of ops.types * lint * fix test * fix syntactic_eq implementation for jax arrays (#405) * Fix recursion error in sizesof (#406) * fix recursion error in sizesof * format * Allow `_BaseOperation` subclasses to have an overrideable `apply` method (#414) * stash * fixes * initial * wip * lint * ensure each subclass has a fresh operation * wip * wip * lint * wip * wip * lint * refactor class method support * move defops * fix test * remove singledispatch case and add test * move definition * cleanup * simplify * cleanup * lint * fix failing test * fix classmethod * __isabstractmethod__ * revert --------- Co-authored-by: Eli <eli@basis.ai> * Try pulling in pyproject.toml from staging-llm to master (#425) --------- Co-authored-by: Jack Feser <jack.feser@gmail.com> Co-authored-by: Tim Cooijmans <cooijmans.tim@gmail.com> * Logging retries * Make linter happy with the exception_cls * More linting * Revert the changes to synthesis, it belongs to a different PR * Add test for retry handler * More linting * Removing mypy for llm --------- Co-authored-by: eb8680 <eb8680@users.noreply.github.com> Co-authored-by: Jack Feser <jack.feser@gmail.com> Co-authored-by: Tim Cooijmans <cooijmans.tim@gmail.com> * Merge `master` into `staging-llm` again (#443) * Release v0.2.3 (#374) * Install prettyprinter for term when library is available (#386) * install prettyprinter for term when library is available * lint * move code into types.py * fix pypandoc issue (#397) * Convert evaluate to a singledispatch (#398) * convert evaluate to a singledispatch * lint * add jnp.pi and ArrayTerm.T (#394) * Deprecate defterm (#399) * deprecate defterm * remove defterm case * remove defterm * lint * evaluate distribution arguments * lint * remove interpreter * Revert "remove interpreter" This reverts commit 3044277. * wip * lint * Rework numpyro distribution handling to enable symbolic distributions and handling of distribution methods (#311) * refactor distribution operations * add a test for typeof of distributions * add tests for symbolic dists/arguments * introduce operations for distribution methods * comment * fix tests * work around #310 * replace hack with new hack * tweak repr for _BaseOperation * lint * work around #312 * clean up access to dist ops * wip * wip * add type annotations to get correct term conversion * lint * include distribution arguments as properties * fix distribution calls * try again * fixes * format * Box the output of `__type_rule__` (#387) * box the output of __type_rule__ * fix tests * fix tests * require callers of __type_rule__ to box arguments * fix * move Box out of ops.types * lint * fix test * fix syntactic_eq implementation for jax arrays (#405) * Fix recursion error in sizesof (#406) * fix recursion error in sizesof * format * Allow `_BaseOperation` subclasses to have an overrideable `apply` method (#414) * stash * fixes * initial * wip * lint * ensure each subclass has a fresh operation * wip * wip * lint * wip * wip * lint * refactor class method support * move defops * fix test * remove singledispatch case and add test * move definition * cleanup * simplify * cleanup * lint * fix failing test * fix classmethod * __isabstractmethod__ * revert --------- Co-authored-by: Eli <eli@basis.ai> * Try pulling in pyproject.toml from staging-llm to master (#425) * Generate instance-level `Operation`s for bound methods (#351) * generalize __get__ * nits * coverage of methoddescriptor api * methodtype * simplify * simplify * simplify * format * revert * restore * simplify * simplify * retain instance op on term construction * Simplify apply inheritance * assign * put call next to init_subclass * add explanatory comment * Operation.apply -> Operation.__apply__ * add test based on issue description * fix doctest * Fix dataclass @defops and added dataclass metaclass (#439) * fixed dataclass ordering and added metaclass for simplifying construction of dataclass terms * ensure term fields are not being overriden * added decorator and dataclass * updated to make defdata registration automatic * simplified dataclass loop * updated to give property op an appropriate name * added failing tests * fixed failing test * fixed numpyro/pyro/torch interfaces * minor fix + test for deffn kwargs --------- Co-authored-by: Jack Feser <jack.feser@gmail.com> Co-authored-by: Tim Cooijmans <cooijmans.tim@gmail.com> Co-authored-by: Kiran Gopinathan <23038502+kiranandcode@users.noreply.github.com> * Implements a unified `encode`ing/`decode`ing pipeline for `llm` (#442) * implemented unified encoding type * implemented decoding * unified __init__ * added tests for basemodels * s/@property/@functools.cached_property/ * type for encode and decode * removed handling for numbers.Number and explicit tests for complex * fixed is_dataclass checks * updated to check parameter annotations in Tool.of_operation constructor * updated serializer to be more selective in what is an image * reducing number of #type: ignores, and switching to typing.Any * removed comment * dropped dataclass support * dropped tests for dataclass with image * updated dataclass tests to stop assuming pydantic models * test for tool that returns list of images * made serialization a parameter of encodable and thus type-directed * dropped test for tool that returns list of images * dropped registration of encodable types * dropped unused typevar * s/_Encodable/EncodableAs/ * Initial version of Lexical Context Collection - Collecting Tools and Template (#434) * Adding lexical context collection * Allow model input to refer to anything within the lexical context * Different handling between representable object and other types when referred to in Template * More edge case handling * More edge case handling * More edge case handling * Register more instead of _get_source_for_object * Tune down the scope to only collect relevant tools templates * Constructing Tool from Template, allowing higher-order Tempalte * Linting * Fix exception type * Fix exception type to NotHandled * Fix stringinified annotation * Fix tool.define signature * Trim Tool.define * Trip format_value.register * Add warning on no doc case * Removing default for __name__ * More specific lexical context type annotation * Additional tools for pydandic-compatible conversion and recursion depth limiting * Linting * Linting * Linting * Linting * Minor fixes * Merge encoding/decoding * Update a lot of prompt with "do not use any tools" * Update a lot of prompt with "do not use any tools" * More tool limiting prompts * Lint * Lint * Factoring out book and poem tools after multiple failed prompt modification attempts * Revert semantics * Lint * Update name of lexical_context to __context__ * Lint * Minor * More prompt modification * Make sure notebook run * Update `staging-llm` from `master` (#457) * Release v0.2.3 (#374) * Install prettyprinter for term when library is available (#386) * install prettyprinter for term when library is available * lint * move code into types.py * fix pypandoc issue (#397) * Convert evaluate to a singledispatch (#398) * convert evaluate to a singledispatch * lint * add jnp.pi and ArrayTerm.T (#394) * Deprecate defterm (#399) * deprecate defterm * remove defterm case * remove defterm * lint * evaluate distribution arguments * lint * remove interpreter * Revert "remove interpreter" This reverts commit 3044277. * wip * lint * Rework numpyro distribution handling to enable symbolic distributions and handling of distribution methods (#311) * refactor distribution operations * add a test for typeof of distributions * add tests for symbolic dists/arguments * introduce operations for distribution methods * comment * fix tests * work around #310 * replace hack with new hack * tweak repr for _BaseOperation * lint * work around #312 * clean up access to dist ops * wip * wip * add type annotations to get correct term conversion * lint * include distribution arguments as properties * fix distribution calls * try again * fixes * format * Box the output of `__type_rule__` (#387) * box the output of __type_rule__ * fix tests * fix tests * require callers of __type_rule__ to box arguments * fix * move Box out of ops.types * lint * fix test * fix syntactic_eq implementation for jax arrays (#405) * Fix recursion error in sizesof (#406) * fix recursion error in sizesof * format * Allow `_BaseOperation` subclasses to have an overrideable `apply` method (#414) * stash * fixes * initial * wip * lint * ensure each subclass has a fresh operation * wip * wip * lint * wip * wip * lint * refactor class method support * move defops * fix test * remove singledispatch case and add test * move definition * cleanup * simplify * cleanup * lint * fix failing test * fix classmethod * __isabstractmethod__ * revert --------- Co-authored-by: Eli <eli@basis.ai> * Try pulling in pyproject.toml from staging-llm to master (#425) * Generate instance-level `Operation`s for bound methods (#351) * generalize __get__ * nits * coverage of methoddescriptor api * methodtype * simplify * simplify * simplify * format * revert * restore * simplify * simplify * retain instance op on term construction * Simplify apply inheritance * assign * put call next to init_subclass * add explanatory comment * Operation.apply -> Operation.__apply__ * add test based on issue description * fix doctest * Fix dataclass @defops and added dataclass metaclass (#439) * fixed dataclass ordering and added metaclass for simplifying construction of dataclass terms * ensure term fields are not being overriden * added decorator and dataclass * updated to make defdata registration automatic * simplified dataclass loop * updated to give property op an appropriate name * added failing tests * fixed failing test * fixed numpyro/pyro/torch interfaces * minor fix + test for deffn kwargs * Type check and lint example code (#449) * format example code * type check examples * Add beam search example using thermometer continuations (#431) * add beam search example using thermometer continuations * address comments * add docstring * lint * Fix for jax 0.8.2 (#455) * fix for jax 0.8.2 * add more register * format --------- Co-authored-by: Jack Feser <jack.feser@gmail.com> Co-authored-by: Tim Cooijmans <cooijmans.tim@gmail.com> Co-authored-by: Kiran Gopinathan <23038502+kiranandcode@users.noreply.github.com> * Convert `Template` into an operation (#424) * Release v0.2.3 (#374) * stash * Install prettyprinter for term when library is available (#386) * install prettyprinter for term when library is available * lint * move code into types.py * fixes * fix pypandoc issue (#397) * Convert evaluate to a singledispatch (#398) * convert evaluate to a singledispatch * lint * add jnp.pi and ArrayTerm.T (#394) * Deprecate defterm (#399) * deprecate defterm * remove defterm case * remove defterm * lint * evaluate distribution arguments * lint * remove interpreter * Revert "remove interpreter" This reverts commit 3044277. * wip * lint * Rework numpyro distribution handling to enable symbolic distributions and handling of distribution methods (#311) * refactor distribution operations * add a test for typeof of distributions * add tests for symbolic dists/arguments * introduce operations for distribution methods * comment * fix tests * work around #310 * replace hack with new hack * tweak repr for _BaseOperation * lint * work around #312 * clean up access to dist ops * wip * wip * add type annotations to get correct term conversion * lint * include distribution arguments as properties * fix distribution calls * try again * fixes * format * Box the output of `__type_rule__` (#387) * box the output of __type_rule__ * fix tests * fix tests * require callers of __type_rule__ to box arguments * fix * move Box out of ops.types * lint * fix test * fix syntactic_eq implementation for jax arrays (#405) * Fix recursion error in sizesof (#406) * fix recursion error in sizesof * format * initial * wip * lint * ensure each subclass has a fresh operation * wip * wip * lint * wip * wip * lint * refactor class method support * move defops * fix test * remove singledispatch case and add test * move definition * cleanup * simplify * cleanup * lint * fix failing test * wip * Allow `_BaseOperation` subclasses to have an overrideable `apply` method (#414) * stash * fixes * initial * wip * lint * ensure each subclass has a fresh operation * wip * wip * lint * wip * wip * lint * refactor class method support * move defops * fix test * remove singledispatch case and add test * move definition * cleanup * simplify * cleanup * lint * fix failing test * fix classmethod * __isabstractmethod__ * revert --------- Co-authored-by: Eli <eli@basis.ai> * wip * fix define signature * wip * wip * wip * wip * wip * wip * wip * lint * wip * test fix * fix merge error * lint * wip * respond to feedback * respond to feedback * clean up notebook * remove breakpoint * relax typing requirement * improve handling of method templates * lint * remove imprecise check * handle nested classes using qualname * feedback * fix scoping issue and add tests * address feedback * add xfail tests * lint --------- Co-authored-by: Eli <eli@basis.ai> Co-authored-by: Tim Cooijmans <cooijmans.tim@gmail.com> Co-authored-by: eb8680 <eb8680@users.noreply.github.com> * Fail when encoding terms or operations (#474) * raise error when encoding terms or operations * change error type * Implemented record and replay fixtures for LLM calls (#467) * implemented record and replay fixtures for LLM calls * changed defaults to not rebuild fixtures, with dedicated make rule to rebuild * switched pytest to use request fixture instead of env * Remove program synthesis code (#475) * remove program synthesis code * lint * mark xfail * Disables direct recursion on templates by default (#466) * fixed failing test * made templates non-recursive by default + annotation for recursion * added docstring for IsRecursive * call IsRecursive.infer_annotations in tool's init * added check for sig.parameters * drop k-ahead sampler (#479) * Document `Template` and `Tool` (#478) * add documentation to template * Add dataclass import --------- Co-authored-by: eb8680 <eb8680@users.noreply.github.com> * Remove `agent.py` from handler and add docs on how to implement Agents (#464) * added agent example * Fix main check condition in agent.py * removed agent * Fix RetryHandler not to use Template.replace() (#483) * Working notebook * Update notebook * Linting * Remove Template.replace * Linting * Update notebook and linting * Linting * Rename `handlers.llm.providers` to `completions` (#485) * Rename handlers.llm.providers -> completions * lint --------- Co-authored-by: Jack Feser <jack.feser@gmail.com> Co-authored-by: Kiran Gopinathan <kiran@basis.ai> Co-authored-by: Kiran Gopinathan <23038502+kiranandcode@users.noreply.github.com> Co-authored-by: Dat Nguyen (Marc) <15943389+datvo06@users.noreply.github.com> Co-authored-by: Tim Cooijmans <cooijmans.tim@gmail.com>
eb8680
added a commit
that referenced
this pull request
Jan 15, 2026
* Staging branch for LLM module * Move LLM interface code from `robotl` (#358) * import llm code from robotl * make optional deps optional * replace template with Template.define * bring in tests * typing bullshit * disable incremental type checking to avoid crash * format * remove anthropic and cache modules * fix * remove dupe * rename * fix not handled * reorganize * wip * wip * wip * wip * restrict python version * specify python version * rename * rename * Implement basic tool calling (#366) * import llm code from robotl * make optional deps optional * replace template with Template.define * bring in tests * typing bullshit * disable incremental type checking to avoid crash * format * remove anthropic and cache modules * fix * remove dupe * rename * fix not handled * reorganize * wip * wip * wip * wip * restrict python version * specify python version * rename * work on a tool call interface * wip * wip * format * wip * revert whitespace changes * fix decorator calls * lint * lint * allow no arguments to decorator * work * enable strict mode for tool calling (#375) * add structured generation and remove unused `decode` operation (#376) * implemented support for class methods in `Template.define` (#377) * Revert "implemented support for class methods in `Template.define` (#377)" This reverts commit 02c4378. * Add support for methods in `Template.define` (#377) (#378) * Adding a lower-level event and a logger example (#382) * Adding a lower-level event and a logger example * Bring back the note * Linting * Add a logging handler that expose python logging * Lint * Miscs * Update LoggingHandler interface * Linting and fixing signatures * Add support for tools returning images (#385) * added support for images * updated conversion to be delegated through a dynamic pydantic model instead * Revert "updated conversion to be delegated through a dynamic pydantic model instead" This reverts commit d41e810. * updated implementation to use OpenAI types and unify conversion logic under Tool. * Implement Caching Handler for LLM (#392) * Adding caching handler * Adding caching handler * Linting and cleaner implementation of hashing keys * Minor linting * Slightly more precise type for cacher * Slightly more precise type for cacher * Addressing comments * Attempt to fix issue with pypandoc * Fixing CI * Linting * implement first to k-ahead sampler (#412) * Add inheritable class for stateful templates (#416) * refactor OpenAIAPIProvider Template.__call__ now desugars into more primitive operations * add an inheritable class that provides stateful behavior to templates * remove unnecessary unset * Support multiple providers (via `litellm`) (#418) * add litellm to the dependencies for effectful llm * switched internals to use litellm & fixed bug with dicts * updated schemas and imports * fixed encoding-decoding issue with shim * switched to completions and litellm * removed old compatibility shim * replaced openai type with litellm type * s/LLMProvider/LiteLLMProvider/ * modified LiteLLMProvider to inject params into llm_request * fixed ipynb imports * updated s/llm_request/completion and functools.wraps * s/llm_request/completion * dropped try-catch on litellm import * updated providers and dropped openai dep * updated completion to be more compositional * minor fix for single positional argument for completion * updated init to default to gpt-4o to preserve previous behaviour * store source of generated functions in `__src__` attribute (#403) * updates ProgramSynthesis to use linecache and integrate with inspect.getsource * Adds type-based encoding and support for legacy APIs (#411) * refactored encoding to be type directed * updated handling of str and bytes * Add LLM Integration tests to the workflows. (#420) * Merge master into llm-staging (#423) * Release v0.2.3 (#374) * Install prettyprinter for term when library is available (#386) * install prettyprinter for term when library is available * lint * move code into types.py * fix pypandoc issue (#397) * Convert evaluate to a singledispatch (#398) * convert evaluate to a singledispatch * lint * add jnp.pi and ArrayTerm.T (#394) * Deprecate defterm (#399) * deprecate defterm * remove defterm case * remove defterm * lint * evaluate distribution arguments * lint * remove interpreter * Revert "remove interpreter" This reverts commit 3044277. * wip * lint * Rework numpyro distribution handling to enable symbolic distributions and handling of distribution methods (#311) * refactor distribution operations * add a test for typeof of distributions * add tests for symbolic dists/arguments * introduce operations for distribution methods * comment * fix tests * work around #310 * replace hack with new hack * tweak repr for _BaseOperation * lint * work around #312 * clean up access to dist ops * wip * wip * add type annotations to get correct term conversion * lint * include distribution arguments as properties * fix distribution calls * try again * fixes * format * Box the output of `__type_rule__` (#387) * box the output of __type_rule__ * fix tests * fix tests * require callers of __type_rule__ to box arguments * fix * move Box out of ops.types * lint * fix test * fix syntactic_eq implementation for jax arrays (#405) * Fix recursion error in sizesof (#406) * fix recursion error in sizesof * format * Allow `_BaseOperation` subclasses to have an overrideable `apply` method (#414) * stash * fixes * initial * wip * lint * ensure each subclass has a fresh operation * wip * wip * lint * wip * wip * lint * refactor class method support * move defops * fix test * remove singledispatch case and add test * move definition * cleanup * simplify * cleanup * lint * fix failing test * fix classmethod * __isabstractmethod__ * revert --------- Co-authored-by: Eli <eli@basis.ai> * lint --------- Co-authored-by: Tim Cooijmans <cooijmans.tim@gmail.com> Co-authored-by: Eli <eli@basis.ai> * Fix `staging-llm` diff against `master` (#426) * Release v0.2.3 (#374) * Install prettyprinter for term when library is available (#386) * install prettyprinter for term when library is available * lint * move code into types.py * fix pypandoc issue (#397) * Convert evaluate to a singledispatch (#398) * convert evaluate to a singledispatch * lint * add jnp.pi and ArrayTerm.T (#394) * Deprecate defterm (#399) * deprecate defterm * remove defterm case * remove defterm * lint * evaluate distribution arguments * lint * remove interpreter * Revert "remove interpreter" This reverts commit 3044277. * wip * lint * Rework numpyro distribution handling to enable symbolic distributions and handling of distribution methods (#311) * refactor distribution operations * add a test for typeof of distributions * add tests for symbolic dists/arguments * introduce operations for distribution methods * comment * fix tests * work around #310 * replace hack with new hack * tweak repr for _BaseOperation * lint * work around #312 * clean up access to dist ops * wip * wip * add type annotations to get correct term conversion * lint * include distribution arguments as properties * fix distribution calls * try again * fixes * format * Box the output of `__type_rule__` (#387) * box the output of __type_rule__ * fix tests * fix tests * require callers of __type_rule__ to box arguments * fix * move Box out of ops.types * lint * fix test * fix syntactic_eq implementation for jax arrays (#405) * Fix recursion error in sizesof (#406) * fix recursion error in sizesof * format * Allow `_BaseOperation` subclasses to have an overrideable `apply` method (#414) * stash * fixes * initial * wip * lint * ensure each subclass has a fresh operation * wip * wip * lint * wip * wip * lint * refactor class method support * move defops * fix test * remove singledispatch case and add test * move definition * cleanup * simplify * cleanup * lint * fix failing test * fix classmethod * __isabstractmethod__ * revert --------- Co-authored-by: Eli <eli@basis.ai> * Try pulling in pyproject.toml from staging-llm to master (#425) --------- Co-authored-by: Jack Feser <jack.feser@gmail.com> Co-authored-by: Tim Cooijmans <cooijmans.tim@gmail.com> * Implement a RetryHandler for LLM module (#428) * Logging retries * Remove redundant assignment * Linting * Adding mypy check and test * SynthesizedFunction for constrained decoding, adapt the tests * Linting * Let LLM generate param names * Pydantic field annotation * Adding validation error example * Linting * Fix minor formatting for type context * Update llm dependencies to include mypy * More comprehensive error message * linting * More comprehensive import * Fix `staging-llm` diff against `master` (#426) * Release v0.2.3 (#374) * Install prettyprinter for term when library is available (#386) * install prettyprinter for term when library is available * lint * move code into types.py * fix pypandoc issue (#397) * Convert evaluate to a singledispatch (#398) * convert evaluate to a singledispatch * lint * add jnp.pi and ArrayTerm.T (#394) * Deprecate defterm (#399) * deprecate defterm * remove defterm case * remove defterm * lint * evaluate distribution arguments * lint * remove interpreter * Revert "remove interpreter" This reverts commit 3044277. * wip * lint * Rework numpyro distribution handling to enable symbolic distributions and handling of distribution methods (#311) * refactor distribution operations * add a test for typeof of distributions * add tests for symbolic dists/arguments * introduce operations for distribution methods * comment * fix tests * work around #310 * replace hack with new hack * tweak repr for _BaseOperation * lint * work around #312 * clean up access to dist ops * wip * wip * add type annotations to get correct term conversion * lint * include distribution arguments as properties * fix distribution calls * try again * fixes * format * Box the output of `__type_rule__` (#387) * box the output of __type_rule__ * fix tests * fix tests * require callers of __type_rule__ to box arguments * fix * move Box out of ops.types * lint * fix test * fix syntactic_eq implementation for jax arrays (#405) * Fix recursion error in sizesof (#406) * fix recursion error in sizesof * format * Allow `_BaseOperation` subclasses to have an overrideable `apply` method (#414) * stash * fixes * initial * wip * lint * ensure each subclass has a fresh operation * wip * wip * lint * wip * wip * lint * refactor class method support * move defops * fix test * remove singledispatch case and add test * move definition * cleanup * simplify * cleanup * lint * fix failing test * fix classmethod * __isabstractmethod__ * revert --------- Co-authored-by: Eli <eli@basis.ai> * Try pulling in pyproject.toml from staging-llm to master (#425) --------- Co-authored-by: Jack Feser <jack.feser@gmail.com> Co-authored-by: Tim Cooijmans <cooijmans.tim@gmail.com> * Logging retries * Make linter happy with the exception_cls * More linting * Revert the changes to synthesis, it belongs to a different PR * Add test for retry handler * More linting * Removing mypy for llm --------- Co-authored-by: eb8680 <eb8680@users.noreply.github.com> Co-authored-by: Jack Feser <jack.feser@gmail.com> Co-authored-by: Tim Cooijmans <cooijmans.tim@gmail.com> * Merge `master` into `staging-llm` again (#443) * Release v0.2.3 (#374) * Install prettyprinter for term when library is available (#386) * install prettyprinter for term when library is available * lint * move code into types.py * fix pypandoc issue (#397) * Convert evaluate to a singledispatch (#398) * convert evaluate to a singledispatch * lint * add jnp.pi and ArrayTerm.T (#394) * Deprecate defterm (#399) * deprecate defterm * remove defterm case * remove defterm * lint * evaluate distribution arguments * lint * remove interpreter * Revert "remove interpreter" This reverts commit 3044277. * wip * lint * Rework numpyro distribution handling to enable symbolic distributions and handling of distribution methods (#311) * refactor distribution operations * add a test for typeof of distributions * add tests for symbolic dists/arguments * introduce operations for distribution methods * comment * fix tests * work around #310 * replace hack with new hack * tweak repr for _BaseOperation * lint * work around #312 * clean up access to dist ops * wip * wip * add type annotations to get correct term conversion * lint * include distribution arguments as properties * fix distribution calls * try again * fixes * format * Box the output of `__type_rule__` (#387) * box the output of __type_rule__ * fix tests * fix tests * require callers of __type_rule__ to box arguments * fix * move Box out of ops.types * lint * fix test * fix syntactic_eq implementation for jax arrays (#405) * Fix recursion error in sizesof (#406) * fix recursion error in sizesof * format * Allow `_BaseOperation` subclasses to have an overrideable `apply` method (#414) * stash * fixes * initial * wip * lint * ensure each subclass has a fresh operation * wip * wip * lint * wip * wip * lint * refactor class method support * move defops * fix test * remove singledispatch case and add test * move definition * cleanup * simplify * cleanup * lint * fix failing test * fix classmethod * __isabstractmethod__ * revert --------- Co-authored-by: Eli <eli@basis.ai> * Try pulling in pyproject.toml from staging-llm to master (#425) * Generate instance-level `Operation`s for bound methods (#351) * generalize __get__ * nits * coverage of methoddescriptor api * methodtype * simplify * simplify * simplify * format * revert * restore * simplify * simplify * retain instance op on term construction * Simplify apply inheritance * assign * put call next to init_subclass * add explanatory comment * Operation.apply -> Operation.__apply__ * add test based on issue description * fix doctest * Fix dataclass @defops and added dataclass metaclass (#439) * fixed dataclass ordering and added metaclass for simplifying construction of dataclass terms * ensure term fields are not being overriden * added decorator and dataclass * updated to make defdata registration automatic * simplified dataclass loop * updated to give property op an appropriate name * added failing tests * fixed failing test * fixed numpyro/pyro/torch interfaces * minor fix + test for deffn kwargs --------- Co-authored-by: Jack Feser <jack.feser@gmail.com> Co-authored-by: Tim Cooijmans <cooijmans.tim@gmail.com> Co-authored-by: Kiran Gopinathan <23038502+kiranandcode@users.noreply.github.com> * Implements a unified `encode`ing/`decode`ing pipeline for `llm` (#442) * implemented unified encoding type * implemented decoding * unified __init__ * added tests for basemodels * s/@property/@functools.cached_property/ * type for encode and decode * removed handling for numbers.Number and explicit tests for complex * fixed is_dataclass checks * updated to check parameter annotations in Tool.of_operation constructor * updated serializer to be more selective in what is an image * reducing number of #type: ignores, and switching to typing.Any * removed comment * dropped dataclass support * dropped tests for dataclass with image * updated dataclass tests to stop assuming pydantic models * test for tool that returns list of images * made serialization a parameter of encodable and thus type-directed * dropped test for tool that returns list of images * dropped registration of encodable types * dropped unused typevar * s/_Encodable/EncodableAs/ * Initial version of Lexical Context Collection - Collecting Tools and Template (#434) * Adding lexical context collection * Allow model input to refer to anything within the lexical context * Different handling between representable object and other types when referred to in Template * More edge case handling * More edge case handling * More edge case handling * Register more instead of _get_source_for_object * Tune down the scope to only collect relevant tools templates * Constructing Tool from Template, allowing higher-order Tempalte * Linting * Fix exception type * Fix exception type to NotHandled * Fix stringinified annotation * Fix tool.define signature * Trim Tool.define * Trip format_value.register * Add warning on no doc case * Removing default for __name__ * More specific lexical context type annotation * Additional tools for pydandic-compatible conversion and recursion depth limiting * Linting * Linting * Linting * Linting * Minor fixes * Merge encoding/decoding * Update a lot of prompt with "do not use any tools" * Update a lot of prompt with "do not use any tools" * More tool limiting prompts * Lint * Lint * Factoring out book and poem tools after multiple failed prompt modification attempts * Revert semantics * Lint * Update name of lexical_context to __context__ * Lint * Minor * More prompt modification * Make sure notebook run * Update `staging-llm` from `master` (#457) * Release v0.2.3 (#374) * Install prettyprinter for term when library is available (#386) * install prettyprinter for term when library is available * lint * move code into types.py * fix pypandoc issue (#397) * Convert evaluate to a singledispatch (#398) * convert evaluate to a singledispatch * lint * add jnp.pi and ArrayTerm.T (#394) * Deprecate defterm (#399) * deprecate defterm * remove defterm case * remove defterm * lint * evaluate distribution arguments * lint * remove interpreter * Revert "remove interpreter" This reverts commit 3044277. * wip * lint * Rework numpyro distribution handling to enable symbolic distributions and handling of distribution methods (#311) * refactor distribution operations * add a test for typeof of distributions * add tests for symbolic dists/arguments * introduce operations for distribution methods * comment * fix tests * work around #310 * replace hack with new hack * tweak repr for _BaseOperation * lint * work around #312 * clean up access to dist ops * wip * wip * add type annotations to get correct term conversion * lint * include distribution arguments as properties * fix distribution calls * try again * fixes * format * Box the output of `__type_rule__` (#387) * box the output of __type_rule__ * fix tests * fix tests * require callers of __type_rule__ to box arguments * fix * move Box out of ops.types * lint * fix test * fix syntactic_eq implementation for jax arrays (#405) * Fix recursion error in sizesof (#406) * fix recursion error in sizesof * format * Allow `_BaseOperation` subclasses to have an overrideable `apply` method (#414) * stash * fixes * initial * wip * lint * ensure each subclass has a fresh operation * wip * wip * lint * wip * wip * lint * refactor class method support * move defops * fix test * remove singledispatch case and add test * move definition * cleanup * simplify * cleanup * lint * fix failing test * fix classmethod * __isabstractmethod__ * revert --------- Co-authored-by: Eli <eli@basis.ai> * Try pulling in pyproject.toml from staging-llm to master (#425) * Generate instance-level `Operation`s for bound methods (#351) * generalize __get__ * nits * coverage of methoddescriptor api * methodtype * simplify * simplify * simplify * format * revert * restore * simplify * simplify * retain instance op on term construction * Simplify apply inheritance * assign * put call next to init_subclass * add explanatory comment * Operation.apply -> Operation.__apply__ * add test based on issue description * fix doctest * Fix dataclass @defops and added dataclass metaclass (#439) * fixed dataclass ordering and added metaclass for simplifying construction of dataclass terms * ensure term fields are not being overriden * added decorator and dataclass * updated to make defdata registration automatic * simplified dataclass loop * updated to give property op an appropriate name * added failing tests * fixed failing test * fixed numpyro/pyro/torch interfaces * minor fix + test for deffn kwargs * Type check and lint example code (#449) * format example code * type check examples * Add beam search example using thermometer continuations (#431) * add beam search example using thermometer continuations * address comments * add docstring * lint * Fix for jax 0.8.2 (#455) * fix for jax 0.8.2 * add more register * format --------- Co-authored-by: Jack Feser <jack.feser@gmail.com> Co-authored-by: Tim Cooijmans <cooijmans.tim@gmail.com> Co-authored-by: Kiran Gopinathan <23038502+kiranandcode@users.noreply.github.com> * Convert `Template` into an operation (#424) * Release v0.2.3 (#374) * stash * Install prettyprinter for term when library is available (#386) * install prettyprinter for term when library is available * lint * move code into types.py * fixes * fix pypandoc issue (#397) * Convert evaluate to a singledispatch (#398) * convert evaluate to a singledispatch * lint * add jnp.pi and ArrayTerm.T (#394) * Deprecate defterm (#399) * deprecate defterm * remove defterm case * remove defterm * lint * evaluate distribution arguments * lint * remove interpreter * Revert "remove interpreter" This reverts commit 3044277. * wip * lint * Rework numpyro distribution handling to enable symbolic distributions and handling of distribution methods (#311) * refactor distribution operations * add a test for typeof of distributions * add tests for symbolic dists/arguments * introduce operations for distribution methods * comment * fix tests * work around #310 * replace hack with new hack * tweak repr for _BaseOperation * lint * work around #312 * clean up access to dist ops * wip * wip * add type annotations to get correct term conversion * lint * include distribution arguments as properties * fix distribution calls * try again * fixes * format * Box the output of `__type_rule__` (#387) * box the output of __type_rule__ * fix tests * fix tests * require callers of __type_rule__ to box arguments * fix * move Box out of ops.types * lint * fix test * fix syntactic_eq implementation for jax arrays (#405) * Fix recursion error in sizesof (#406) * fix recursion error in sizesof * format * initial * wip * lint * ensure each subclass has a fresh operation * wip * wip * lint * wip * wip * lint * refactor class method support * move defops * fix test * remove singledispatch case and add test * move definition * cleanup * simplify * cleanup * lint * fix failing test * wip * Allow `_BaseOperation` subclasses to have an overrideable `apply` method (#414) * stash * fixes * initial * wip * lint * ensure each subclass has a fresh operation * wip * wip * lint * wip * wip * lint * refactor class method support * move defops * fix test * remove singledispatch case and add test * move definition * cleanup * simplify * cleanup * lint * fix failing test * fix classmethod * __isabstractmethod__ * revert --------- Co-authored-by: Eli <eli@basis.ai> * wip * fix define signature * wip * wip * wip * wip * wip * wip * wip * lint * wip * test fix * fix merge error * lint * wip * respond to feedback * respond to feedback * clean up notebook * remove breakpoint * relax typing requirement * improve handling of method templates * lint * remove imprecise check * handle nested classes using qualname * feedback * fix scoping issue and add tests * address feedback * add xfail tests * lint --------- Co-authored-by: Eli <eli@basis.ai> Co-authored-by: Tim Cooijmans <cooijmans.tim@gmail.com> Co-authored-by: eb8680 <eb8680@users.noreply.github.com> * Fail when encoding terms or operations (#474) * raise error when encoding terms or operations * change error type * Implemented record and replay fixtures for LLM calls (#467) * implemented record and replay fixtures for LLM calls * changed defaults to not rebuild fixtures, with dedicated make rule to rebuild * switched pytest to use request fixture instead of env * Remove program synthesis code (#475) * remove program synthesis code * lint * mark xfail * Disables direct recursion on templates by default (#466) * fixed failing test * made templates non-recursive by default + annotation for recursion * added docstring for IsRecursive * call IsRecursive.infer_annotations in tool's init * added check for sig.parameters * drop k-ahead sampler (#479) * Document `Template` and `Tool` (#478) * add documentation to template * Add dataclass import --------- Co-authored-by: eb8680 <eb8680@users.noreply.github.com> * Remove `agent.py` from handler and add docs on how to implement Agents (#464) * added agent example * Fix main check condition in agent.py * removed agent * Fix RetryHandler not to use Template.replace() (#483) * Working notebook * Update notebook * Linting * Remove Template.replace * Linting * Update notebook and linting * Linting * Rename `handlers.llm.providers` to `completions` (#485) * Rename handlers.llm.providers -> completions * lint * Bump version from 0.2.3 to 0.3.0 * Update docs dependencies in pyproject.toml --------- Co-authored-by: Jack Feser <jack.feser@gmail.com> Co-authored-by: Kiran Gopinathan <kiran@basis.ai> Co-authored-by: Kiran Gopinathan <23038502+kiranandcode@users.noreply.github.com> Co-authored-by: Dat Nguyen (Marc) <15943389+datvo06@users.noreply.github.com> Co-authored-by: Tim Cooijmans <cooijmans.tim@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Partially closes #482