v0.3.1: Image models, TypeScript rewrite, Bedrock support
This change has been in the works for over a month. The most significant change comprises a rewrite of the entire front-end of ChainForge, which is tens of thousands of lines of code, into TypeScript. More details below.
Support for image models (with Dall-E OpenAI models being the first)
![Screen Shot 2024-03-30 at 7 03 45 PM](https://private-user-images.githubusercontent.com/5251713/318233272-25d3d977-d3b4-4172-ac81-3094dd842e69.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM5OTIyNTksIm5iZiI6MTcyMzk5MTk1OSwicGF0aCI6Ii81MjUxNzEzLzMxODIzMzI3Mi0yNWQzZDk3Ny1kM2I0LTQxNzItYWM4MS0zMDk0ZGQ4NDJlNjkucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgxOCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MThUMTQzOTE5WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZGFkY2U1M2JhNmRiOGIzOGJiNmFhZmVhZjQ2OTdkNWM1NTQxNGQyZTI2ZDIzMTcyOTc2OWI2NmM3NGZiNmJmMSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.BSRGCnmiV1edQnEvzVlttEm0SD9ijqBIx8b-DqtfFeg)
Images are compressed by default using compressorjs
, with no visible impact on quality and average compression around 60% of the original. Users can turn off compression in the Advanced tab of the Settings window. It is recommended to keep it on, however.
Custom Providers for Image Models
Your custom providers can return image data instead of text. From your provider, you would return a JSON dict in the format: { t: "img", d: <base64_str_png_encoded_image> }
. (Only put the base64 data, not the "data...image:png," etc metadata before it.)
Note that we don't yet support:
- Exporting images into cells of an Excel sheet (Export to Excel will be disabled if it detects an image)
Warning
Be warned that images eat up browser storage space fast and will quickly disable autosaving.
Rewrite of the front-end into TypeScript
The entire front-end has been converted to tsx
files, with the appropriate typing. Other nice refactorings were made along the way.
Among identifying bugs or never reached code, we have worked to simplify and standardize the ways LLMResponses are stored on the backend and front-end. It is not perfect, but now LLMResponse
and LLMResponseData
make the format of stored responses transparent to developers.
This change makes it easier for developers to extend ChainForge with confidence, chiefly because TypeScript flags side-effects of changes to core code. For instance, TypeScript enabled us to add image models in only 2 hours, since all that was required was changing the type LLMResponseData
from a string to a string | object-with-image-data. This change would not have been easy to perform with confidence without the visibility TypeScript provides into the downstream effects of changing a core datatype. In the future, it will help us add support for multi-modal vision models.
Custom Right-click Actions on Nodes
Right-clicking on a node can now present more options.
- TextFields can be converted into Items Nodes
- Items Nodes can be converted into TextFields Nodes
- Prompt and Chat Turn node LLM response cache can be cleared:
![Screen Shot 2024-03-30 at 7 41 37 PM](https://private-user-images.githubusercontent.com/5251713/318233518-5dae6ef0-c043-4d7f-9d00-66b5a67ebd9c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM5OTIyNTksIm5iZiI6MTcyMzk5MTk1OSwicGF0aCI6Ii81MjUxNzEzLzMxODIzMzUxOC01ZGFlNmVmMC1jMDQzLTRkN2YtOWQwMC02NmI1YTY3ZWJkOWMucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgxOCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MThUMTQzOTE5WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NDFkYjU4MzhjMGI1Mzk5Njk5Mjg5ZjdlOTcwZjI4MWI2NWNkODYyNGQxZDcwNzI5ZGUzMmRkN2FhNmRjYzgyYSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.PiIvyn0bI6z7G3XYRyc4YVu45wpsyDHaTEO3KnbtCmE)
Amazon Bedrock Support
Thanks to @massi-ang , Amazon Bedrock hosted models have been added to ChainForge. We've just added these endpoints, and I wasn't able to test them directly (I don't have access), so if you encounter problems please open an Issue and poke @massi-ang to let him know.
Nested Add Model Menu on Prompt nodes
![Screen Shot 2024-03-30 at 10 47 06 PM](https://private-user-images.githubusercontent.com/5251713/318238046-5987f4c6-8684-4ef0-a533-5300b38472bd.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM5OTIyNTksIm5iZiI6MTcyMzk5MTk1OSwicGF0aCI6Ii81MjUxNzEzLzMxODIzODA0Ni01OTg3ZjRjNi04Njg0LTRlZjAtYTUzMy01MzAwYjM4NDcyYmQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgxOCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MThUMTQzOTE5WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YjVmYWQxNDIyYTRlZmM3NzdhYzU3YzhlYzQ5Y2ZmNzQxOWFmZWFmYTg3NjhjNTVlZDljYjQwNTZiZTI1NGM3OSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.ynIlvDdU_maLnenVXGWiSrAayyWvv2zUrl3goTU5AhU)
Thanks also to @massi-ang , clicking Add+ to add a model now brings up a nested menu. The list is still limited (use the Settings to access more specific models), but we were facing a growing problem as the number of providers increased.
Better Rate Limiting
Rate limiting has been improved to use Bottleneck
. Rate limiting is now performed on a rolling timing window rather than the naive "block and wait" approach we used before. Note that this doesn't take into account what "tier" of access you have for OpenAI and Anthropic models, as there's no way for us to know that, so it's based on Tier 2 access. If you encounter a rate limit just re-run the node.
Problems? Let us know!
This change is comprehensive, and while I have tested it extensively, it is possible I have missed something. If you encounter an error, please open an Issue.