feat: LLMImageProcessor and model routing example

[toby-coleman]

Summary

Adds a new component for image processing using multimodal LLMs.

Changes

• New LLMImageProcessor library component to allow images to be passed to multimodal LLMs.
• A new notebook demo for the docs, demonstrating how to perform local and remote image classification.

[gemini-code-assist]

Summary of Changes

Hello @toby-coleman, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the plugboard library by integrating multimodal LLM capabilities for image processing. It introduces the LLMImageProcessor component, allowing users to send images to vision-capable LLMs with flexible input options and structured output. Accompanying this is a new example demonstrating an event-driven routing pipeline where a lightweight local model pre-filters images, sending only relevant ones to a more powerful, remote LLM. This approach optimizes resource usage and processing time for AI applications involving image analysis.

Highlights

• New LLMImageProcessor Component: Introduced a new component designed to process images using vision-capable Large Language Models, supporting both URL and byte inputs for images.
• Model Routing Example: Added a comprehensive notebook demonstration showcasing how to build a hybrid AI pipeline that uses a local model for initial image classification and then routes relevant images to a remote multimodal LLM for further processing, optimizing cost and speed.
• LLM Base Class Refactoring: Refactored the LLMChat component by introducing a shared _LLMBase abstract class to centralize common LLM initialization and structured output handling logic, improving code reusability and maintainability.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[github-actions]

Benchmark comparison for 1b5b0794 (base) vs 0fc85868 (PR)

------------------------------------------------------------------------------------------------------------------ benchmark: 2 tests -----------------------------------------------------------------------------------------------------------------
Name (time in ms)                                                                         Min                 Max                Mean            StdDev              Median               IQR            Outliers     OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_benchmark_process_run (main/.benchmarks/Linux-CPython-3.12-64bit/0001_base)     360.4286 (1.0)      368.3158 (1.0)      363.9692 (1.0)      2.8719 (1.0)      363.5651 (1.0)      3.1897 (1.0)           2;0  2.7475 (1.0)           5           1
test_benchmark_process_run (pr/.benchmarks/Linux-CPython-3.12-64bit/0001_pr)         364.0778 (1.01)     372.8787 (1.01)     367.9266 (1.01)     3.7838 (1.32)     368.0231 (1.01)     6.5994 (2.07)          2;0  2.7179 (0.99)          5           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean

[gemini-code-assist]

Code Review

This pull request introduces a new LLMImageProcessor component for handling multimodal LLM interactions and a comprehensive example notebook demonstrating its use in an event-driven routing pipeline. The code is well-structured, notably with the introduction of a _LLMBase class to abstract common LLM logic, which improves maintainability. The new functionality is also accompanied by good unit tests.

My review focuses on the new example notebook, where I've identified a few areas for improvement regarding comment accuracy, code efficiency, and readability. These are minor changes to enhance the quality of the example for users.

[codecov]

Codecov Report

❌ Patch coverage is 88.70968% with 7 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
plugboard/library/llm.py	88.52%	3 Missing and 4 partials ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

Benchmark comparison for 1b5b0794 (base) vs 9259aee9 (PR)

------------------------------------------------------------------------------------------------------------------ benchmark: 2 tests ------------------------------------------------------------------------------------------------------------------
Name (time in ms)                                                                         Min                 Max                Mean            StdDev              Median                IQR            Outliers     OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_benchmark_process_run (pr/.benchmarks/Linux-CPython-3.12-64bit/0001_pr)         439.1329 (1.0)      447.1280 (1.0)      442.2844 (1.0)      3.0923 (1.0)      442.0599 (1.0)       4.0468 (1.0)           2;0  2.2610 (1.0)           5           1
test_benchmark_process_run (main/.benchmarks/Linux-CPython-3.12-64bit/0001_base)     449.2865 (1.02)     464.5900 (1.04)     457.1064 (1.03)     6.1869 (2.00)     456.2359 (1.03)     10.0533 (2.48)          2;0  2.1877 (0.97)          5           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean

[chrisk314]

Looks good! Nice example of event driven models. Just a couple of small questions/comments.

[github-actions]

Benchmark comparison for 7b81c221 (base) vs 887441bd (PR)

------------------------------------------------------------------------------------------------------------------ benchmark: 2 tests ------------------------------------------------------------------------------------------------------------------
Name (time in ms)                                                                         Min                 Max                Mean            StdDev              Median                IQR            Outliers     OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_benchmark_process_run (pr/.benchmarks/Linux-CPython-3.12-64bit/0001_pr)         441.7126 (1.0)      459.7290 (1.0)      450.6251 (1.0)      7.0982 (1.03)     449.8128 (1.0)      11.0745 (1.30)          2;0  2.2191 (1.0)           5           1
test_benchmark_process_run (main/.benchmarks/Linux-CPython-3.12-64bit/0001_base)     449.3920 (1.02)     468.1059 (1.02)     458.8874 (1.02)     6.8930 (1.0)      460.0865 (1.02)      8.5097 (1.0)           2;0  2.1792 (0.98)          5           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean

[github-actions]

Benchmark comparison for 7b81c221 (base) vs 56022652 (PR)

------------------------------------------------------------------------------------------------------------------ benchmark: 2 tests -----------------------------------------------------------------------------------------------------------------
Name (time in ms)                                                                         Min                 Max                Mean            StdDev              Median               IQR            Outliers     OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_benchmark_process_run (main/.benchmarks/Linux-CPython-3.12-64bit/0001_base)     445.5126 (1.0)      456.1136 (1.0)      451.8491 (1.0)      4.5339 (6.34)     453.9877 (1.0)      7.3112 (6.65)          1;0  2.2131 (1.0)           5           1
test_benchmark_process_run (pr/.benchmarks/Linux-CPython-3.12-64bit/0001_pr)         457.6366 (1.03)     459.4523 (1.01)     458.5336 (1.01)     0.7148 (1.0)      458.6552 (1.01)     1.0992 (1.0)           2;0  2.1809 (0.99)          5           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean