Repository navigation
Replies: 1 comment
|
good catch on all of these. the reasoning_effort leak is the subtle one since some providers silently ignore it and you only find out when outputs change unexpectedly. for that one, the cleanest fix is a small normalize_extra_kwargs helper in the provider adapter that strips or remaps params the target provider doesn't support. the model id defaults are easy wins, just thread SMART_LLM/FAST_LLM through the benchmark argparse defaults. the .env.example gap is real too, a commented anthropic/ollama block would cut the "how do i use a different provider" questions in half. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hi! I ran a static provider-portability scan over gpt-researcher (main, 60 of 493 candidate files) to see how easy it would be to swap LLM vendors. The good news is that the project already routes most calls through an abstraction layer, so the overall picture is quite portable. Below are the spots where provider specifics still leak through. Each one is a small, local change. Sharing in case it helps the multi-provider effort.
1. Hardcoded model ids (6)
backend/report_type/deep_research/example.pyL18-19:"gpt-4o"/"o3-mini"as module constants.deep_agents/{breadth,hybrid,recency}_benchmark.pyandreport_benchmark.py:--grader-model/--judge-modeldefault to"gpt-5.4".SMART_LLM/FAST_LLM, which the project already uses) so the benchmarks run as-is with a non-OpenAI provider.2. Provider-only params (2)
example.pyL82 and L148 passreasoning_effort=.... That is an OpenAI reasoning-model param. Other providers either reject it or ignore it silently.3. Provider-named secrets (8)
OPENAI_API_KEY/GOOGLE_API_KEYindocker-compose.yml(L7, L13, L51),.env.exampleL1,server_utils.pyL282/L284 and.github/workflows/docker-build.ymlL17..env.exampleand docker-compose present OpenAI as the only path. A commented block for Anthropic / Gemini / Ollama keys would make the multi-provider support easier to discover.4. Direct
langchain_openai.ChatOpenAIimports (4)deep_agents/*_benchmark.pyfiles importChatOpenAIdirectly instead of going through the project's LLM provider factory. Routing them through the factory would let the benchmarks use any configured provider.5. Tool-call wire format (1)
backend/chat/chat.pyL36 builds an OpenAI-style{"type": "function", ...}tool schema. If chat is meant to work with Anthropic or Gemini tool use, normalizing that shape in the adapter would avoid a provider-specific branch.This is a heuristic static scan, so please treat it as a checklist to review, not a verdict. I hand-checked all 21 hits. One of them (
tests.ymlL140) is a comment, so call it borderline. Happy to clarify any item.(Disclosure: the scan comes from a small CLI I sell. The full report and rules are here if useful: https://renevibe76.gumroad.com/l/vdmrtn?ref=gh-discussion)
All reactions