vision plugin

Upstream: this is a fork of opencode-vision (MIT) by Nicolás Ríos Herrera. It has been meaningfully changed (in-process plugin, NIM provider chain, config-in-opencode, dual-describe) but the original work is theirs. This code was generated by an AI assistant; verify before trusting.

Adds vision_describe, vision_ocr, vision_analyze to opencode so a text-only model can still read images. Runs in-process: no MCP server, no subprocess, no Python.

Install

Zero config — copy the bundled file into your plugin dir and restart:

cp opencode-vision-plugin.js ~/.config/opencode/plugin/opencode-vision-plugin.js

Keys come from your environment (NVIDIA_API_KEY, GEMINI_API_KEY / GOOGLE_API_KEY). No config block needed.

Optional config block — if you want to override defaults (key auth, endpoints, timeouts, model-of-choice), pin the npm package and drop a block into ~/.config/opencode/opencode.jsonc:

{
  "$schema": "https://opencode.ai/config.json",
  "plugin": [
    ["opencode-vision-plugin", {
      "provider": "chain",
      "chain": ["gemini", "nim"],
      "models": {
        "gemini": "gemini-3.6-flash",
        "nim": "meta/llama-3.2-90b-vision-instruct"
      },
      "keys": {
        "gemini": "{env:GEMINI_API_KEY}",
        "nim": "{env:NVIDIA_API_KEY}"
      },
      "baseUrls": {
        "gemini": "https://generativelanguage.googleapis.com/v1beta",
        "nim": "https://integrate.api.nvidia.com/v1"
      },
      "timeout_ms": 60000
    }]
  ]
}

Every string supports {env:VAR}, resolved at load time.

What the config actually controls

The tool itself already knows every model and keeps sensible defaults, so the config block exists to override defaults, not to enable the tools.

Key	Default	What it does
provider	chain	Which provider to hit first
chain	["gemini","nim"]	Fallthrough order — if the first fails, try the next
models.gemini	gemini-3.6-flash	Model used when you call without a model arg
models.nim	meta/llama-3.2-90b-vision-instruct	Model used when you call without a model arg
keys.*	env vars	API keys (endpoints reject unauthenticated calls)
baseUrls.*	standard endpoints	Where each provider lives
timeout_ms	60000	Global budget; a hanging provider gives up so the chain can fall through

You can still pick any model at call time via the model argument — the config just sets the backstop default. If you're happy with the defaults, skip the block entirely.

Behavior

• Default chain tries gemini then nim — if Gemini's daily quota is out, NIM keeps you working.
• vision_describe with no model runs two NIM VLMs and returns each labelled section (structured + detail) — this is the recommended default call. The tool description steers agents to it and against per-call model-hopping.
• image_path accepts a local path, an https URL, or a data: URL.

Available models

• meta/llama-3.2-90b-vision-instruct — structured
• nvidia/nemotron-nano-12b-v2-vl — detail
• meta/llama-3.2-11b-vision-instruct — light
• nvidia/llama-3.1-nemotron-nano-vl-8b-v1 — concise
• gemini-3.6-flash

Requirements

Node/bun global fetch + node fs/path. Needs NVIDIA_API_KEY and/or GEMINI_API_KEY.

Restart

Config is read once at startup. After changing the config or the plugin file, quit and restart opencode. If you installed via npm, also clear ~/.cache/opencode/packages/opencode-vision-plugin — opencode won't refetch.