Vision Language Models

Every model in rAPIdtools, hosted or local, is built through one factory (rapidtools.models.load) and exposes the same run_inference call, so pipeline code does not change when you switch providers.

Loading a model

from rapidtools.models import catalog, list_providers, load

list_providers()              # all registry keys
list_providers('local')       # 'api', 'local' or 'segmentation'
catalog('gemini')             # curated model IDs, no network
model = load('gemini', api_key='AIza...', model_id='gemini-3.8-flash')
model = load('gemma4', model_id='google/gemma-4-E2B-it')
model = load('hf', model_id='Qwen/Qwen3.5-4B', load_in_4bit=True)

Key	Runs	Default model	API key read from
gemini	API	gemini-3.8-flash	GOOGLE_API_KEY, GEMINI_API_KEY
claude	API	claude-opus-5	ANTHROPIC_API_KEY
openai	API	gpt-5.5	OPENAI_API_KEY
muse_spark	API	muse-spark-1.3	MODEL_API_KEY, META_API_KEY
qwen	API	qwen3.8-max	DASHSCOPE_API_KEY, QWEN_API_KEY
gemma4	local	google/gemma-4-E2B-it	Hugging Face login for gated weights
llama	local	meta-llama/Llama-3.2-11B-Vision-Instruct	Hugging Face login
muse_glimmer	local	meta-models/Muse-Glimmer-30B	Hugging Face login
qwen_vl	local	Qwen/Qwen3.5-4B	Hugging Face login
hf	local	any repo with a chat template	Hugging Face login
sam3	local	facebook/sam3	Hugging Face login; segmentation, not text

api_key may be the key itself or a path to a file holding it; when omitted, the environment variables above are tried. Every wrapper offers list_known_models(), and hosted ones list_available_models(api_key) for a live listing. The curated catalogue and hardware notes are on Supported Models; see Installation for rapidtools.login().

Wrapper options

Kind	Constructor arguments
Hosted (gemini, claude, openai, muse_spark, qwen)	api_key, model_id, max_workers, max_retries, system_instruction, temperature, max_tokens, timeout (60 s; raise for large images)
Local (gemma4, llama)	model_id, device ('auto', 'cuda', 'cpu'), temperature, max_tokens
Local (hf, qwen_vl, muse_glimmer)	the above plus load_in_4bit; hf also trust_remote_code

Generation options

GenerationConfig carries per-call options and maps them to each provider's native flags. Precedence is wrapper defaults, then GenerationConfig, then explicit keyword overrides.

Field	Default	Effect
temperature	None (model default)	Use 0.0 for repeatable classifications.
max_tokens	None	Cap on generated tokens.
json_mode	False	Native JSON flag where supported, prompt instruction otherwise.
system_instruction	None	System prompt for this call.

GenerationConfig(temperature=0.4).merged(max_tokens=512) returns a copy with overrides applied.

Analyzing a collection

AssetAnalyzer is the ANALYZE step: it sends each asset's images and the prompt to the model and writes the answer to the asset's attributes.

from rapidtools import AssetAnalyzer, Pipeline
from rapidtools.models import GenerationConfig, load

analyzer = AssetAnalyzer(
    load('claude', api_key='sk-ant-...'),
    prompt='prompts/damage.txt',                 # text, or a path ending in .txt/.md/.prompt/.text
    max_images_per_asset=2,
    image_filter=lambda img: 'aerial' in img.id,
    generation=GenerationConfig(json_mode=True, temperature=0.0),
)
buildings = Pipeline([analyzer]).run(buildings)
buildings['bldg_01'].attributes['claude_damage_level']

Parameter	Default	Notes
model	required	Any wrapper from load().
prompt	''	Text or a prompt file. download_dataset ships 'aerial_chs_prompts', 'street_chs_prompts' and 'street_recovery_prompts'.
attribute_prefix	provider key	'claude_damage_level'; set your own to compare runs.
max_workers	5 for APIs, 1 for local	Local inference is never threaded.
batch_size	4	Assets per batch for local models.
max_images_per_asset	None	Cap on images sent per asset.
image_filter	None	Predicate on ImageAsset, e.g. by img.id or img.path.
rate_limit	RateLimitPolicy() for APIs	See below.
generation	None	A GenerationConfig.
mode	inferred	Force 'api' or 'local'.
cancel_event	None	Checked between assets and batches.

What gets written: if the answer parses as a JSON object (fenced or bare), each key becomes <prefix>_<key>; otherwise Key: Value lines are parsed, and failing that the whole text is stored as <prefix>_analysis_raw. Every analyzed asset also gets ai_model_used and images_analyzed_count. The analyzer's prompt, batch_size, max_images_per_asset and image_filter can be reassigned between runs to reuse a loaded local model.

Provider-specific classes such as GeminiAssetAnalyzer and Gemma4AssetAnalyzer still exist but are deprecated; use AssetAnalyzer(load(...)).

Rate limits, retries and failures

• RateLimitPolicy(cooldown_seconds=30.0, max_asset_retries=2) is the default for hosted models. After a transient failure every worker pauses for cooldown_seconds times the number of consecutive failures; the first success resets it. Assets that failed get extra passes once the first pass is done, so a burst of 429s does not leave holes. RateLimitPolicy(0) disables pausing.
• Hosted wrappers share rapidtools.models.api_base.api_session(), which re-sends a POST only after 429, 503 or 529, statuses that guarantee the provider did not process (or bill) the request. A 500, 502 or 504 is reported instead of repeated.
• run_inference returns None for transient failures (timeouts, rate limits). Provider rejections (refusals, safety blocks, 4xx) come back as a ModelOutput with error set, retryable=False and failed == True; the analyzer records them as failed without pausing the workers or retrying.
• Each hosted wrapper takes timeout= for slow responses and max_retries= for the session.

Single images

Outside a pipeline, call the model directly. The result is a ModelOutput with text and, for SAM 3, masks and bounding_boxes.

from rapidtools.models import load

model = load('gemma4', model_id='google/gemma-4-E2B-it')
output = model.run_inference('scene.jpg', 'Describe the damage in this scene.')
if output is not None and not output.failed:
    print(output.text)

sam = load('sam3')
seg = sam.run_inference('crop.jpg', 'building roof')
seg.masks, seg.bounding_boxes          # (N, H, W) array and [x0, y0, x1, y1] boxes

image_inputs may be one path or URL or a list; prompt may be text or a prompt file. ModelOutput also exposes has_text, has_masks, has_bounding_boxes and the provider's raw_response.

Writing prompts

• Ask for JSON with named keys and allowed values; the keys become attribute names.
• Refer to the outline: "the building marked with cyan corner brackets" (see outline options in Aerial Imagery Workflows).
• Keep one task per prompt and run the analyzer twice (the recovery example reuses the same model for damage, then recovery).
• The Graphical Interface has a prompt builder that drafts and reviews prompts with any backend.

Common pitfalls

• Attributes missing for some assets. Check images_analyzed_count; assets without downloaded images are skipped. Use filter_empty() before export.
• Answer stored as one text blob. The model did not return a JSON object; set json_mode=True and ask for keys explicitly.
• Every worker keeps pausing. That is the cooldown after repeated 429s; lower max_workers or raise cooldown_seconds.
• Refusals reappear on retry. They do not: a retryable=False output is final. Rephrase the prompt.
• Two local models at once. Release one before loading the next, or run SAM 3 steps before the analyzer; Pipeline already orders stages.
• Gated weights fail to download. Call rapidtools.login() or set a Hugging Face token first.
• A file name was sent as the prompt. Prompt files must exist and end in .txt, .md, .prompt or .text; otherwise the string itself is sent with a warning.

See also Supported Models, Street-Level Imagery Workflows and Troubleshooting and FAQ.