v0.9.14

🚀 Added

LMMs (GPT-4V and CogVLM) 🤝 workflows

Now, with Roboflow workflows LMMs integration is made easy 💪 . Just look at the demo! 🤯

lmms_in_workflows.mp4

As always, we encourage you to visit workflows docs 📖 and examples.

This is how to create a multi-functional app with workflows and LMMs:

inference server start

from inference_sdk import InferenceHTTPClient

LOCAL_CLIENT = InferenceHTTPClient(
    api_url="http://127.0.0.1:9001", 
    api_key=ROBOFLOW_API_KEY,
)
FLEXIBLE_SPECIFICATION = {
    "version": "1.0",
    "inputs": [
        { "type": "InferenceImage", "name": "image" },
        { "type": "InferenceParameter", "name": "open_ai_key" },
        { "type": "InferenceParameter", "name": "lmm_type" },
        { "type": "InferenceParameter", "name": "prompt" },
        { "type": "InferenceParameter", "name": "expected_output" },
    ],
    "steps": [     
        {
            "type": "LMM",
            "name": "step_1",
            "image": "$inputs.image",
            "lmm_type": "$inputs.lmm_type",
            "prompt": "$inputs.prompt",
            "json_output": "$inputs.expected_output",
            "remote_api_key": "$inputs.open_ai_key",
        },
    ],
    "outputs": [
        { "type": "JsonField", "name": "structured_output", "selector": "$steps.step_1.structured_output" },
        { "type": "JsonField", "name": "llm_output", "selector": "$steps.step_1.*" },
    ]   
}

response_gpt = LOCAL_CLIENT.infer_from_workflow(
    specification=FLEXIBLE_SPECIFICATION,
    images={
        "image": cars_image,
    },
    parameters={
        "open_ai_key": OPEN_AI_KEY,
        "lmm_type": "gpt_4v",
        "prompt": "You are supposed to act as object counting expert. Please provide number of **CARS** visible in the image",
        "expected_output": {
            "objects_count": "Integer value with number of objects",
        }
    }
)

🌱 Changed

• developer friendly theming aka dark theme by @onuralpszr in #270 (thanks for contribution 🥇 )
• YoloWorld docs @capjamesg in #276
• @ryanjball made their first contribution in #271 with his cookbook for RGB anomaly detection

🔨 Fixed

• turn off instant page for load to cookbook page properly by @onuralpszr in #275 (thanks for contribution 🥇 )
• bug in workflows that made cropping in multi-detection set-up

Full Changelog: v0.9.13...v0.9.14