Images and Assets EN

Images & Multimodal Assets

English | 简体中文

---

The TypeSafe AI Go SDK provides full support for multimodal evaluations. You can include images directly in state, instructions, or criteria, evaluate visual attributes with Noul, Choice, and Score, or locate objects precisely with Box bounding box questions.

---

Quickstart

package main

import (
	"context"
	"fmt"
	"log"

	typesafe "github.com/guchengod/typesafe-sdk-go"
)

func main() {
	// 1. Initialize client with images enabled
	client, err := typesafe.NewClient(typesafe.WithImages(true))
	if err != nil {
		log.Fatal(err)
	}
	defer client.Close()

	// 2. Load an image from disk
	photo, err := typesafe.NewImageFromFile("kitchen.jpg")
	if err != nil {
		log.Fatal(err)
	}

	// 3. Embed the image directly into state
	resp, err := client.SystemOne(
		context.Background(),
		map[string]any{
			"room":  "kitchen",
			"photo": photo, // SDK auto-collects and serializes to a reference ID
		},
		typesafe.Questions{
			"has_cup": typesafe.NewNoul("Is there a coffee cup in `photo`?"),
			"cup_box": typesafe.NewBox("the coffee cup in `photo`"),
		},
		typesafe.WithModel(typesafe.ModelJevPreview), // recommend jev-preview for multimodal & box
	)
	if err != nil {
		log.Fatal(err)
	}

	// 4. Inspect answers
	if resp.Nouls()["has_cup"].Noul > 0.8 {
		box := resp.Boxes()["cup_box"].Box
		fmt.Printf("Coffee cup detected! Box: Top=%.2f, Bottom=%.2f, Left=%.2f, Right=%.2f\n",
			box.Top, box.Bottom, box.Left, box.Right)
	}
}

---

Image Structure & Constructors

The SDK provides typesafe.Image to wrap image bytes and format specifications:

type Image struct {
	Ref    string // Unique reference identifier (e.g., "img-7f8e...")
	Format string // Normalized format ("png" or "jpg")
	Data   []byte // Raw binary bytes
	Base64 string // Standard Base64 encoded string
}

1. From file (NewImageFromFile)

Reads the file from disk, infers format from extension, and validates size:

img, err := typesafe.NewImageFromFile("document.png")

2. From raw bytes (NewImage)

Generates a cryptographically random unique Ref:

img, err := typesafe.NewImage(rawBytes, "jpeg")

3. With explicit reference (NewImageWithRef)

Assign a custom reference identifier (e.g., "front_cover", "id_card"):

img, err := typesafe.NewImageWithRef("id_card", rawBytes, "png")

4. From Base64 string (NewImageFromBase64)

img, err := typesafe.NewImageFromBase64("avatar", b64String, "png")

---

Automatic Asset Collection & Reference

In the wire protocol, image data is transmitted in the root assets map, while state refers to images using string references (e.g. "img-01a2...").

The Go SDK automates this entirely:

1. Direct embedding: Place *typesafe.Image instances anywhere inside a struct, map, slice, or nested structure in state.
2. Transparent serialization: Image implements json.Marshaler, encoding to its reference ID string.
3. Auto-collection: client.SystemOne walks the state recursively, extracts all embedded *Image instances, and mounts them to the request's assets payload.

Explicit Assets (WithAssets)

If image assets are stored outside the state object, attach them explicitly using the call option:

assets := typesafe.CollectAssets(img1, img2)

resp, err := client.SystemOne(ctx, state, questions,
	typesafe.WithAssets(assets),
)

WithAssets accepts typesafe.Assets, map[string]typesafe.Asset, or map[string]any.

---

Danger Zone Header

Because multimodal processing requires specific server capabilities and quota handling, TypeSafe API requires explicit opt-in:

• Client level:

  client, err := typesafe.NewClient(typesafe.WithImages(true))

• Per-call level:

  resp, err := client.SystemOne(ctx, state, questions,
      typesafe.WithCallImages(true),
  )

This option sends x-typesafe-images: true (server also accepts TypeSafe-Danger-Zone-Images: true). If a request includes image assets or box questions without this header, the server rejects it.

---

Specifications & Limits

Constraint	Limit	Behavior when exceeded
Supported image formats	png, jpeg / jpg	Other formats (e.g. gif, webp) fail client validation with *SDKError
Single image size limit	16 MiB (MaxImageSizeLimit)	Rejected in NewImage or by server with HTTP 413
Total request body limit	32 MiB (MaxRequestBodyLimit)	Rejected by server with HTTP 413 *PayloadTooLargeError
Recommended model	jev-preview (ModelJevPreview)	Optimized for multimodal reasoning and bounding box localization