feat(inpainting): add inpainting endpoint, wire ImageGenerationFunc and return generated image URL #7328

gmaOCR · 2025-11-21T16:09:35Z

Description

This patch implements an OpenAI-compatible inpainting endpoint and wires image generation into the LocalAI backend. Changes are limited to the LocalAI Go codebase.

Adds InpaintingEndpoint handling POST /v1/images/inpainting with multipart/form-data support for image and mask.
Endpoint writes a JSON payload with base64 image/mask and prepares a temporary output PNG path, then calls backend.ImageGenerationFunc(...) to perform generation.
Introduces and uses the test-friendly indirection ImageGenerationFunc in image.go so the generation logic can be stubbed in tests.
Adds unit tests for the inpainting handler (inpainting_test.go) covering missing-files and the happy path (with backend.ImageGenerationFunc stubbed).
Registers the inpainting routes and a default model name middleware in openai.go.
Updates Swagger (swagger.yaml) and adds a Swaggo-style docstring on the new endpoint for proper OpenAPI generation.

Notes for Reviewers

inpainting.go (new endpoint)
inpainting_test.go (new tests)
image.go (ImageGenerationFunc indirection)
mcp.go (minor related adjustments)
openai.go (routes registration)
swagger.yaml (OpenAPI updates)

Signed commits

Yes, I signed my commits.

netlify · 2025-11-21T16:09:40Z

✅ Deploy Preview for localai ready!

Name	Link
🔨 Latest commit	`d45913b`
🔍 Latest deploy log	https://app.netlify.com/projects/localai/deploys/6923029be438ca00087308b8
😎 Deploy Preview	https://deploy-preview-7328--localai.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

core/http/endpoints/openai/inpainting.go

+		// write JSON
+		enc := json.NewEncoder(jf)
+		if err := enc.Encode(jsonFile); err != nil {
+			jf.Close()


core/http/endpoints/openai/inpainting.go

+		enc := json.NewEncoder(jf)
+		if err := enc.Encode(jsonFile); err != nil {
+			jf.Close()
+			os.Remove(jf.Name())


core/http/endpoints/openai/inpainting.go

+			os.Remove(jf.Name())
+			return err
+		}
+		jf.Close()


core/http/endpoints/openai/inpainting.go

+		jf.Close()
+		// rename to desired name
+		if err := os.Rename(jf.Name(), jsonPath); err != nil {
+			os.Remove(jf.Name())


core/http/endpoints/openai/inpainting.go

+		// prepare dst
+		outTmp, err := os.CreateTemp(tmpDir, "out_")
+		if err != nil {
+			os.Remove(jsonPath)


core/http/endpoints/openai/inpainting.go

+			os.Remove(jsonPath)
+			return err
+		}
+		outTmp.Close()


core/http/endpoints/openai/inpainting.go

+		outTmp.Close()
+		dst := outTmp.Name() + ".png"
+		if err := os.Rename(outTmp.Name(), dst); err != nil {
+			os.Remove(jsonPath)


core/http/endpoints/openai/inpainting.go

+		// Note: ImageGenerationFunc will call into the loaded model's GenerateImage which expects src JSON
+		fn, err := backend.ImageGenerationFunc(height, width, 0, steps, 0, prompt, "", jsonPath, dst, ml, *cfg, appConfig, nil)
+		if err != nil {
+			os.Remove(jsonPath)


core/http/endpoints/openai/inpainting.go

+
+		// Execute generation function (blocking)
+		if err := fn(); err != nil {
+			os.Remove(jsonPath)


core/http/endpoints/openai/inpainting.go

+		// Execute generation function (blocking)
+		if err := fn(); err != nil {
+			os.Remove(jsonPath)
+			os.Remove(dst)


mudler · 2025-11-21T16:28:43Z

core/http/endpoints/openai/inpainting.go

+
+		// Call backend image generation via indirection so tests can stub it
+		// Note: ImageGenerationFunc will call into the loaded model's GenerateImage which expects src JSON
+		fn, err := backend.ImageGenerationFunc(height, width, 0, steps, 0, prompt, "", jsonPath, dst, ml, *cfg, appConfig, nil)


ImageGeneration actually take a list of refimages files. We could wire these up to the image and mask, so it becomes open to the backend to implement this correctly.

Hi thanks for the pointer, good catch.

I wired the uploaded image and mask as reference image files and pass them to the backend via the refImages parameter so backends that expect file references can use them directly. For compatibility I kept the existing JSON src (base64) as well, and added robust best‑effort cleanup of all temp files. The inpainting endpoint now does:

write image and mask to temp files (origRef, maskRef)

write the src JSON (base64) as before (for backward compatibility)

call backend.ImageGenerationFunc(..., refImages) with both paths

- Created gallery/dreamshaper-8-inpainting.yaml with diffusers backend - Configured StableDiffusionInpaintPipeline for inpainting tasks - Added gallery index entry with model file mapping - Maps 'dreamshaper-8-inpainting' API name to DreamShaper_8_pruned.safetensors file - Fixes CI test failures: model file resolution now works via gallery system - Addresses maintainer feedback on PR mudler#7328 Signed-off-by: Greg <marianigregory@pm.me>

gmaOCR · 2025-11-23T08:31:38Z

@mudler Can i do something for [TIMEDOUT] in [It] - /Users/runner/work/LocalAI/LocalAI/core/http/app_test.go:994 @ 11/22/25 22:28:28.921 in the apple CI ? Or the problem is on CI side ?

Copilot

Pull request overview

This PR adds an OpenAI-compatible inpainting endpoint (/v1/images/inpainting) that accepts multipart form data containing an image, mask, and prompt to generate inpainted images. The implementation extends LocalAI's image generation capabilities by introducing a test-friendly indirection pattern for backend image generation.

Key Changes:

New inpainting endpoint with multipart/form-data support for image and mask uploads
Introduces ImageGenerationFunc variable in backend/image.go to enable test stubbing
Comprehensive test coverage for both error cases and successful image generation
Enhanced MCP logging to include model names for better debugging

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`core/http/endpoints/openai/inpainting.go`	New inpainting handler that processes multipart uploads, creates base64-encoded JSON payloads, manages temporary files with cleanup on error, and calls backend image generation
`core/http/endpoints/openai/inpainting_test.go`	Unit tests covering missing file validation and happy path with stubbed backend generation function
`core/http/routes/openai.go`	Registers inpainting routes (`/v1/images/inpainting`, `/images/inpainting`) with image middleware for model selection and configuration
`core/backend/image.go`	Adds `ImageGenerationFunc` variable pointing to `ImageGeneration` function to enable test stubbing
`core/http/endpoints/openai/mcp.go`	Enhances logging by adding model name (`config.Name`) to tool call and result log messages
`swagger/swagger.yaml`	Documents the new inpainting endpoint with all parameters (model, prompt, steps, image, mask) and response schema

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-23T12:34:58Z

core/http/endpoints/openai/inpainting.go

+		b64Mask := base64.StdEncoding.EncodeToString(maskBytes)
+
+		// get model config from context (middleware set it)
+		cfg, ok := c.Get("MODEL_CONFIG").(*config.ModelConfig)


Use the constant middleware.CONTEXT_LOCALS_KEY_MODEL_CONFIG instead of the hardcoded string "MODEL_CONFIG" for consistency with other endpoints (e.g., image.go line 77, chat.go line 193). This follows the codebase convention and makes refactoring safer.

Suggested change

cfg, ok := c.Get("MODEL_CONFIG").(*config.ModelConfig)

cfg, ok := c.Get(middleware.CONTEXT_LOCALS_KEY_MODEL_CONFIG).(*config.ModelConfig)

Copilot · 2025-11-23T12:34:59Z

core/http/endpoints/openai/inpainting.go

+		jsonName := fmt.Sprintf("inpaint_%s.json", id)
+		jsonPath := filepath.Join(tmpDir, jsonName)


The variable jsonName is declared but never used. The code computes it at line 116 but then creates a temp file with os.CreateTemp (line 122) and renames that to jsonPath (line 211), making jsonName unnecessary. Consider removing this unused variable to improve code clarity.

Suggested change

jsonName := fmt.Sprintf("inpaint_%s.json", id)

jsonPath := filepath.Join(tmpDir, jsonName)

jsonPath := filepath.Join(tmpDir, fmt.Sprintf("inpaint_%s.json", id))

Copilot · 2025-11-23T12:34:59Z

core/http/endpoints/openai/inpainting_test.go

+	c := e.NewContext(reqBuf, rec)
+
+	// set a minimal model config in context as handler expects
+	c.Set("MODEL_CONFIG", &config.ModelConfig{Backend: "diffusers"})


Use the constant middleware.CONTEXT_LOCALS_KEY_MODEL_CONFIG instead of the hardcoded string "MODEL_CONFIG" for consistency with the rest of the codebase.

Signed-off-by: Greg <marianigregory@pm.me>

gmaOCR force-pushed the fix/inpainting-backend-response-and-cuda branch from 1a714e7 to 68bfe00 Compare November 21, 2025 16:15

github-advanced-security bot found potential problems Nov 21, 2025

View reviewed changes

gmaOCR force-pushed the fix/inpainting-backend-response-and-cuda branch from 68bfe00 to 1d33fbf Compare November 21, 2025 16:19

github-actions bot added the kind/documentation Improvements or additions to documentation label Nov 21, 2025

gmaOCR force-pushed the fix/inpainting-backend-response-and-cuda branch from 1d33fbf to 5884e9c Compare November 21, 2025 16:20

mudler reviewed Nov 21, 2025

View reviewed changes

mudler added enhancements and removed kind/documentation Improvements or additions to documentation labels Nov 21, 2025

gmaOCR force-pushed the fix/inpainting-backend-response-and-cuda branch from 7e414a1 to 8aa0b43 Compare November 21, 2025 17:25

github-actions bot added the area/ai-model label Nov 22, 2025

gmaOCR force-pushed the fix/inpainting-backend-response-and-cuda branch 2 times, most recently from 80af855 to 8bfe58a Compare November 22, 2025 13:40

github-actions bot added the kind/documentation Improvements or additions to documentation label Nov 22, 2025

gmaOCR force-pushed the fix/inpainting-backend-response-and-cuda branch 10 times, most recently from 79941c8 to 874b052 Compare November 22, 2025 20:43

gmaOCR marked this pull request as ready for review November 23, 2025 12:30

Copilot AI review requested due to automatic review settings November 23, 2025 12:30

Copilot started reviewing on behalf of gmaOCR November 23, 2025 12:30 View session

Copilot finished reviewing on behalf of gmaOCR November 23, 2025 12:34

Copilot AI reviewed Nov 23, 2025

View reviewed changes

gmaOCR force-pushed the fix/inpainting-backend-response-and-cuda branch 2 times, most recently from e356139 to dd5ffe7 Compare November 23, 2025 12:47

feat(inpainting): add inpainting endpoint with automatic model selection

d45913b

Signed-off-by: Greg <marianigregory@pm.me>

gmaOCR force-pushed the fix/inpainting-backend-response-and-cuda branch from dd5ffe7 to d45913b Compare November 23, 2025 12:48

	cfg, ok := c.Get("MODEL_CONFIG").(*config.ModelConfig)
	cfg, ok := c.Get(middleware.CONTEXT_LOCALS_KEY_MODEL_CONFIG).(*config.ModelConfig)

		jsonName := fmt.Sprintf("inpaint_%s.json", id)
		jsonPath := filepath.Join(tmpDir, jsonName)

	jsonName := fmt.Sprintf("inpaint_%s.json", id)
	jsonPath := filepath.Join(tmpDir, jsonName)
	jsonPath := filepath.Join(tmpDir, fmt.Sprintf("inpaint_%s.json", id))

Uh oh!

feat(inpainting): add inpainting endpoint, wire ImageGenerationFunc and return generated image URL #7328

Are you sure you want to change the base?

feat(inpainting): add inpainting endpoint, wire ImageGenerationFunc and return generated image URL #7328

Conversation

gmaOCR commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for localai ready!

Uh oh!

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

mudler Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmaOCR Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

gmaOCR commented Nov 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Nov 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 23, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gmaOCR commented Nov 21, 2025 •

edited

Loading

netlify bot commented Nov 21, 2025 •

edited

Loading

mudler Nov 21, 2025 •

edited

Loading

gmaOCR commented Nov 23, 2025 •

edited

Loading