feat: migrate to echo and enable cancellation of non-streaming requests #7270
Conversation
✅ Deploy Preview for localai ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| // Handle 404 errors with HTML rendering when appropriate | ||
| if code == http.StatusNotFound { | ||
| notFoundHandler(c) |
Check warning
Code scanning / gosec
Errors unhandled Warning
| c.JSON(code, schema.ErrorResponse{ | ||
| Error: &schema.APIError{Message: err.Error(), Code: code}, | ||
| }) |
Check warning
Code scanning / gosec
Errors unhandled Warning
| if errors.As(err, &he) { | ||
| code = he.Code | ||
| } | ||
| c.NoContent(code) |
Check warning
Code scanning / gosec
Errors unhandled Warning
| return metricsService.Shutdown() | ||
| e.Use(localai.LocalAIMetricsAPIMiddleware(metricsService)) | ||
| e.Server.RegisterOnShutdown(func() { | ||
| metricsService.Shutdown() |
Check warning
Code scanning / gosec
Errors unhandled Warning
| if body := c.Body(); len(body) > 0 { | ||
| body := make([]byte, 0) | ||
| if c.Request().Body != nil { | ||
| c.Request().Body.Read(body) |
Check warning
Code scanning / gosec
Errors unhandled Warning
| } | ||
|
|
||
| // Decode each request's message content | ||
| imgIndex, vidIndex, audioIndex := 0, 0, 0 |
Check warning
Code scanning / gosec
Errors unhandled Warning
| _, err := tmpl.New(templateName).Parse(string(data)) | ||
| if err != nil { | ||
| // If parsing fails, try parsing without explicit name (for templates with {{define}}) | ||
| tmpl.Parse(string(data)) |
Check warning
Code scanning / gosec
Errors unhandled Warning
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
mudler
force-pushed
the
echo
branch
4 times, most recently
from
f773750 to
58a0b6c
Compare
| app := echo.New() | ||
| // Set up a simple renderer for the test | ||
| app.Renderer = &testRenderer{} | ||
| app.POST("/import-model", ImportModelEndpoint(modelConfigLoader, applicationConfig)) |
There was a problem hiding this comment.
changed PUT to POST, I'm not sure if that is deliberate?
There was a problem hiding this comment.
not really, good catch! but it's just used in tests, shouldn't make any difference 🤷
mudler
force-pushed
the
echo
branch
2 times, most recently
from
2d1e3ab to
a7c4e44
Compare
mudler
changed the title
Description
This PR migrates to echo. One of the main reason to migrate away from go-fiber/fasthttp is the lack of support of detecting properly client cancellation valyala/fasthttp#468 (comment), which would have the unwanted effect of booking resources for AI inferencing even if clients disconnect.
I've tried several workarounds in #7187 , but none of these was reliable ( had to revert in #7266 ). Finally, I've came to the conclusion that the effort involved in maintaining go-fiber (which is based over fasthttp) is higher overall for the little gain that we would have (the real computation is offloaded, so we don't perform high-intensive tasks in the golang layer) by using it.
Aside from migrating away, this PR brings back support of cancellation of client requests, which propagates back cancellation of request to the grpc backend.
Notes for Reviewers
Signed commits