Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix the response status code logged in 'query stats' on error #8407

Merged
merged 6 commits into from
Jun 18, 2024
Merged

Fix the response status code logged in 'query stats' on error #8407

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
e8cc9b6
Fix the response status code logged in 'query stats' on error
pracucci Jun 17, 2024
9567322
Added unit test
pracucci Jun 18, 2024
f83caa0
Add CHANGELOG entry
pracucci Jun 18, 2024
2c20298
Fix PR number in CHANGELOG
pracucci Jun 18, 2024
b20987d
Added more test cases to TestWriteError
pracucci Jun 18, 2024
793436b
Removed named return parameter to make sure we don't forget setting it
pracucci Jun 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
34 changes: 27 additions & 7 deletions pkg/frontend/transport/handler.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -220,8 +220,9 @@ func (f *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
queryResponseTime := time.Since(startTime)

if err != nil {
writeError(w, err)
f.reportQueryStats(r, params, startTime, queryResponseTime, 0, queryDetails, 0, err)
// TODO unit test it
statusCode := writeError(w, err)
f.reportQueryStats(r, params, startTime, queryResponseTime, 0, queryDetails, statusCode, err)
return
}

Expand Down Expand Up @@ -428,7 +429,8 @@ func formatRequestHeaders(h *http.Header, headersToLog []string) (fields []any)
return fields
}

func writeError(w http.ResponseWriter, err error) {
// writeError writes the error response to http.ResponseWriter, and returns the response HTTP status code.
func writeError(w http.ResponseWriter, err error) (statusCode int) {
switch {
case errors.Is(err, context.Canceled):
err = errCanceled
Expand All @@ -440,13 +442,31 @@ func writeError(w http.ResponseWriter, err error) {
}
}

// if the error is an APIError, ensure it gets written as a JSON response
if resp, ok := apierror.HTTPResponseFromError(err); ok {
_ = httpgrpc.WriteResponse(w, resp)
var (
res *httpgrpc.HTTPResponse
ok bool
)

// If the error is an APIError, ensure it gets written as a JSON response.
// Otherwise, check if there's a response encoded in the gRPC error.
res, ok = apierror.HTTPResponseFromError(err)
if !ok {
res, ok = httpgrpc.HTTPResponseFromError(err)
}

// If we've been able to get the HTTP response from the error, then we send
// it with the right status code and response body content.
if res != nil {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to reviewers: the logic should have not been changed at all (at least not intentionally). What I'm doing here is exactly what we did before calling httpgrpc.WriteResponse(w, resp) and httpgrpc.WriteError(w, err).

statusCode = int(res.Code)
_ = httpgrpc.WriteResponse(w, res)
return
}

httpgrpc.WriteError(w, err)
// Otherwise, we do fallback to a 5xx error, returning the non-formatted error
// message in the response body.
statusCode = http.StatusInternalServerError
krajorama marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think it works better as a fallback if you put this line at the top of the function, then any time we cannot or miss updating it, the result will be http.StatusInternalServerError

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see the problem but I'm not convinced that having a default is the right thing to do. I want to make sure that we always return the right status code. To solve it, I took a different approach and I removed the named return parameter so that it's intentional what we return: 793436b

http.Error(w, err.Error(), statusCode)
return
}

func writeServiceTimingHeader(queryResponseTime time.Duration, headers http.Header, stats *querier_stats.Stats) {
Expand Down
8 changes: 7 additions & 1 deletion pkg/frontend/transport/handler_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ import (
"github.com/stretchr/testify/require"
"go.uber.org/atomic"

apierror "github.com/grafana/mimir/pkg/api/error"
"github.com/grafana/mimir/pkg/frontend/querymiddleware"
"github.com/grafana/mimir/pkg/querier/api"
"github.com/grafana/mimir/pkg/util/activitytracker"
Expand All @@ -54,12 +55,17 @@ func TestWriteError(t *testing.T) {
}{
{http.StatusInternalServerError, errors.New("unknown")},
{http.StatusGatewayTimeout, context.DeadlineExceeded},
{http.StatusGatewayTimeout, errors.Wrap(context.DeadlineExceeded, "an error occurred")},
{StatusClientClosedRequest, context.Canceled},
{StatusClientClosedRequest, errors.Wrap(context.Canceled, "an error occurred")},
{http.StatusBadRequest, httpgrpc.Errorf(http.StatusBadRequest, "")},
{http.StatusBadRequest, errors.Wrap(httpgrpc.Errorf(http.StatusBadRequest, ""), "an error occurred")},
{http.StatusBadRequest, apierror.New(apierror.TypeBadData, "")},
pracucci marked this conversation as resolved.
Show resolved Hide resolved
{http.StatusBadRequest, errors.Wrap(apierror.New(apierror.TypeBadData, "invalid request"), "an error occurred")},
} {
t.Run(test.err.Error(), func(t *testing.T) {
w := httptest.NewRecorder()
writeError(w, test.err)
require.Equal(t, test.status, writeError(w, test.err))
require.Equal(t, test.status, w.Result().StatusCode)
})
}
Expand Down