Explicit and customizable error messages for Endpoint BadRequest (#2650)

[987Nabil]

@jdegoes I did not follow the plan as described in the ticket due to learnings while implementing it.

If the user does not handle the error explicitly, then we encode an error in a content-type acceptable to the HTTP client, perhaps defaulting to application/json. Moreover, we use a format that is somewhat standardized for returning error details. The main consideration, however, is just making sure that when users don't handle encoding/decoding errors, we do something that helps end-users diagnose and fix their encoding/decoding issues.

✅ The solution is checking the accept header and falling back to json
❌ I did not follow any of these error formats. Why? Because there are so many and I have no good reason to follow any. I chose the simplest possible solution. If devs want to follow a format, they should imho implement it themselves.

We need an operator that lifts encoding/decoding errors (currently defects) into E, and we need to document that, show how it's used, and make it common practice to deal with it.

In the solution on main, the decoder errors are handled in Endpoint#implement. And I think this makes sense. But this means, that after we have routes/handler, we can't lift the error type anymore, since it is handled. Lifting it on the level of Endpoint seems wrong, because I think we should not make it possible to create a decoding error in the handler handed over to implement. I instead chose, to have a new parameter of Endpoint, that only defines how decoding errors are handled in terms of a Codec. This way, we do not interfere with the Endpoint error type, but still have the possibility to analyse the possible errors and add them for example to open api (not impl. in this PR)

We need another operator that logs the codec errors in a standard way, useful for development.

Endpoint#logCodecError does the job. But I am thinking if a more general tabCodecErrorwould be better.

The operators should be defined probably on Handler all the way up to Routes (including Route).

This does not seem to fit the general design.

fixes #2650
/claim #2650

[swoogles]

The errors returned to the client are great! 👍

Clarification on .logCodecError -
I thought that by adding that to my endpoint definition, zio-http would start logging the errors on the backend as well, but that doesn't seem to be happening.
Do I need to enable something else at the server level to start seeing these errors?

[987Nabil]

@swoogles I just tried it and .logCodecError(e => e.getMessage()) leads to a log message being printed with error level. Maybe your logging settings are the issue?

[swoogles]

@987Nabil Can you point me towards a working example? I've been tweaking settings without success.

[987Nabil]

@swoogles btw this does not work for mismatches in the path. It will always lead to 404 without any log or custom error response.

  val getUser =
    Endpoint(Method.GET / "users" / int("userId")).in[Int].out[Int].logCodecError(e => e.getMessage())

  val getUserRoute =
    getUser.implement {
      handler { (id: Int, _: Int) =>
        id
      }
    }

[swoogles]

@987Nabil Yep, that's what I would expect. In my case I am giving bad payloads to correct URLs and hoping to see the failure details for 400s.

[jdegoes]

Implicits should not have to be imported. This means this should be located in the companion object of Dom or in the companion object of HttpContentCodec.

[jdegoes]

This has a super weird structure. Usually when I make something with this type of structure, I try to step back and throw away irrelevant details or rethink the abstraction.

First off, I would inline & delete MapError, as without it, this class doesn't carry its weight.

Then you have:

  log: Option[HttpCodecError => String],
  f: (HttpCodecError, Chunk[MediaTypeWithQFactor]) => Err,
  codec: HttpCodec[HttpCodecType.ResponseType, Err],

This is still super weird. Picking it apart by capabilities:

1. It MIGHT have the capability to turn an HttpCodecError into a String (for purposes of logging?).
2. It can turn an HttpCodecError into an Err.
3. It can turn an Err into response, and a response into an Err.

A few observations:

• It seems like we are going from HttpCodecError to Err only so that we can then go into a response.
• It is odd to pass Chunk[MediaTypeWithQFactor] into the function f. Why? Because it must return an Err, which is a user-defined error type that, presumably, has no protocol-specific details, such as media type. The only case where this is used seems to be a type-unsafe Err (wildcard type used, _), which further supports the hypothesis that the media type should not be relevant for Err.

Now, let's consider the use case:

The user wants to encode HttpCodecError in a way that is consistent with how they are encoding their own errors, so they can provide a uniform API experience to developers working with the API.

Note that this use case, if correct, is one-sided: there is no requirement for a ZIO HTTP Endpoint client of the API to be able to decode some codec error into a typed error (in fact, it's not really recoverable at that stage).

That means we can probably get away with:

decodeError: (HttpCodecError, Request) => Response,
domainError: HttpCodec[HttpCodecType.ResponseType, Err],

(This allows you to go to string if you want, for logging purposes, inasmuch as Response can be string-ified intelligently.)

On the other hand, if we need the codec errors in the typed error channel, then we might try:

decodeError: HttpCodec[HttpCodecType.ResponseType, HttpCodecError],
domainError: HttpCodec[HttpCodecType.ResponseType, Err],

This one says we can go both ways: from codec errors into responses, and back again; and from typed errors into responses, and back again. The user has control over how to turn HttpCodecError into a response, without "contaminating" their own E (typed error channel).

This encoding also gets ride of the media types because you can use fallback, perhaps, for different media types.

What do you think?

[jdegoes]

Looking back at this, I see the second formulation does not allow the automatic HTTP client to handle Err in a type-safe way. I am not sure if that's a problem because codec errors should not happen.

[jdegoes]

emptyErrorResponse

[jdegoes]

This should be in BinaryCodecWithSchema.scala

[987Nabil]

done

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 49.52681% with 160 lines in your changes are missing coverage. Please review.

Project coverage is 64.48%. Comparing base (2591f9f) to head (24e5161).
Report is 2 commits behind head on main.

Files	Patch %	Lines
...cala/zio/http/codec/internal/TextBinaryCodec.scala	20.68%	92 Missing ⚠️
...ed/src/main/scala/zio/http/endpoint/Endpoint.scala	55.76%	23 Missing ⚠️
...n/scala/zio/http/endpoint/openapi/OpenAPIGen.scala	60.86%	18 Missing ⚠️
.../scala/zio/http/endpoint/HttpCodecErrorCodec.scala	55.88%	15 Missing ⚠️
...c/main/scala/zio/http/codec/HttpContentCodec.scala	85.71%	5 Missing ⚠️
...main/scala/zio/http/codec/internal/BodyCodec.scala	66.66%	4 Missing ⚠️
...io-http/shared/src/main/scala/zio/http/Route.scala	0.00%	1 Missing ⚠️
...o-http/shared/src/main/scala/zio/http/Routes.scala	0.00%	1 Missing ⚠️
...scala/zio/http/codec/internal/EncoderDecoder.scala	90.90%	1 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2714      +/-   ##
==========================================
+ Coverage   64.41%   64.48%   +0.06%     
==========================================
  Files         148      150       +2     
  Lines        8669     8779     +110     
  Branches     1545     1566      +21     
==========================================
+ Hits         5584     5661      +77     
- Misses       3085     3118      +33     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jdegoes]

I'd prefer a 2nd matches, rather than introducing a boolean parameter with a default. Maybe matchesIgnoreParams.

[jdegoes]

Why were these all renamed??

[987Nabil]

After changing the order, the compiler could not infer the right apply. So I gave them explicit names.

[jdegoes]

You don't know this is an HttpCodecError. It could be other defects as well.

[987Nabil]

I do know. That is what line 202 isHttpCodecError is checking :) That was even there before my changes. All others go to the default case

[jdegoes]

It would be best to do it in one step then, rather than check first and extract later (Option extraction, for example). In any case, if the logic is correct due to the branch its in (I didn't see that), my mistake!

[jdegoes]

Dead code

[jdegoes]

👍

[jdegoes]

This has type HttpCodec, so it should be inside the companion object of HttpCodec. Could be called responseErrorCodec or something. That would also eliminate this useless object that only has a single static field.

[jdegoes]

@987Nabil Ping me when ready for a final review. Looking great!

[987Nabil]

@jdegoes Should be ready now

[swoogles]

@987Nabil Looks like this needs latest changes in main before it's ready.

[987Nabil]

@swoogles no. There are changes, but it is mergeable.