Add an option to allow 404 responses from ExceptionHandlers

[JunTaoLuo]

Addresses #26528

Description

We addressed the issue #21510 in RC1 but introduced a breaking change in the default behaviour of the middleware. Specifically, we decided that a 404 response from the exception handler should be considered an error in the server and we will rethrow the original exception instead of returning the 404 response. This is to address a common pitfall where the exception handler middleware was misconfigured with a non-existent path.

However, the change in default behaviour regressed use cases where the exception handler intentionally produces a 404 response as mentioned in #26528. This PR introduces a fix where we add an option on the middleware to allow such responses. When enabled, this option considers 404 responses produced by the exception handler to be valid and will not rethrow the original exception.

Customer Impact

This issue was reported by @danikun in #26528. The breaking change introduced in 5.0 RC1 regressed the customer's exception handling logic which maps some exceptions to 404s. There is no acceptable workaround. We considered alternatives such as invoking CompleteAsync in the exception handler to flush the 404 response but this won't work in all environments; for example, unit testing using TestServer would not work since it doesn't implement CompleteAsync as discussed in #25288 (comment).

Regression?

Yes this is a regression that was introduced in #25062 in 5.0 RC1

Risk

Low. The solution is limited to this middleware and an unit test is added to verify its correctness.

[ghost]

Hello human! Please make sure you've included the Shiproom Template in a comment or (preferably) the PR description. Also, make sure this PR is not marked as a draft and is ready-to-merge.

[Tratcher]

@danikun

[JunTaoLuo]

Given this will require 5.0 shiproom approval, it's unlikely this will be merged today. If I don't respond promptly while I'm OOF, I nominate @Tratcher to merge as needed.

[danikun]

Thanks! Good job

[JunTaoLuo]

@Pilchie Is this being looked at by shiproom?

[Pilchie]

@JunTaoLuo look again 😉

[davidfowl]

This isn't documented as part of that original breaking change.