Consider routing messages that fail due to a fatal exception straight into the dead letter topic

[v-chernyshev]

Expected Behavior

It would be great to have a setting that would instruct the retryable topics system to skip the recovery attempts if one of the fatal-by-default exceptions is encountered during the message processing as it is highly likely that the malformed messages will fail in the retry topics too.

Current Behavior

The current implementation (assuming no notRetryOn overrides) routes the failed messages to the next retry topic in the recovery chain, until they eventually reach the DLT. There doesn't seem to be a way to programmatically query the default list of fatal exceptions either, which basically means that I have to copy-paste them somewhere to pass into noRetryOn.

Context

The main goal here is to avoid pointless recovery attempts for poison pill messages. See this StackOverflow question for more information. The only workaround I can think of is declaring a duplicate list of fatal exceptions, e.g.:

private static final List<Class<? extends Throwable>> FATAL_EXCEPTIONS = List.of(
        DeserializationException.class,
        MessageConversionException.class,
        ConversionException.class,
        MethodArgumentResolutionException.class,
        NoSuchMethodException.class,
        ClassCastException.class);

It may then be passed into notRetryOn.

[tomazfernandes]

Hi @v-chernyshev, thanks for the suggestion.

Just to be clear, we're talking about a manageable global set of FATAL exceptions that will route the record straight to the DLT, different from the topic-by-topic classification configuration we currently have.

I'm currently working on another issue that is related and should address this too, so hopefully we should have this ready soon.

[tomazfernandes]

@v-chernyshev, I opened a PR with this feature, if maybe you want to take a look
#2123

Thanks again for the suggestion

[v-chernyshev]

Thanks for implementing this suggestion so quickly! I had a look through the PR and it looks good to me, but I'm by no means comfortable with the implementation details of Spring Kafka :)

[tomazfernandes]

@v-chernyshev thanks for looking into the PR and for the feature suggestion. This feature will be available in today’s release, feel free to try it out and provide feedback!