Tomcat exception being tagged as an error incorrectly

[Jarda8]

Hi team. We use the DD agent together with Spring Boot 3 REST application. We use the trace.servlet.request.errors metric to monitor errors. From what I read and expect, it should not tag 4XX responses as an error by default and in majority of cases it really does not. However, when we get a request with some invalid characters in the URL (a client error we do not care about), it is marked as an error, even though the response has 400 status. There is an exception thrown in Tomcat visible in the DD trace, but it is caught immediately and handled still in Tomcat. It does not even touches our code, as the validation and response creation happens within Tomcat. An example of the exception:

java.lang.IllegalArgumentException: Invalid character found in the request target [/hierarchy-entities/custom-fields-values?file:\ ]. The valid characters are defined in RFC 7230 and RFC 3986
	at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:482)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:263)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:894)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Thread.java:1589)

It seems like a problem within the DD agent, tagging this as an error incorrectly. Or is it an expected behaviour for some reason?

[amarziali]

Actually the error is wrapped by tomcat in the response hence then extracted by the DD agent and reported. Older version of tomcat are accepting backslashes hence the error is not thrown in the same case. We are working to understand if there is a reasonable fix to do without loosing information.
On the same time you can workaround the issue by configuring tomcat to accept backslashes.
The sys props are:

• -Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true
• -Dorg.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=true

[amarziali]

should be fixed by #5128

[Jarda8]

Thank you for the insight. And the fix.