Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ClassNotFoundException when loading resteasy application on Wildfy application server #4368

Closed
vallihalldors opened this issue Dec 2, 2022 · 13 comments
Assignees
Labels

Comments

@vallihalldors
Copy link

We have an issue after upgrading to version 1.1.x (1.1.4). During startup of a service running on Wildfly version 26.1.2.Final when loading RESTEasy application (RESTEasy version 4.7.4.Final) that uses Spring (version 5.3.20) to load the context we get a ClassNotFoundException on datadog.trace.instrumentation.springweb.OrderedServletPathRequestFilter:

[dd.trace 2022-12-02 11:04:54:317 +0000] [dd-task-scheduler] WARN datadog.telemetry.dependency.DependencyResolver - Unable to guess nested dependency for uri 'jar:file:/tmp/wildfly-bootable-server14374401890295614616/modules/system/layers/base/org/wildfly/security/elytron-base/main/wildfly-elytron-util-1.19.0.Final.jar!/': '!/' not found
[dd.trace 2022-12-02 11:04:54:317 +0000] [dd-task-scheduler] WARN datadog.telemetry.dependency.DependencyResolverQueue - unable to detect dependency for URI jar:file:/tmp/wildfly-bootable-server14374401890295614616/modules/system/layers/base/org/wildfly/security/elytron-base/main/wildfly-elytron-util-1.19.0.Final.jar!/
[dd.trace 2022-12-02 11:04:54:440 +0000] [ServerService Thread Pool -- 46] WARN datadog.telemetry.dependency.JbossVirtualFileHelper - Physical file /tmp/wildfly-bootable-server14374401890295614616/standalone/tmp/vfs/deployment/deploymenta2d37e72c3863c9c/spring-expression-5.3.20.jar-f70643092e071abd/contents is not a jar
[dd.trace 2022-12-02 11:04:54:641 +0000] [ServerService Thread Pool -- 46] WARN datadog.telemetry.dependency.JbossVirtualFileHelper - Physical file /tmp/wildfly-bootable-server14374401890295614616/standalone/tmp/vfs/deployment/deploymenta2d37e72c3863c9c/spring-aop-5.3.20.jar-18ee855565844f58/contents is not a jar
11:04:54,777 ERROR [org.springframework.web.context.ContextLoader] (ServerService Thread Pool -- 46) Context initialization failed: java.lang.IllegalStateException: RESTEASY013005: Could not convert 'datadog.trace.instrumentation.springweb.OrderedServletPathRequestFilter' to a class.
       at deployment.time.ear//org.jboss.resteasy.plugins.spring.SpringBeanProcessor.getBeanClass(SpringBeanProcessor.java:505)
       at deployment.time.ear//org.jboss.resteasy.plugins.spring.SpringBeanProcessor.getBeanClass(SpringBeanProcessor.java:414)
       at deployment.time.ear//org.jboss.resteasy.plugins.spring.SpringBeanProcessor.processBean(SpringBeanProcessor.java:306)
       at deployment.time.ear//org.jboss.resteasy.plugins.spring.SpringBeanProcessor.postProcessBeanFactory(SpringBeanProcessor.java:279)
       at deployment.time.ear//org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:325)
       at deployment.time.ear//org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:148)
       at deployment.time.ear//org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:746)
       at deployment.time.ear//org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:564)
       at deployment.time.ear//org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401)
       at deployment.time.ear//com.betware.rest.resteasy.listener.BwSpringContextLoaderListener.configureAndRefreshWebApplicationContext(BwSpringContextLoaderListener.java:32)
       at deployment.time.ear//org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292)
       at deployment.time.ear//org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
       at deployment.time.ear//org.jboss.resteasy.plugins.spring.SpringContextLoaderListener.contextInitialized(SpringContextLoaderListener.java:57)
       at io.undertow.servlet@2.2.17.Final//io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:187)
       at io.undertow.servlet@2.2.17.Final//io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:219)
       at io.undertow.servlet@2.2.17.Final//io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:187)
       at io.undertow.servlet@2.2.17.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
       at io.undertow.servlet@2.2.17.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
       at org.wildfly.extension.undertow@26.1.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
       at org.wildfly.extension.undertow@26.1.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
       at org.wildfly.extension.undertow@26.1.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
       at org.wildfly.extension.undertow@26.1.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
       at io.undertow.servlet@2.2.17.Final//io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:255)
       at org.wildfly.extension.undertow@26.1.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:105)
       at org.wildfly.extension.undertow@26.1.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:87)
       at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
       at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
       at org.jboss.threads@2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
       at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
       at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
       at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
       at java.base/java.lang.Thread.run(Thread.java:829)
       at org.jboss.threads@2.4.0.Final//org.jboss.threads.JBossThread.run(JBossThread.java:513)
Caused by: java.lang.ClassNotFoundException: datadog.trace.instrumentation.springweb.OrderedServletPathRequestFilter from [Module "deployment.time.ear.time-cc.war" from Service Module Loader]
       at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:200)
       at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:410)
       at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
       at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116)
       at java.base/java.lang.Class.forName0(Native Method)
       at java.base/java.lang.Class.forName(Class.java:398)
       at deployment.time.ear//org.springframework.util.ClassUtils.forName(ClassUtils.java:284)
       at deployment.time.ear//org.jboss.resteasy.plugins.spring.SpringBeanProcessor.getBeanClass(SpringBeanProcessor.java:501)
       ... 32 more

I was also wondering if there is a way to get rid of those warnings from the DependencyResolver since we have a lot of those?

@mcculls
Copy link
Contributor

mcculls commented Dec 6, 2022

Hi @vallihalldors, the DependencyResolver warnings come from instrumentation telemetry which was recently enabled.

Instrumentation telemetry can be turned off by setting this environment variable
DD_INSTRUMENTATION_TELEMETRY_ENABLED=false

or by adding this system property to your JVM command-line
-Ddd.instrumentation.telemetry.enabled=false

Turning it off will remove the DependencyResolver warnings without affecting traces or metrics because it's a separate feature.

@mcculls
Copy link
Contributor

mcculls commented Dec 6, 2022

Regarding the OrderedServletPathRequestFilter exception - it looks like Spring is attempting to re-resolve that class by using Class.forName even though the Java Tracer registers the already loaded class as part of its bean definition. The Class.forName call is failing due to visibility restrictions in Wildfly.

You can disable addition of OrderedServletPathRequestFilter with this environment variable:
DD_INTEGRATION_SPRING_PATH_FILTER_ENABLED=false

or by adding this system property to your JVM command-line
-Ddd.integration.spring-path-filter.enabled=false

while we investigate ways to workaround this Spring lookup issue on Wildfly.

@vallihalldors
Copy link
Author

Thanks @mcculls. I've applied the workaround you suggested and it works fine.

@erodewald
Copy link

Also seeing similar issues while using the following configuration:

Framework: Ratpack
Java: 17
Gradle: 7.6
Datadog agent: 1.2.0 (does not appear unique to this version FYI)

It hangs during a Gradle test when a particular dependency is included on the classpath. I haven't figured out what characteristic of the dependency causes it but excluding it entirely also fixes the issue.

With the standard DD agent configuration:

[dd.trace 2022-12-09 18:26:10:119 -0600] [dd-task-scheduler] WARN datadog.telemetry.dependency.DependencyResolverQueue - unable to detect dependency for URI file:/Users/user/app-web/build/classes/groovy/test/
[dd.trace 2022-12-09 18:26:41:117 -0600] [dd-task-scheduler] WARN datadog.telemetry.dependency.DependencyResolver - Unable to guess nested dependency for uri 'jar:file:/user/.gradle/caches/modules-2/files-2.1/{{dependencyName}}/2.1.3/2feca93ff0fe8b239606a5542fb956b23a53c95c/{{dependencyName}}-2.1.3.jar!/SpockConfig.groovy': '!/' not found
[dd.trace 2022-12-09 18:26:41:117 -0600] [dd-task-scheduler] WARN datadog.telemetry.dependency.DependencyResolverQueue - unable to detect dependency for URI jar:file:/Users/user/.gradle/caches/modules-2/files-2.1/{{dependencyName}}/2.1.3/2feca93ff0fe8b239606a5542fb956b23a53c95c/{{dependencyName}}-2.1.3.jar!/SpockConfig.groovy

It also just hangs indefinitely if I add the following environment variables:

DD_CIVISIBILITY_AGENTLESS_ENABLED=true
DD_INSTRUMENTATION_TELEMETRY_ENABLED=false
DD_INTEGRATION_SPRING_PATH_FILTER_ENABLED=false
./gradlew project:check -Pdd-civisibility -i
DATADOG TRACER CONFIGURATION
 {
 	"version": "1.2.0~a26b8ca0ff",
 	"os_name": "Mac OS X",
 	"os_version": "12.5",
 	"architecture": "x86_64",
 	"lang": "jvm",
 	"lang_version": "17.0.2",
 	"jvm_vendor": "Amazon.com Inc.",
 	"jvm_version": "17.0.2+8-LTS",
 	"java_class_version": "61.0",
 	"http_nonProxyHosts": "null",
 	"http_proxyHost": "null",
 	"enabled": true,
 	"service": "warrantor",
 	"agent_url": "http://localhost:8126",
 	"agent_error": true,
 	"debug": false,
 	"analytics_enabled": false,
 	"sampling_rules": [{}, {}],
 	"priority_sampling_enabled": true,
 	"logs_correlation_enabled": true,
 	"profiling_enabled": false,
 	"remote_config_enabled": false,
 	"debugger_enabled": false,
 	"appsec_enabled": "FULLY_DISABLED",
 	"telemetry_enabled": false,
 	"dd_version": "",
 	"health_checks_enabled": true,
 	"configuration_file": "no config file present",
 	"runtime_id": "b7eff517-1251-43b1-9cc2-8d9d6c3add1f",
 	"logging_settings": {
 		"levelInBrackets": false,
 		"dateTimeFormat": "'[dd.trace 'yyyy-MM-dd HH:mm:ss:SSS Z']'",
 		"logFile": "System.err",
 		"configurationFile": "simplelogger.properties",
 		"showShortLogName": false,
 		"showDateTime": true,
 		"showLogName": true,
 		"showThreadName": true,
 		"defaultLogLevel": "INFO",
 		"warnLevelString": "WARN",
 		"embedException": false
 	},
 	"cws_enabled": false,
 	"cws_tls_refresh": 5000
 }

Any suggestions?

@smola smola added the comp: telemetry Telemetry label Jan 2, 2023
@mcculls
Copy link
Contributor

mcculls commented Jan 25, 2023

The OrderedServletPathRequestFilter exception should be fixed by #4543 which was included in 1.5.0 (released last week)

@mcculls
Copy link
Contributor

mcculls commented Feb 13, 2023

Closing as the ClassNotFoundException mentioned in the original description has been fixed in 1.5.0

@mcculls mcculls closed this as completed Feb 13, 2023
@pacome2018
Copy link

I still have this warning with version 1.5.0 :
[dd.trace 2023-02-20 16:07:28:980 +0000] [dd-task-scheduler] WARN datadog.telemetry.dependency.DependencyResolverQueue - unable to detect dependency for URI file:/var/task/

After that, Spring boot restarts and runs fine, but the dual booting is a problem.

@mcculls
Copy link
Contributor

mcculls commented Feb 21, 2023

@pacome2018 this looks unrelated to the original reported issue which was a class-loading issue related to a tracer bean definition, please open a new issue with as much detail as possible from your application logs before Spring-Boot decided to restart.

You may want to enable tracer debug to see if that gives you further details.

@vallihalldors
Copy link
Author

@mcculls I still get the same error with version 1.10.0. Do I need to apply the following workaround as a permanent solution?
DD_INTEGRATION_SPRING_PATH_FILTER_ENABLED=false

@mcculls
Copy link
Contributor

mcculls commented Mar 16, 2023

@vallihalldors can you paste the exception you see with 1.10.0? The factory that was added in #4543 should avoid the ClassNotFoundException originally reported (assuming that org.jboss.resteasy.plugins.spring.SpringBeanProcessor is interacting with the bean via Spring's bean factory - if it's circumventing that factory by doing its own class-loading then we'll need to add a separate fix just for resteasy-spring)

Could you also enable tracer debug and see if there are any related exceptions (if you need further investigation of the debug log then you can send it via your support contact)

@vallihalldors
Copy link
Author

@mcculls hope this helps:

time-service-time-1  | [dd.trace 2023-03-16 13:12:12:097 +0000] [ServerService Thread Pool -- 49] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextRequestRewriter - Found context-store access - instrumentation.class=jax-rs
time-service-time-1  | [dd.trace 2023-03-16 13:12:12:097 +0000] [ServerService Thread Pool -- 49] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextRequestRewriter - Rewriting context-store map fetch - instrumentation.class=jax-rs instrumentation.target.context=javax.ws.rs.container.AsyncResponse->datadog.trace.bootstrap.instrumentation.api.AgentSpan
time-service-time-1  | [dd.trace 2023-03-16 13:12:12:097 +0000] [ServerService Thread Pool -- 49] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextRequestRewriter - Found context-store access - instrumentation.class=jax-rs
time-service-time-1  | [dd.trace 2023-03-16 13:12:12:097 +0000] [ServerService Thread Pool -- 49] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextRequestRewriter - Rewriting context-store map fetch - instrumentation.class=jax-rs instrumentation.target.context=javax.ws.rs.container.AsyncResponse->datadog.trace.bootstrap.instrumentation.api.AgentSpan
time-service-time-1  | [dd.trace 2023-03-16 13:12:12:097 +0000] [ServerService Thread Pool -- 49] DEBUG datadog.trace.agent.tooling.AgentInstaller$TransformLoggingListener - Transformed - instrumentation.target.class=com.betware.rest.BwSystemServiceImpl instrumentation.target.classloader=ModuleClassLoader for Module "deployment.time.ear" from Service Module Loader
time-service-time-1  | [dd.trace 2023-03-16 13:12:12:109 +0000] [ServerService Thread Pool -- 49] DEBUG datadog.trace.agent.tooling.InstrumenterState - Instrumentation applied - instrumentation.names=[jax-rs,jaxrs,jax-rs-annotations] instrumentation.class=datadog.trace.instrumentation.jaxrs2.JaxRsAnnotationsInstrumentation instrumentation.target.classloader=ModuleClassLoader for Module "deployment.time.ear.time-cc.war" from Service Module Loader
time-service-time-1  | [dd.trace 2023-03-16 13:12:12:109 +0000] [ServerService Thread Pool -- 49] DEBUG datadog.trace.agent.tooling.HelperInjector - Injecting helper classes - instrumentation.class=JaxRsAnnotationsInstrumentation instrumentation.target.classloader=ModuleClassLoader for Module "deployment.time.ear.time-cc.war" from Service Module Loader instrumentation.helper_classes=[datadog.trace.agent.tooling.ClassHierarchyIterable,datadog.trace.agent.tooling.ClassHierarchyIterable$ClassIterator,datadog.trace.instrumentation.jaxrs2.JaxRsAnnotationsDecorator]
time-service-time-1  | [dd.trace 2023-03-16 13:12:12:113 +0000] [ServerService Thread Pool -- 49] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextRequestRewriter - Found context-store access - instrumentation.class=jax-rs
time-service-time-1  | [dd.trace 2023-03-16 13:12:12:113 +0000] [ServerService Thread Pool -- 49] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextRequestRewriter - Rewriting context-store map fetch - instrumentation.class=jax-rs instrumentation.target.context=javax.ws.rs.container.AsyncResponse->datadog.trace.bootstrap.instrumentation.api.AgentSpan
time-service-time-1  | [dd.trace 2023-03-16 13:12:12:113 +0000] [ServerService Thread Pool -- 49] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextRequestRewriter - Found context-store access - instrumentation.class=jax-rs
time-service-time-1  | [dd.trace 2023-03-16 13:12:12:113 +0000] [ServerService Thread Pool -- 49] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextRequestRewriter - Rewriting context-store map fetch - instrumentation.class=jax-rs instrumentation.target.context=javax.ws.rs.container.AsyncResponse->datadog.trace.bootstrap.instrumentation.api.AgentSpan
time-service-time-1  | [dd.trace 2023-03-16 13:12:12:113 +0000] [ServerService Thread Pool -- 49] DEBUG datadog.trace.agent.tooling.AgentInstaller$TransformLoggingListener - Transformed - instrumentation.target.class=com.betware.time.cc.rest.v1.BwTimeServiceImpl instrumentation.target.classloader=ModuleClassLoader for Module "deployment.time.ear.time-cc.war" from Service Module Loader
time-service-time-1  | [dd.trace 2023-03-16 13:12:12:130 +0000] [ServerService Thread Pool -- 49] DEBUG datadog.trace.agent.tooling.context.ShouldInjectFieldsMatcher - Added context-store field - instrumentation.target.class=org.springframework.beans.factory.support.DisposableBeanAdapter instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
time-service-time-1  | [dd.trace 2023-03-16 13:12:12:132 +0000] [ServerService Thread Pool -- 49] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextRequestRewriter - Found context-store access - instrumentation.class=java_concurrent
time-service-time-1  | [dd.trace 2023-03-16 13:12:12:132 +0000] [ServerService Thread Pool -- 49] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextRequestRewriter - Rewriting context-store map fetch - instrumentation.class=java_concurrent instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
time-service-time-1  | [dd.trace 2023-03-16 13:12:12:133 +0000] [ServerService Thread Pool -- 49] DEBUG datadog.trace.agent.tooling.AgentInstaller$TransformLoggingListener - Transformed - instrumentation.target.class=org.springframework.beans.factory.support.DisposableBeanAdapter instrumentation.target.classloader=ModuleClassLoader for Module "deployment.time.ear" from Service Module Loader
time-service-time-1  | 13:12:12,144 ERROR [org.springframework.web.context.ContextLoader] (ServerService Thread Pool -- 49) Context initialization failed: java.lang.IllegalStateException: RESTEASY013005: Could not convert 'datadog.trace.instrumentation.springweb.OrderedServletPathRequestFilter' to a class.
time-service-time-1  | 	at deployment.time.ear//org.jboss.resteasy.plugins.spring.SpringBeanProcessor.getBeanClass(SpringBeanProcessor.java:505)
time-service-time-1  | 	at deployment.time.ear//org.jboss.resteasy.plugins.spring.SpringBeanProcessor.getBeanClass(SpringBeanProcessor.java:414)
time-service-time-1  | 	at deployment.time.ear//org.jboss.resteasy.plugins.spring.SpringBeanProcessor.processBean(SpringBeanProcessor.java:306)
time-service-time-1  | 	at deployment.time.ear//org.jboss.resteasy.plugins.spring.SpringBeanProcessor.postProcessBeanFactory(SpringBeanProcessor.java:279)
time-service-time-1  | 	at deployment.time.ear//org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:325)
time-service-time-1  | 	at deployment.time.ear//org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:148)
time-service-time-1  | 	at deployment.time.ear//org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:746)
time-service-time-1  | 	at deployment.time.ear//org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:564)
time-service-time-1  | 	at deployment.time.ear//org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401)
time-service-time-1  | 	at deployment.time.ear//com.betware.rest.resteasy.listener.BwSpringContextLoaderListener.configureAndRefreshWebApplicationContext(BwSpringContextLoaderListener.java:32)
time-service-time-1  | 	at deployment.time.ear//org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292)
time-service-time-1  | 	at deployment.time.ear//org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
time-service-time-1  | 	at deployment.time.ear//org.jboss.resteasy.plugins.spring.SpringContextLoaderListener.contextInitialized(SpringContextLoaderListener.java:57)
time-service-time-1  | 	at io.undertow.servlet@2.2.17.Final//io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:187)
time-service-time-1  | 	at io.undertow.servlet@2.2.17.Final//io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:219)
time-service-time-1  | 	at io.undertow.servlet@2.2.17.Final//io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:187)
time-service-time-1  | 	at io.undertow.servlet@2.2.17.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
time-service-time-1  | 	at io.undertow.servlet@2.2.17.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
time-service-time-1  | 	at org.wildfly.extension.undertow@26.1.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
time-service-time-1  | 	at org.wildfly.extension.undertow@26.1.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
time-service-time-1  | 	at org.wildfly.extension.undertow@26.1.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
time-service-time-1  | 	at org.wildfly.extension.undertow@26.1.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
time-service-time-1  | 	at io.undertow.servlet@2.2.17.Final//io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:255)
time-service-time-1  | 	at org.wildfly.extension.undertow@26.1.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:105)
time-service-time-1  | 	at org.wildfly.extension.undertow@26.1.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:87)
time-service-time-1  | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
time-service-time-1  | 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
time-service-time-1  | 	at org.jboss.threads@2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
time-service-time-1  | 	at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
time-service-time-1  | 	at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
time-service-time-1  | 	at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
time-service-time-1  | 	at java.base/java.lang.Thread.run(Thread.java:829)
time-service-time-1  | 	at org.jboss.threads@2.4.0.Final//org.jboss.threads.JBossThread.run(JBossThread.java:513)
time-service-time-1  | Caused by: java.lang.ClassNotFoundException: datadog.trace.instrumentation.springweb.OrderedServletPathRequestFilter from [Module "deployment.time.ear.time-cc.war" from Service Module Loader]
time-service-time-1  | 	at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:200)
time-service-time-1  | 	at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:410)
time-service-time-1  | 	at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
time-service-time-1  | 	at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116)
time-service-time-1  | 	at java.base/java.lang.Class.forName0(Native Method)
time-service-time-1  | 	at java.base/java.lang.Class.forName(Class.java:398)
time-service-time-1  | 	at deployment.time.ear//org.springframework.util.ClassUtils.forName(ClassUtils.java:284)
time-service-time-1  | 	at deployment.time.ear//org.jboss.resteasy.plugins.spring.SpringBeanProcessor.getBeanClass(SpringBeanProcessor.java:501)
time-service-time-1  | 	... 32 more
time-service-time-1  | 
time-service-time-1  | 13:12:12,146 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 49) MSC000001: Failed to start service jboss.deployment.subunit."time.ear"."time-cc.war".undertow-deployment: org.jboss.msc.service.StartException in service jboss.deployment.subunit."time.ear"."time-cc.war".undertow-deployment: java.lang.RuntimeException: java.lang.IllegalStateException: RESTEASY013005: Could not convert 'datadog.trace.instrumentation.springweb.OrderedServletPathRequestFilter' to a class.
time-service-time-1  | 	at org.wildfly.extension.undertow@26.1.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:90)
time-service-time-1  | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
time-service-time-1  | 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
time-service-time-1  | 	at org.jboss.threads@2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
time-service-time-1  | 	at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
time-service-time-1  | 	at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
time-service-time-1  | 	at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
time-service-time-1  | 	at java.base/java.lang.Thread.run(Thread.java:829)
time-service-time-1  | 	at org.jboss.threads@2.4.0.Final//org.jboss.threads.JBossThread.run(JBossThread.java:513)
time-service-time-1  | Caused by: java.lang.RuntimeException: java.lang.IllegalStateException: RESTEASY013005: Could not convert 'datadog.trace.instrumentation.springweb.OrderedServletPathRequestFilter' to a class.
time-service-time-1  | 	at io.undertow.servlet@2.2.17.Final//io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:257)
time-service-time-1  | 	at org.wildfly.extension.undertow@26.1.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:105)
time-service-time-1  | 	at org.wildfly.extension.undertow@26.1.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:87)
time-service-time-1  | 	... 8 more
time-service-time-1  | Caused by: java.lang.IllegalStateException: RESTEASY013005: Could not convert 'datadog.trace.instrumentation.springweb.OrderedServletPathRequestFilter' to a class.
time-service-time-1  | 	at deployment.time.ear//org.jboss.resteasy.plugins.spring.SpringBeanProcessor.getBeanClass(SpringBeanProcessor.java:505)
time-service-time-1  | 	at deployment.time.ear//org.jboss.resteasy.plugins.spring.SpringBeanProcessor.getBeanClass(SpringBeanProcessor.java:414)
time-service-time-1  | 	at deployment.time.ear//org.jboss.resteasy.plugins.spring.SpringBeanProcessor.processBean(SpringBeanProcessor.java:306)
time-service-time-1  | 	at deployment.time.ear//org.jboss.resteasy.plugins.spring.SpringBeanProcessor.postProcessBeanFactory(SpringBeanProcessor.java:279)
time-service-time-1  | 	at deployment.time.ear//org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:325)
time-service-time-1  | 	at deployment.time.ear//org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:148)
time-service-time-1  | 	at deployment.time.ear//org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:746)
time-service-time-1  | 	at deployment.time.ear//org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:564)
time-service-time-1  | 	at deployment.time.ear//org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401)
time-service-time-1  | 	at deployment.time.ear//com.betware.rest.resteasy.listener.BwSpringContextLoaderListener.configureAndRefreshWebApplicationContext(BwSpringContextLoaderListener.java:32)
time-service-time-1  | 	at deployment.time.ear//org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292)
time-service-time-1  | 	at deployment.time.ear//org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
time-service-time-1  | 	at deployment.time.ear//org.jboss.resteasy.plugins.spring.SpringContextLoaderListener.contextInitialized(SpringContextLoaderListener.java:57)
time-service-time-1  | 	at io.undertow.servlet@2.2.17.Final//io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:187)
time-service-time-1  | 	at io.undertow.servlet@2.2.17.Final//io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:219)
time-service-time-1  | 	at io.undertow.servlet@2.2.17.Final//io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:187)
time-service-time-1  | 	at io.undertow.servlet@2.2.17.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
time-service-time-1  | 	at io.undertow.servlet@2.2.17.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
time-service-time-1  | 	at org.wildfly.extension.undertow@26.1.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
time-service-time-1  | 	at org.wildfly.extension.undertow@26.1.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
time-service-time-1  | 	at org.wildfly.extension.undertow@26.1.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
time-service-time-1  | 	at org.wildfly.extension.undertow@26.1.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
time-service-time-1  | 	at io.undertow.servlet@2.2.17.Final//io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:255)
time-service-time-1  | 	... 10 more
time-service-time-1  | Caused by: java.lang.ClassNotFoundException: datadog.trace.instrumentation.springweb.OrderedServletPathRequestFilter from [Module "deployment.time.ear.time-cc.war" from Service Module Loader]
time-service-time-1  | 	at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:200)
time-service-time-1  | 	at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:410)
time-service-time-1  | 	at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
time-service-time-1  | 	at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116)
time-service-time-1  | 	at java.base/java.lang.Class.forName0(Native Method)
time-service-time-1  | 	at java.base/java.lang.Class.forName(Class.java:398)
time-service-time-1  | 	at deployment.time.ear//org.springframework.util.ClassUtils.forName(ClassUtils.java:284)
time-service-time-1  | 	at deployment.time.ear//org.jboss.resteasy.plugins.spring.SpringBeanProcessor.getBeanClass(SpringBeanProcessor.java:501)
time-service-time-1  | 	... 32 more
time-service-time-1  | 
time-service-time-1  | [dd.trace 2023-03-16 13:12:12:149 +0000] [ServerService Thread Pool -- 49] DEBUG datadog.trace.agent.tooling.context.ShouldInjectFieldsMatcher - Will not add context-store field, alternate target found org.jboss.msc.service.ServiceControllerImpl$ControllerTask - instrumentation.target.class=org.jboss.msc.service.ServiceControllerImpl$DependencyFailedTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
time-service-time-1  | [dd.trace 2023-03-16 13:12:12:153 +0000] [ServerService Thread Pool -- 49] DEBUG datadog.trace.agent.tooling.AgentInstaller$TransformLoggingListener - Transformed - instrumentation.target.class=org.jboss.msc.service.ServiceControllerImpl$DependencyFailedTask instrumentation.target.classloader=ModuleClassLoader for Module "org.jboss.msc" version 1.4.13.Final from local module loader @3c87e6b7 (finder: local module finder @77bbadc (roots: /tmp/wildfly-bootable-server1955166504740324169/modules,/tmp/wildfly-bootable-server1955166504740324169/modules/system/layers/base))
time-service-time-1  | [dd.trace 2023-03-16 13:12:12:154 +0000] [ServerService Thread Pool -- 49] DEBUG datadog.trace.agent.tooling.context.ShouldInjectFieldsMatcher - Will not add context-store field, alternate target found org.jboss.msc.service.ServiceControllerImpl$ControllerTask - instrumentation.target.class=org.jboss.msc.service.ServiceControllerImpl$RemoveChildrenTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
time-service-time-1  | [dd.trace 2023-03-16 13:12:12:156 +0000] [ServerService Thread Pool -- 49] DEBUG datadog.trace.agent.tooling.AgentInstaller$TransformLoggingListener - Transformed - instrumentation.target.class=org.jboss.msc.service.ServiceControllerImpl$RemoveChildrenTask instrumentation.target.classloader=ModuleClassLoader for Module "org.jboss.msc" version 1.4.13.Final from local module loader @3c87e6b7 (finder: local module finder @77bbadc (roots: /tmp/wildfly-bootable-server1955166504740324169/modules,/tmp/wildfly-bootable-server1955166504740324169/modules/system/layers/base))
time-service-time-1  | 13:12:12,170 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([("deployment" => "time.ear")]) - failure description: {
time-service-time-1  |     "WFLYCTL0080: Failed services" => {"jboss.deployment.subunit.\"time.ear\".\"time-cc.war\".undertow-deployment" => "java.lang.RuntimeException: java.lang.IllegalStateException: RESTEASY013005: Could not convert 'datadog.trace.instrumentation.springweb.OrderedServletPathRequestFilter' to a class.
time-service-time-1  |     Caused by: java.lang.RuntimeException: java.lang.IllegalStateException: RESTEASY013005: Could not convert 'datadog.trace.instrumentation.springweb.OrderedServletPathRequestFilter' to a class.
time-service-time-1  |     Caused by: java.lang.IllegalStateException: RESTEASY013005: Could not convert 'datadog.trace.instrumentation.springweb.OrderedServletPathRequestFilter' to a class.
time-service-time-1  |     Caused by: java.lang.ClassNotFoundException: datadog.trace.instrumentation.springweb.OrderedServletPathRequestFilter from [Module \"deployment.time.ear.time-cc.war\" from Service Module Loader]"},
time-service-time-1  |     "WFLYCTL0412: Required services that are not installed:" => [
time-service-time-1  |         "jboss.deployment.subunit.\"time.ear\".\"time-api.war\".deploymentCompleteService",
time-service-time-1  |         "jboss.deployment.subunit.\"time.ear\".\"time-web.war\".deploymentCompleteService"
time-service-time-1  |     ],
time-service-time-1  |     "WFLYCTL0180: Services with missing/unavailable dependencies" => [
time-service-time-1  |         "jboss.deployment.subunit.\"time.ear\".\"time-api.war\".INSTALL is missing [jboss.deployment.subunit.\"time.ear\".\"time-web.war\".deploymentCompleteService]",
time-service-time-1  |         "jboss.deployment.unit.\"time.ear\".deploymentCompleteService is missing [jboss.deployment.subunit.\"time.ear\".\"time-web.war\".deploymentCompleteService, jboss.deployment.subunit.\"time.ear\".\"time-api.war\".deploymentCompleteService]"
time-service-time-1  |     ]
time-service-time-1  | }
time-service-time-1  | 13:12:12,174 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0010: Deployed "time.ear" (runtime-name : "time.ear")
time-service-time-1  | 13:12:12,186 INFO  [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report
time-service-time-1  | WFLYCTL0184:    New missing/unsatisfied dependencies:
time-service-time-1  |       service jboss.deployment.subunit."time.ear"."time-api.war".deploymentCompleteService (missing) dependents: [service jboss.deployment.unit."time.ear".deploymentCompleteService] 
time-service-time-1  |       service jboss.deployment.subunit."time.ear"."time-web.war".deploymentCompleteService (missing) dependents: [service jboss.deployment.subunit."time.ear"."time-api.war".INSTALL, service jboss.deployment.unit."time.ear".deploymentCompleteService] 
time-service-time-1  | WFLYCTL0186:   Services which failed to start:      service jboss.deployment.subunit."time.ear"."time-cc.war".undertow-deployment: java.lang.RuntimeException: java.lang.IllegalStateException: RESTEASY013005: Could not convert 'datadog.trace.instrumentation.springweb.OrderedServletPathRequestFilter' to a class.
time-service-time-1  | WFLYCTL0448: 4 additional services are down due to their dependencies being missing or failed

@mcculls
Copy link
Contributor

mcculls commented Mar 16, 2023

@vallihalldors thanks - I dug into the org.jboss.resteasy.plugins.spring.SpringBeanProcessor code and it is bypassing the Spring bean factory to try and peek into the filter class without triggering the usual resolution (it also makes some assumptions about class-loading.)

We'll need to add specific instrumentation for SpringBeanProcessor to help it find the right classes.

@mcculls
Copy link
Contributor

mcculls commented Mar 20, 2023

Hi @vallihalldors - I've updated the SpringBeanProcessor instrumentation in #4921 which will be in the next release, 1.11.0

Until then it is also available in this snapshot of the Java tracer: https://oss.sonatype.org/content/repositories/snapshots/com/datadoghq/dd-java-agent/1.11.0-SNAPSHOT/dd-java-agent-1.11.0-20230320.182318-87.jar

@bantonsson bantonsson modified the milestone: 1.11.0 Mar 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants