[Bug] [Backend] Resource Center can not work

[jieguangzhou]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

When I configure Resource Center, I can't start DolphinScheduler.

[ERROR] 2022-09-22 06:51:08.089 +0000 org.springframework.boot.SpringApplication:[824] - Application run failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'workerServer': Unsatisfied dependency expressed through field 'workerRpcServer'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'workerRpcServer': Unsatisfied dependency expressed through field 'taskDispatchProcessor'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'taskDispatchProcessor': Unsatisfied dependency expressed through field 'storageOperate'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'storageOperate' defined in class path resource [org/apache/dolphinscheduler/common/config/StoreConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.dolphinscheduler.common.storage.StorageOperate]: Factory method 'storageOperate' threw exception; nested exception is com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:659)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:639)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1431)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:953)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:147)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:734)
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:408)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:308)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295)
        at org.apache.dolphinscheduler.StandaloneServer.main(StandaloneServer.java:45)
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'workerRpcServer': Unsatisfied dependency expressed through field 'taskDispatchProcessor'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'taskDispatchProcessor': Unsatisfied dependency expressed through field 'storageOperate'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'storageOperate' defined in class path resource [org/apache/dolphinscheduler/common/config/StoreConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.dolphinscheduler.common.storage.StorageOperate]: Factory method 'storageOperate' threw exception; nested exception is com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:659)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:639)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1431)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1389)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1309)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:656)
        ... 20 common frames omitted
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'taskDispatchProcessor': Unsatisfied dependency expressed through field 'storageOperate'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'storageOperate' defined in class path resource [org/apache/dolphinscheduler/common/config/StoreConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.dolphinscheduler.common.storage.StorageOperate]: Factory method 'storageOperate' threw exception; nested exception is com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:659)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:639)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1431)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1389)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1309)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:656)
        ... 34 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'storageOperate' defined in class path resource [org/apache/dolphinscheduler/common/config/StoreConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.dolphinscheduler.common.storage.StorageOperate]: Factory method 'storageOperate' threw exception; nested exception is com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:658)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:486)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1389)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1309)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:656)
        ... 48 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.dolphinscheduler.common.storage.StorageOperate]: Factory method 'storageOperate' threw exception; nested exception is com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)
        ... 61 common frames omitted
Caused by: com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2053)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3966)
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3989)
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4950)
        at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4956)
        at org.apache.dolphinscheduler.common.utils.HadoopUtils.getInstance(HadoopUtils.java:104)
        at org.apache.dolphinscheduler.common.config.StoreConfiguration.storageOperate(StoreConfiguration.java:47)
        at org.apache.dolphinscheduler.common.config.StoreConfiguration$$EnhancerBySpringCGLIB$$334a9eba.CGLIB$storageOperate$0(<generated>)
        at org.apache.dolphinscheduler.common.config.StoreConfiguration$$EnhancerBySpringCGLIB$$334a9eba$$FastClassBySpringCGLIB$$1238a78.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
        at org.apache.dolphinscheduler.common.config.StoreConfiguration$$EnhancerBySpringCGLIB$$334a9eba.storageOperate(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
        ... 62 common frames omitted
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder
        at org.apache.hadoop.fs.FsTracer.get(FsTracer.java:42)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3370)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:125)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3424)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3392)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:485)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:233)
        at org.apache.dolphinscheduler.common.utils.HadoopUtils.lambda$init$1(HadoopUtils.java:159)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
        at org.apache.dolphinscheduler.common.utils.HadoopUtils.init(HadoopUtils.java:158)
        at org.apache.dolphinscheduler.common.utils.HadoopUtils.<init>(HadoopUtils.java:99)
        at org.apache.dolphinscheduler.common.utils.HadoopUtils.<init>(HadoopUtils.java:70)
        at org.apache.dolphinscheduler.common.utils.HadoopUtils$1.load(HadoopUtils.java:88)
        at org.apache.dolphinscheduler.common.utils.HadoopUtils$1.load(HadoopUtils.java:84)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3533)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2282)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049)
        ... 78 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.apache.htrace.core.Tracer$Builder
        at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 98 common frames omitted

What you expected to happen

empty

How to reproduce

I build the package from 3.1.0-prepare, work fine in 3.0.0

Anything else

No response

Version

dev

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.

• In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
• If you haven't received a reply for a long time, you can join our slack and send your question to channel #troubleshooting

[boy-xiaozhang]

Your mistake is because you applied HDFS as the base of the resource center, but your resource.hdfs.fs.defaultFs configuration is wrong.

[jieguangzhou]

Your mistake is because you applied HDFS as the base of the resource center, but your resource.hdfs.fs.defaultFs configuration is wrong.

I want to use Local File Resource.

I thinks we miss something

com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder

[boy-xiaozhang]

Which version do you use

[jieguangzhou]

Which version do you use

3.1.0

[boy-xiaozhang]

Which version do you use

3.1.0

Can you tell me about your Hadoop version? According to your configuration, hdfs is the bottom socket of the resource center.

[boy-xiaozhang]

Which version do you use

3.1.0

And the resource center does not support local

[jieguangzhou]

Which version do you use

3.1.0

Can you tell me about your Hadoop version? According to your configuration, hdfs is the bottom socket of the resource center.

I didn't install Hadoop, but ds support local, it works fine in 3.0.0

https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/guide/resource/configuration.html

Local File Resource Configuration

[boy-xiaozhang]

Which version do you use

3.1.0

Can you tell me about your Hadoop version? According to your configuration, hdfs is the bottom socket of the resource center.

I didn't install Hadoop, but ds support local, it works fine in 3.0.0

https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/guide/resource/configuration.html

Local File Resource Configuration

Misread the code version,This mistake of yours is more like dependence not in the classpath.s there hadoop dependency under the dir