Genie 3.0.1 - Exception: No bean named 'file.system.s3' is defined

[jelez]

Genie installed from jar: genie-app-3.0.1.jar

Starting genie as:
$ export JAVA_OPTS="-Djava.io.tmpdir=/opt/genie/tmp -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true"
$ exec java ${JAVA_OPTS} -jar /opt/genie/lib/genie-app-3.0.1.jar --spring.config.location=/opt/genie/config/

In /opt/genie/config/ I have application-dev.yml application-prod.yml application-s3.yml application.yml

Cluster YAML config looks like:

name: spark cluster
user: hadoop
version: 2.7.3
status: UP
tags:

• type:yarn
• ver:2.7.3
• emr.cluster.release:emr-5.1.0
  configs:
• s3://mybucket/genie/clusters/j-XXXXXXXXXX/conf/hadoop/core-site.xml
• s3://mybucket/genie/clusters/j-XXXXXXXXXX/conf/hadoop/mapred-site.xml
• s3://mybucket/genie/clusters/j-XXXXXXXXXX/conf/hadoop/yarn-site.xml

When i run a job I get exception. Could you point me to what I could be doing wrong? Do you have an example config that works?

Exception details:

2017-03-04 02:57:02.176 ERROR 1459 --- [taskScheduler-1] com.netflix.genie.core.jobs.JobLauncher : Unable to submit job due to exception: Failed getting the file s3://mybucket/genie/clusters/j-XXXXXXXXXX/conf/hadoop/yarn-site.xml

com.netflix.genie.common.exceptions.GenieServerException: Failed getting the file s3://mybucket/genie/clusters/j-XXXXXXXXXX/conf/hadoop/yarn-site.xml
at com.netflix.genie.core.services.impl.CacheGenieFileTransferService.getFile(CacheGenieFileTransferService.java:117) ~[genie-core-3.0.1.jar!/:3.0.1]
at com.netflix.genie.core.jobs.workflow.impl.ClusterTask.executeTask(ClusterTask.java:131) ~[genie-core-3.0.1.jar!/:3.0.1]
at com.netflix.genie.core.services.impl.LocalJobRunner.executeJob(LocalJobRunner.java:285) ~[genie-core-3.0.1.jar!/:3.0.1]
at com.netflix.genie.core.services.impl.LocalJobRunner.submitJob(LocalJobRunner.java:151) ~[genie-core-3.0.1.jar!/:3.0.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_121]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_121]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) ~[spring-aop-4.2.8.RELEASE.jar!/:4.2.8.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.2.8.RELEASE.jar!/:4.2.8.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.2.8.RELEASE.jar!/:4.2.8.RELEASE]
at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:139) ~[spring-context-4.2.8.RELEASE.jar!/:4.2.8.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.2.8.RELEASE.jar!/:4.2.8.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) ~[spring-aop-4.2.8.RELEASE.jar!/:4.2.8.RELEASE]
at com.sun.proxy.$Proxy138.submitJob(Unknown Source) ~[na:na]
at com.netflix.genie.core.jobs.JobLauncher.lambda$run$0(JobLauncher.java:99) ~[genie-core-3.0.1.jar!/:3.0.1]
at com.netflix.spectator.api.AbstractTimer.record(AbstractTimer.java:48) ~[spectator-api-0.30.0.jar!/:0.30.0]
at com.netflix.genie.core.jobs.JobLauncher.run(JobLauncher.java:96) ~[genie-core-3.0.1.jar!/:3.0.1]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-4.2.8.RELEASE.jar!/:4.2.8.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'file.system.s3' is defined
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-17.0.jar!/:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-17.0.jar!/:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-17.0.jar!/:na]
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-17.0.jar!/:na]
at com.netflix.genie.core.services.impl.CacheGenieFileTransferService.getFile(CacheGenieFileTransferService.java:100) ~[genie-core-3.0.1.jar!/:3.0.1]
... 25 common frames omitted
Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'file.system.s3' is defined
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:704) ~[spring-beans-4.2.8.RELEASE.jar!/:4.2.8.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:1175) ~[spring-beans-4.2.8.RELEASE.jar!/:4.2.8.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:284) ~[spring-beans-4.2.8.RELEASE.jar!/:4.2.8.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-4.2.8.RELEASE.jar!/:4.2.8.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:275) ~[spring-beans-4.2.8.RELEASE.jar!/:4.2.8.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-4.2.8.RELEASE.jar!/:4.2.8.RELEASE]
at org.springframework.beans.factory.config.ServiceLocatorFactoryBean$ServiceLocatorInvocationHandler.invokeServiceLocatorMethod(ServiceLocatorFactoryBean.java:373) ~[spring-beans-4.2.8.RELEASE.jar!/:4.2.8.RELEASE]
at org.springframework.beans.factory.config.ServiceLocatorFactoryBean$ServiceLocatorInvocationHandler.invoke(ServiceLocatorFactoryBean.java:363) ~[spring-beans-4.2.8.RELEASE.jar!/:4.2.8.RELEASE]
at com.sun.proxy.$Proxy136.get(Unknown Source) ~[na:na]
at com.netflix.genie.core.services.impl.GenieFileTransferService.getFileTransfer(GenieFileTransferService.java:96) ~[genie-core-3.0.1.jar!/:3.0.1]
at com.netflix.genie.core.services.impl.CacheGenieFileTransferService.loadFile(CacheGenieFileTransferService.java:153) ~[genie-core-3.0.1.jar!/:3.0.1]
at com.netflix.genie.core.services.impl.CacheGenieFileTransferService$1.load(CacheGenieFileTransferService.java:55) ~[genie-core-3.0.1.jar!/:3.0.1]
at com.netflix.genie.core.services.impl.CacheGenieFileTransferService$1.load(CacheGenieFileTransferService.java:51) ~[genie-core-3.0.1.jar!/:3.0.1]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-17.0.jar!/:na]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guava-17.0.jar!/:na]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) ~[guava-17.0.jar!/:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guava-17.0.jar!/:na]

[tgianos]

Hi @jelez,

There are a few things that could be going on. I saw your question in the google group as well so I'm just going to address it here.

Do you need full copies of all the properties files? They're already built into the released Jar file so you really should only override properties you need.

One thing that is definitely a problem is that you're not actually activating the profiles. Not sure how familiar you are with Spring but application.yml (resolved up the classpath) is the default and anything application-{profile}.yml needs to be activated using spring.profiles.active property.

You should try launching your app with spring.profiles.active=prod,s3 and see if that solves the problem as it should take all the properties in those two extra YAML files. Also we defaulted to having the jar itself, the local directory and then {user.home}/.genie/ as the config locations. It looks like you're trying to override that by setting spring.config.location yourself. I haven't tested that myself but if setting the active profiles doesn't solve your problem we can continue trying other options and I can try to reproduce myself.

There is the demo application if you want to see a working example (granted it doesn't have s3 setup). Let us know if this helps.

[jelez]

@tgianos I have taken {user.home}/.genie path. However, now I get
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: com.amazonaws.AmazonClientException: Unable to load AWS credentials from the /opt/genie/lib/AwsCredentials.properties file on the classpath

Its not working for me neither with absolute nor relative path to the creds file. I am not sure what I am misisng - do I need to set loader.path or some other property?

[tgianos]

@jelez try setting genie.aws.credentials.file property to have the file system location of your credential file.

We use instance roles internally so I haven't personally tried this. @amitsharma1981 implemented the S3 stuff and may have more insight to help you out with how he tested it.

[edrevo]

@jelez, were you able to fix this? it would be great to know how you were able to get S3 working, since i am running in the exact same pitfalls as you have

[tgianos]

@edrevo did trying the steps I outlined above fix the issue for you? @jelez closed this so I don't know if it was solved or not. If those steps still aren't working we'd have to look into it more.

[edrevo]

I was finally able to solve it, but it was hard to figure it out, so I'll outline how:

1. You need to activate the S3 profile, as you mentioned. Upon startup, Genie will display a message saying which profiles are active.
2. The file.system.s3 bean won't be able to be instantiated unless there are settings in the application-s3.yml file (which should be placed in ~/.genie/) and it contains either a genie.aws.credentials.role value or a genie.aws.credentials.file value. I wanted to use the file credentials, so that's what I'll outline here.
3. genie.aws.credentials.file needs to point to a file in the classpath which contains the aws access key and secret. The key here is knowing that Genie considers whatever you put in ~/.genie/lib as part of the classpath, so you can just drop the credentials file there.

[edrevo]

By the way, I tried writing my own Bean to use the DefaultAWSCredentialsProviderChain, which is what most people will need in my opinion, and I was unable to get Spring to load it. It would be nice if there was some documentation on how to extend Genie with new FileTransfer implementations or any other extension points that Genie offers.

Also, it would be nice if Genie had an option to use the DefaultAWSCredentialsProviderChain. The code could look something like this:

    @Bean
    @ConditionalOnProperty(value = "genie.aws.credentials.useDefault", havingValue = "true")
    public DefaultAWSCredentialsProviderChain awsCredentialsDefaultProvider() {
        return new DefaultAWSCredentialsProviderChain();
    }

If you'd like, I can open two new issues for this (the documentation and the DefaultAWSCredentialsProviderChain)

[tgianos]

@edrevo Thanks for the exploration. Since we use InstanceCredentials internally this code path hasn't been used much hence the lack of documentation. We should definitely add it to the documentation. We probably should have a default provider chain. We'd have to test it all out.

Pull requests are always welcome!

As for getting your beans to be picked up there's probably a missing set of documentation as well. I'd have to look into the spring documentation for how to add packages to the classpath scanning but if you put your @Configuration classes inside com.netflix.genie.* packages they should be found since the default classpath scanning is under that package.

[tgianos]

Also I think you should be able to change the location of ~/.genie/* for properties files by setting --spring.config.location when you start Genie.

[mprimi]

Here's a PR to document S3 configuration (thanks @edrevo!): #506

We're also internally tracking improvements to the AWS configurations and credentials.