Asp.Net Core Not picking up profile from appsettings.json

[ralsu091]

Hello,

Following: https://aws.amazon.com/blogs/developer/configuring-aws-sdk-with-net-core/
I have setup appsettings.json as the following:

{
  "AWS": {
    "Profile": "profileName",
    "Region": "us-east-1"
  }
}

My Startup.cs has the following:

public Startup(IHostingEnvironment env)
        {
            var builder = new ConfigurationBuilder()
                .SetBasePath(env.ContentRootPath)
                .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
                .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
                .AddEnvironmentVariables();


            Configuration = builder.Build();

        }

public void ConfigureServices(IServiceCollection services)
        {
            var mapper = ConfigureAutoMapper.Configure();
            services.AddOptions();
            services.Configure<MongoOptions>(Configuration);
            services.AddSingleton<IHttpContextAccessor, HttpContextAccessor>();
            services.AddSingleton<IRepository, Repository>();
            services.AddSingleton(mapper);
            services.AddMvc();
            services.AddDefaultAWSOptions(Configuration.GetAWSOptions());
            services.AddAWSService<IAmazonSimpleEmailService>();

        }

When I deploy the website to a Windows Server 2008 R2, the website crashes on startup and the logs show the following:

Application startup exception: Amazon.Runtime.AmazonServiceException: Unable to find credentials

Exception 1 of 3:
System.ArgumentException: App.config does not contain credentials information. Either add the AWSAccessKey and AWSSecretKey properties or the AWSProfileName property.
   at Amazon.Runtime.StoredProfileAWSCredentials..ctor(String profileName, String profilesLocation)
   at Amazon.Runtime.FallbackCredentialsFactory.<>c.<Reset>b__6_0()
   at Amazon.Runtime.FallbackCredentialsFactory.GetCredentials(Boolean fallbackToAnonymous)

Exception 2 of 3:
System.InvalidOperationException: The environment variables AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY/AWS_SESSION_TOKEN were not set with AWS credentials.
   at Amazon.Runtime.EnvironmentVariablesAWSCredentials.FetchAWSCredentials()
   at Amazon.Runtime.FallbackCredentialsFactory.<>c.<Reset>b__6_1()
   at Amazon.Runtime.FallbackCredentialsFactory.GetCredentials(Boolean fallbackToAnonymous)

Exception 3 of 3:
System.AggregateException: AsyncHelpers.Run method threw an exception. (An error occurred while sending the request.) ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: The operation timed out
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.WinHttpHandler.<StartRequest>d__105.MoveNext()
   --- End of inner exception stack trace ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpClient.<GetContentAsync>d__32`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.Util.AsyncHelpers.<>c__DisplayClass1_0`1.<<RunSync>b__0>d.MoveNext()
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.Util.AsyncHelpers.ExclusiveSynchronizationContext.BeginMessageLoop()
   at Amazon.Runtime.Internal.Util.AsyncHelpers.RunSync[T](Func`1 task)
   at Amazon.Util.AWSSDKUtils.DownloadStringContent(Uri uri, TimeSpan timeout)
   at Amazon.Runtime.URIBasedRefreshingCredentialHelper.GetContents(Uri uri)
   at Amazon.Runtime.InstanceProfileAWSCredentials.<GetAvailableRoles>d__10.MoveNext()
   at Amazon.Runtime.InstanceProfileAWSCredentials.GetFirstRole()
   at Amazon.Runtime.FallbackCredentialsFactory.ECSEC2CredentialsWrapper()
   at Amazon.Runtime.FallbackCredentialsFactory.GetCredentials(Boolean fallbackToAnonymous)
---> (Inner Exception #0) System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: The operation timed out
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.WinHttpHandler.<StartRequest>d__105.MoveNext()
   --- End of inner exception stack trace ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpClient.<GetContentAsync>d__32`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.Util.AsyncHelpers.<>c__DisplayClass1_0`1.<<RunSync>b__0>d.MoveNext()<---



   at Amazon.Runtime.FallbackCredentialsFactory.GetCredentials(Boolean fallbackToAnonymous)
   at Amazon.Extensions.NETCore.Setup.ClientFactory.CreateCredentials(AWSOptions options)
   at Amazon.Extensions.NETCore.Setup.ClientFactory.CreateServiceClient(Type serviceInterfaceType, AWSOptions options)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite scopedCallSite, ServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, ServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite scopedCallSite, ServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.<>c__DisplayClass16_0.<RealizeService>b__0(ServiceProvider provider)
   at Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider)
   at Microsoft.Extensions.Internal.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters)
   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass3_0.<UseMiddleware>b__0(RequestDelegate next)
   at Microsoft.AspNetCore.Builder.Internal.ApplicationBuilder.Build()
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
crit: Microsoft.AspNetCore.Hosting.Internal.WebHost[6]
      Application startup exception
Amazon.Runtime.AmazonServiceException: Unable to find credentials

Exception 1 of 3:
System.ArgumentException: App.config does not contain credentials information. Either add the AWSAccessKey and AWSSecretKey properties or the AWSProfileName property.
   at Amazon.Runtime.StoredProfileAWSCredentials..ctor(String profileName, String profilesLocation)
   at Amazon.Runtime.FallbackCredentialsFactory.<>c.<Reset>b__6_0()
   at Amazon.Runtime.FallbackCredentialsFactory.GetCredentials(Boolean fallbackToAnonymous)

Exception 2 of 3:
System.InvalidOperationException: The environment variables AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY/AWS_SESSION_TOKEN were not set with AWS credentials.
   at Amazon.Runtime.EnvironmentVariablesAWSCredentials.FetchAWSCredentials()
   at Amazon.Runtime.FallbackCredentialsFactory.<>c.<Reset>b__6_1()
   at Amazon.Runtime.FallbackCredentialsFactory.GetCredentials(Boolean fallbackToAnonymous)

Exception 3 of 3:
System.AggregateException: AsyncHelpers.Run method threw an exception. (An error occurred while sending the request.) ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: The operation timed out
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.WinHttpHandler.<StartRequest>d__105.MoveNext()
   --- End of inner exception stack trace ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpClient.<GetContentAsync>d__32`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.Util.AsyncHelpers.<>c__DisplayClass1_0`1.<<RunSync>b__0>d.MoveNext()
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.Util.AsyncHelpers.ExclusiveSynchronizationContext.BeginMessageLoop()
   at Amazon.Runtime.Internal.Util.AsyncHelpers.RunSync[T](Func`1 task)
   at Amazon.Util.AWSSDKUtils.DownloadStringContent(Uri uri, TimeSpan timeout)
   at Amazon.Runtime.URIBasedRefreshingCredentialHelper.GetContents(Uri uri)
   at Amazon.Runtime.InstanceProfileAWSCredentials.<GetAvailableRoles>d__10.MoveNext()
   at Amazon.Runtime.InstanceProfileAWSCredentials.GetFirstRole()
   at Amazon.Runtime.FallbackCredentialsFactory.ECSEC2CredentialsWrapper()
   at Amazon.Runtime.FallbackCredentialsFactory.GetCredentials(Boolean fallbackToAnonymous)
---> (Inner Exception #0) System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: The operation timed out
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.WinHttpHandler.<StartRequest>d__105.MoveNext()
   --- End of inner exception stack trace ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpClient.<GetContentAsync>d__32`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.Util.AsyncHelpers.<>c__DisplayClass1_0`1.<<RunSync>b__0>d.MoveNext()<---



   at Amazon.Runtime.FallbackCredentialsFactory.GetCredentials(Boolean fallbackToAnonymous)
   at Amazon.Extensions.NETCore.Setup.ClientFactory.CreateCredentials(AWSOptions options)
   at Amazon.Extensions.NETCore.Setup.ClientFactory.CreateServiceClient(Type serviceInterfaceType, AWSOptions options)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite scopedCallSite, ServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, ServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite scopedCallSite, ServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.<>c__DisplayClass16_0.<RealizeService>b__0(ServiceProvider provider)
   at Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider)
   at Microsoft.Extensions.Internal.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters)
   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass3_0.<UseMiddleware>b__0(RequestDelegate next)
   at Microsoft.AspNetCore.Builder.Internal.ApplicationBuilder.Build()
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
Hosting environment: Production
Content root path: C:\Websites\Website
Now listening on: http://localhost:23405
Application started. Press Ctrl+C to shut down.
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
      Request starting HTTP/1.1 POST http://hostname/token application/x-www-form-urlencoded 32
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 266.533ms 500 text/html; charset=utf-8

Am I missing something? Is this the correct way to use AWS profiles with dotnet core apps?

I appreciate your help.

Thanks

[imkheong]

Based on the observation, it appears all the mechanism used to obtain AWS access key id and secret access key have failed.

Exception 1 of 3 reads from the user profile store, which is a folder in the user's folder .aws sub folder. It can be created using AWS CLI command by via 'aws configure'.

Exception 2 of 3 shows it fail to read any environment variable. It is possible to set them in code the following way:

Environment.SetEnvironmentVariable("AWS_ACCESS_KEY_ID", "xxx");
Environment.SetEnvironmentVariable("AWS_SECRET_ACCESS_KEY", "yyy");
Environment.SetEnvironmentVariable("AWS_REGION", "us-east-1");

If user profile store is specified, it will override environment variable.

For a quick and dirty fix, I suggest you put the environment variable in code to make it work. While a long term solution would be to put it in profile which will prevent you from checking in access key id and secret access key to your code repository.

[ralsu091]

@imkheong Thanks for the suggestion. I opted to keep using profile stores. I needed to modify the Application pool to load the user profile in IIS. This allowed the retrieval of the default profile that was set using AWS CLI.

Thanks for the help!

[Shawski]

I know this is closed, but I ran into this and was completely frustrated for a while trying to get everything to work. I found that adding the ProfilesLocation setting to the appconfig.json file worked for me.

"AWS": {
		"Profile": "default",
		"ProfilesLocation": "C:\\code\\myapp\\awscredentials",
		"Region": "us-east-1",

Then create the awscredentials file like so:

[default]
aws_access_key_id = <access-key>
aws_secret_access_key = <secret-key>

[development]
aws_access_key_id = <access-key>
aws_secret_access_key = <secret-key>

You may name the file whatever you want. Just make sure you reference it using ProfilesLocation.

If, instead, you want to use SetEnvironment like @imkheong pointed out, I suggest putting the keys in the appsettings.json file and referencing them like this in ConfigureServices before adding any AWS services:

Environment.SetEnvironmentVariable("AWS_ACCESS_KEY_ID", Configuration["AWS:AccessKey"])
Environment.SetEnvironmentVariable("AWS_SECRET_ACCESS_KEY", Configuration["AWS:SecretKey"])
Environment.SetEnvironmentVariable("AWS_REGION", Configuration["AWS:Region"])

services.AddDefaultAWSOptions(Configuration.GetAWSOptions());
services.AddAWSService<IAmazonSimpleEmailService>();

This is what the AWS section of appsettings.json would look like:

	"AWS": {
		"AccessKey": "abc",
		"SecretKey": "xyz",
		"Region": "us-east-1"
	},

[davismj]

This issue ought to be reopened. I still am not sure why the appsettings.json configuration is getting completely ignored. If I need to write it somewhere else using aws configure, why do I even need the appsettings.json keys?

We're using a continuous deployment model, and the settings were in the appsettings.json file, but I was getting the same error.

[normj]

It is not that the appsettings.json file is ignored. The tricky part is it is referring to a profile that isn't found when run under a system user like IIS is. By default system users don't have a user profile loaded so the SDK can't find the credentials for the profile. That is why @Shawski was having the appsettings.json point to a credentials file outside of the current user's directory.

[davismj]

I took the same keys in my appsettings.json, which threw the above error, and added them with the appropriate casing to my environment variables and everything worked. Either some or all of the appsettings.json keys are getting ignored...

[Kralizek]

In my experience the whole appsettings.json is ignored. I would assume is a problem between Windows and Linux new line

[davismj]

@normj Should I create a new issue or can we reopen this one?

[normj]

Sounds like it is worth opening a new issue. Try to add as much information as you can about how your application is running so I can try understand what is different from your environment to how I have tested.

[davismj]

@normj I'm not sure what other information to add. I have the following in my appsettings.json.

"AWS": {
    "AccessKey": "abc",
    "SecretKey": "xyz",
    "Region": "us-east-1"
  }

Even so, I'm getting a "unable to find a default profile" error.

Amazon.Runtime.AmazonServiceException: Unable to find credentials
Exception 1 of 3:
Amazon.Runtime.AmazonClientException: Unable to find a default profile in CredentialProfileStoreChain.
Exception 2 of 3:
System.InvalidOperationException: The environment variables AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY/AWS_SESSION_TOKEN were not set with AWS credentials.

Edit: I see the problem. There's no logic for picking up an access key or secret key from the appsettings.json. I guess the logic there is that it's not a good idea to commit your secrets to a config file, and it is a much better idea to use environment variables. However, the enviornment-scoped appsettings.json is how I avoid setting global environment variables on my development machine that don't get pushed to production.

I'm going to submit a PR that reads these keys from appsettings.json. I think it will cure a few headaches for new users.