Cannot upload image from AdminUI to S3 bucket using S3 File Adapter when Keystone app hosted in AWS

[ra-external]

We have a KeystoneJS app that is using the S3 File Adapter to upload images to an S3 bucket.

const imageFileAdapter = new S3Adapter({
	accessKeyId: S3_KEY, //S3_KEY,
	secretAccessKey: S3_SECRET,
	region: S3_REGION,
	bucket: S3_BUCKET,
	folder: S3_PATH,
	s3Options: {
		apiVersion: '2006-03-01'
	},
	uploadParams: ({ id }) => ({
		ACL: 'public-read',
		Metadata: {
			keystone_id: String(id)
		}
	})
});

This works fine from the local environment.

However it does not work when the app is hosted in our AWS dev environment. The code is deployed in a container in AWS ECS/Fargate.

My DevOp would like to see more logs about what's going on -- but even when we enable the Keystone logs we don't see anything relevant to the file upload through the file adapter here.

[chtonal]

Before you run your app in ‘AWS ECS/Fargate‘ be sure your environmental variables ‘ S3_KEY’, ‘ S3_SECRET’, ‘ S3_REGION’, ‘S3_BUCKET’ and ‘S3_PATH’ are properly defined. Because in most of the services of AWS, enviromental variables which is ‘.env’ in this case, needs to be defined by yourself on AWS console.

Consequently, you can run your app in your localhost without any problem, but if you run the app in AWS ECS/Fargate service you have to define .env (environmental variables) from AWS Console.

EDIT: Detailed information is available in AWS Fargate Documentation

[ra-external]

@chtonal --Thanks. All those variables are good in the AWS secrets manager and are being translated correctly to the deployed .env file

[chtonal]

@ra-external AWS Fargate does not load .env file from your app. Only variables that you can use comes from fargate console. In order to make sure your app is configured well, just log those variables, or create a test page to display them. If it is fine, then fine.

[ra-external]

@chtonal -- I'm just passing on what the DevOps said -- all the variables are in the 'Secrets Manager' -- I think he's retrieving them from there and on build/deploy writing out a new .env file for the app to read from. In any case, whatever he's doing is working for everything else in the app -- it's all built using Terraform, which is managing this.

[chtonal]

I'm just passing on what the DevOps said -- all the variables are in the 'Secrets Manager' -- I think he's retrieving them from there and on build/deploy writing out a new .env file for the app to read from. In any case, whatever he's doing is working for everything else in the app -- it's all built using Terraform, which is managing this.

😁 If you are not able to reach AWS console, the best way is to load those variables from string. Not secure but for your situation it is a must 😀

Just replace S3_ variables in your js file with the .env 😉

EDIT: If your devops give s3 read-write access right for fargate. Ask him for S3_ variables. In some cases you may use different environmental variable name (For S3_KEY => AWS_ACCESS_KEY_ID, for S3_SECRET => AWS_SECRET_ACCESS_KEY)

[CheikhSambaM]

Hi,

I'm the devops. We passed as env variables S3_KEY and S3_SECRET (and the others). They are stored ins secret manager and passed to the containers in Fargate as ENV Variable during the deployment. As it is working locally do we need to do something more on the Fargate ? Just using the S3_KEY and S3_SECRET should be enough no ? Is there a way to activate some verbose logs to see the call to S3 ?

Much thanks

[gautamsi]

are you assuming the environment variables as global variable or using something like this few lines before.

const process = require('process');
const { S3_KEY, S3_SECRET, S3_REGION, S3_BUCKET, S3_PATH } = process.env;

it is obvious that you would do that, just making sure this was the case.

[chtonal]

@gautamsi As a understand, they have some issue on AWS Services. AWS is completely different subject then keystonejs. They have huge amount of services and AWS Fargate is one of them. In order to clarify your current state i am going to explain a little about the procedures you have to follow;

1. In AWS IAM Console, give read-write permission to Fargate for S3.
2. In AWS Fargate Console, define environment variables (S3_KEY, S3_BUCKET etc.) for your app.
3. In AWS S3 Console, define cors configuration. If you already defined it is fine.
4. If you have any additional AWS service, your setup needs IAM permissions for each. These services are communicating with each other with IAM roles.
5. Make sure that environmental variables are passed keystonejs app correctly by logging out that variables by your preferred method (test page, console, or AWS CloudWatch).

These are the procedures you have to follow. In which step you have encountered with the problem?

[CheikhSambaM]

Hi @chtonal , @gautamsi

thank for your help. i did all the check listed by @chotnal and they were already OK. i finally found the issue.. tricky one..

for S3_PATH we had "/public/images" instead of "public/images". For some reason having "/" at the beginning of the path made the insert in S3 fail..

Removing it solved my issue.

thanks

[chtonal]

So, it is time to close this issue. @ra-external

[ra-external]

Thanks to all, @chtonal, @gautamsi, @CheikhSambaM! The issue was that we had removed the leading '/' a while ago in the .env (took a while to figure it out), which was why it worked locally, but that hadn't gotten translated to the S3 variables.