[Question] Accessing S3 bucket from within the SPA? #66

lawrence-c · 2020-06-18T13:16:32Z

Hi,

Sorry if this is slightly off topic, but I was wondering if it is possible to access the private S3 bucket hosting the SPA, with a jquery .get function from within the SPA?

Background:

I'm trying to setup authentication to access a private S3 bucket, and using this open source repository (https://github.com/rufuspollock/s3-bucket-listing) to list all items within the S3 bucket once authenticated. (I have also opened a ticket there Investigating usage with authentication (Cognito) and Cloudfront rufuspollock/s3-bucket-listing#101, where I go through some of the steps I've done).
I've run the cloudformation template, and replaced the react SPA, with the index.html mentioned in the above ticket, and logging in works great, and I can nearly see the entire index.html page, but it doesn't display anything in the S3 bucket itself.

I have some theories about this, but wasn't sure who to ask, so hoping someone could point me in the right direction.

In the list.js, it uses a $.get(s3_rest_url), however this returns an empty string (from what I can tell), and I think this is due to the list.js not passing in the authentication tokens required to access the S3 bucket (in the same way the user needed to sign in to access the index.html).

If this is the case, would a solution be to download the cookies from the browser post logging by the user, in the list.js file and pass one of the auth tokens in the $.get(s3_rest_url) so that it can access the S3 bucket?

Please let me know if you need any more information, and any advice would be greatly appreciated!

Thank you.

The text was updated successfully, but these errors were encountered:

ottokruse · 2020-06-18T14:32:21Z

Hi @lawrence-c

Access a private S3 bucket? Like the private S3 bucket from the solution here? That should work seamless. For other buckets, you could fromt them with this CloudFront distribution too and thus Lambda@Edge protect them. But tthat bucket needs to add the CloudFront OAI from this solution to its bucket policy, so that CloudFront can read the items.

From the looks of it rufuspollock/s3-bucket-listing is coded against being run as S3 website (not CloudFront) which is another ball game.

lawrence-c · 2020-06-18T14:46:51Z

Hi @ottokruse,

Thank you for the response!

So the cloudformation template sets up the correct permissions to access the S3 bucket, namely:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity <cloudfront_id>"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<bucket_name>/*"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity <cloudfront_id>"
            },
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::<bucket_name>"
        }
    ]
}

And, I am able to access the files within the bucket when I go to something like: https://<cloudfront_url>.cloudfront.net/test/hello-world.txt, but it's just that the index.html file doesn't seem able to list the files out, so I assumed this could be an authentication with jquery/javascript issue in the list.js file.

Should the solution be using the S3 bucket in website mode?

lawrence-c · 2020-06-18T15:13:09Z

I suppose if this is useful, these are the steps I've done so far:

Run the cloudformation template provided from this repository, however removing the HTTPHeaders for debugging reasons.
With the newly created S3 bucket, I removed the react application that was in there, and placed in the index.html and list.js from the s3-bucket-listing repository, however edited the index.html to be:

<!DOCTYPE html>
<html>
<head>
  <title>S3 Bucket Listing Generator</title>
</head>
<body>
  <div id="navigation"></div>
  <div id="listing"></div>

<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/3.1.1/jquery.min.js"></script>
<script type="text/javascript">
  var BUCKET_URL = 'https://<cloudfront_url>.cloudfront.net';
</script>
<script src="list.js"></script>

</body>
</html>

I also added in a hello-world.txt file to the bucket for test purposes.
Navigating to the https://<cloudfront_url>.cloudfront.net works fine, and I can login correctly with the details provided. The index.html just doesn't seem to be able to get the list of files in the S3 bucket correctly however.

rpattcorner · 2020-06-18T18:46:44Z

I think the issue is that accessing a file in an app or the app itself, e.g. index.html is an entirely different matter than having the app itself access a file in the bucket.

In the first case, the security principal is a browser attempting to read a file in a bucket, and the cloudfront is all set up with permissions for the bucket; the principal authenticates through a Cognito user pool, gets directed thru CloudFront to the bucket file requested and, as you point out, all is golden.

In the second case it's an entirely different permissions landscape. The security principal is code inside the SPA or regular app itself (not the currently accessing user from the user pool), and the code has no permissions whatsoever to reach outside the app and do anything, including read from the bucket it is running in. By the way, that presents an interesting bootstrapping problem.

Any permissions CloudFront had via OAI to let a browser read an app file do not transfer to the app code's own permissions.

There are some insecure ways to solve this; one more secure possibility is to give the app itself the desired role using a mechanism like a Cognito identity pool.

lawrence-c · 2020-06-19T13:11:01Z

Hi @rpattcorner

Thank you for the detailed response and explanation of the problem I'm having.

Can you please explain how you'd go about giving the app the necessary role using Cognito identity pool within the index.html file? I suppose the idea being that once the user has authenticated, the index.html page already has the correct permissions to access the bucket and doesn't need additional authentication?

I'm not too sure how to go about this as this is all quite new to me.

rpattcorner · 2020-06-19T14:18:20Z

An index.html might be part of an app, but it's just HTML, not executable code. You'd be doing the identity assumption in the javascript code components in the REACT or Angular app.

This AWS Doc gives a decent overview of the bits and pieces. The javascript section (scroll down) in this article was the best quick example I could find.

You'll need to google around for more examples though. AWS seems very big on promoting their Amplify framework for this purpose, but the projects I work on access the identity pool directly for authorization without Amplify.

A very quick look turned up the very last paragraph in this article and its associated repo, which illustrates in a cursory way how you'd bring an identity pool into an app that is already using a user pool. It's a pre-amplify example, or you could bite the bullet and go down the AWS-favored amplify path, which seems to be better documented.

lawrence-c · 2020-06-19T19:31:09Z

Thanks @rpattcorner for the links and explanations. I'll have a go looking through them and see if I can get something together.

Thank you!

wesleywh · 2020-09-11T21:10:59Z

@lawrence-c Actually @sathed came up with this one because we just had a really similar issue where we're using mkdocs. While we could authenticate and login to the index page fine (even subsequent html pages if you had the direct link). None of the links worked and the search bar that searched for pages didn't work. We actually fixed this by deleting the Content Security Policy that the lambdas use.

You can change this policy to your needs in the template. You will find a section in the template that looks like this. Modify it to your needs to allow urls you will be using.

...
...
HttpHeaders:
    Type: String
    Description: The HTTP headers to set on all responses from CloudFront. To be provided as a JSON object
    Default: |-
      {
        "Content-Security-Policy": "default-src 'none'; img-src 'self'; script-src 'self' https://code.jquery.com https://stackpath.bootstrapcdn.com; style-src 'self' 'unsafe-inline' https://stackpath.bootstrapcdn.com; object-src 'none'; connect-src 'self' https://*.amazonaws.com https://*.amazoncognito.com",
        ...
        ...
      }
...
...

Be sure to also update the version parameter of the lambda's so it will redeploy them. Since you updated the version of the lambdas be sure to update that version reference in your CloudFront behaviors as well. (We were doing it manually so those behaviors were not set by this template). Finally you can verify that the content policy is in your deployed lambdas or not by opening up the lambda and looking at configuration.json

Now with all of this said, I find it a little silly to have the Content-Security-Policy at all since there is authorization in front of this anyway. Anyone scanning this will get an authorization requirement sent back to them anyway. So you can just delete this header altogether and now you can list pages from other pages that live in the same s3 bucket (or other buckets) just fine.

melnur · 2021-11-18T11:58:04Z

@lawrence-c I'm also trying to implement this with https://github.com/rufuspollock/s3-bucket-listing but unfortunately I'm unable to see the bucket contents. Were you able to make this work?
I also use have " var BUCKET_URL = 'https://<cloudfront_url>.cloudfront.net';" in my index.html and disabled "Content-Security-Policy" as @wesleywh suggested. Would appreciate if you share your solution. Thanks

ottokruse · 2021-11-22T15:39:08Z

This really is a question to the author of https://github.com/rufuspollock/s3-bucket-listing: to support S3 bucket access via CloudFront, instead of to S3 directly.

Accessing an S3 bucket via CloudFront is a whole different thing than accessing an S3 bucket directly. CloudFront in front of S3 is supposed to abstract S3, so that the client does not even know the origin is S3.

A call like Bucket.ListObjectsV2 can be done directly against the S3 bucket, but not against CloudFront in front of S3.

@rpattcorner is right that what you need to make direct S3 access work with the solution here, is a Cognito Identity Pool. With that you need to create a web app that uses the JWTs that this solution places in the cookies, and trades them in for AWS credentials using the Cognito Identity Pool. With those credentials you can then use eg the s3-bucket-listing solution you mention.

Since this question comes by more often, it would be great to document in this repo how to setup an identity pool, and do direct bucket access. Maybe even add a parameter "CreateIdentityPool" to make it more easy. That's a fair amount of work though.

ottokruse added the question Further information is requested label Nov 22, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Question] Accessing S3 bucket from within the SPA? #66

[Question] Accessing S3 bucket from within the SPA? #66

lawrence-c commented Jun 18, 2020

ottokruse commented Jun 18, 2020

lawrence-c commented Jun 18, 2020

lawrence-c commented Jun 18, 2020

rpattcorner commented Jun 18, 2020

lawrence-c commented Jun 19, 2020

rpattcorner commented Jun 19, 2020 •

edited

lawrence-c commented Jun 19, 2020

wesleywh commented Sep 11, 2020

melnur commented Nov 18, 2021

ottokruse commented Nov 22, 2021 •

edited

[Question] Accessing S3 bucket from within the SPA? #66

[Question] Accessing S3 bucket from within the SPA? #66

Comments

lawrence-c commented Jun 18, 2020

ottokruse commented Jun 18, 2020

lawrence-c commented Jun 18, 2020

lawrence-c commented Jun 18, 2020

rpattcorner commented Jun 18, 2020

lawrence-c commented Jun 19, 2020

rpattcorner commented Jun 19, 2020 • edited

lawrence-c commented Jun 19, 2020

wesleywh commented Sep 11, 2020

melnur commented Nov 18, 2021

ottokruse commented Nov 22, 2021 • edited

rpattcorner commented Jun 19, 2020 •

edited

ottokruse commented Nov 22, 2021 •

edited