external services (S3 specifically) responds 403 Forbidden to VM hosted HTML #192

btbonval · 2013-12-06T21:11:56Z

A number of assets are coded to https://s3.amazonaws.com/fc_filepicker/. According to web console, Amazon is responding with 403 Forbidden to all of them, be they javascript, svg, or css.

I thought it might be a REFERER filter on S3. Andrew gave me the idea to hack the DNS a bit. I modified /etc/hosts on my host system (where I run my browser) so that karmanotes.org points to 127.0.0.1. That worked in the sense that my URL bar said karmanotes.org:6659 and loaded the VM server. However, Amazon still chucked 403 errors my way.

This is very perplexing. A number of the links can be loaded if copy/pasted into my browser, but when instructed to load the same links from HTML out of the VM, Amazon gives my browser 403.

The text was updated successfully, but these errors were encountered:

btbonval · 2013-12-06T21:15:52Z

Possible workaround: replace all https://s3.amazonaws.com/fc_filepicker/ references in code to /static/ and then have nginx statically serve the files using the location /static/ directive.

Possible problems with the workaround:

that might be a lot of work (finding all possible references and replacing them)
might not fix the problem ultimately if S3 hosts the uploaded files.

My understanding is that S3 only holds uploaded files temporarily for celery to snatch, and then celery does magic with Google Drive and drops files locally rather than on S3. If that is true, then the browser's problem with S3 will not effect uploaded files.

sethwoodworth · 2013-12-09T16:28:44Z

You will either need to login to S3 and find the setting that has allowable domains, and add the new domain:port. Or you need to disable the s3 staticfile management. It's not hard, just checkout the differences between production and development settings.

btbonval · 2013-12-09T21:04:41Z

The super good news is that the static link was programmed appropriately, with just a single config entry as far as I can tell:
https://github.com/FinalsClub/karmaworld/blob/master/karmaworld/settings/prod.py#L125

Actually there is a second instance in the secret file for static_s3.py, which is imported in the above file:
https://github.com/FinalsClub/karmaworld/blob/master/karmaworld/settings/prod.py#L19

That's the only place the static_s3 version of the URL is used, and it is completely ignored by line 125 noted above, which overrides the imported variable from secret.

btbonval · 2013-12-09T21:13:35Z

According to http://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html, the URL format for static website hosting is: <bucket-name>.s3-website-<AWS-region>.amazonaws.com.

We appear to be using the S3 RESTful API (http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html), bucketname.s3.amazonaws.com.

Even though we're using S3 static hosting at present, we aren't actually using the AWS S3 Static Website Hosting feature... Differences are listed as:
http://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteEndpoints.html#WebsiteRestEndpointDiff

None of that hints at why I can copy/paste an S3 static link and load it in my browser, but the same link returns 403 in the same browser when loaded from a web page hosted locally. Again, as far as I can tell, it all comes down to REFERER but I can find no settings in my AWS Management Console that cares about it.

btbonval · 2013-12-09T21:34:36Z

Following up from the previous comment, more info about REST API: http://docs.aws.amazon.com/AmazonS3/latest/API/APIRest.html

After toying around a bit, I found out each individual file has its own permissions. There are grantees on each bucket, each directory, and each file. It wasn't obvious how the grantees are defined, but there's a doc: http://docs.aws.amazon.com/AmazonS3/latest/dev/ACLOverview.html

"When using ACLs, a grantee can be an AWS account or one of the predefined Amazon S3 groups. However, the grantee cannot be an IAM user."

So the good news is that grantees are either everybody, all authenticated AWS users, or particular AWS users. There is no possible way for REFERER to matter at all here.

btbonval · 2013-12-09T21:42:08Z

Someone was having 403 forbidden on S3 and solved the problem, but two of the answers mentioned that Cloudfront CDN needs to be configured in certain ways:
http://stackoverflow.com/a/16339642/1867779

So here's a question: Does Cloudfront CDN host S3 transparently? If so, this black box is much harder to penetrate. It'd be super good to just hop on the group's AWS Management Console and poke around to see how exactly things are setup and whether anything needs to be changed.

The alternative is to stop using S3 for static hosting and have nginx hijack static hosting locally.

btbonval · 2013-12-10T02:09:03Z

re: workaround.

comment out everything about static_s3 from imports
comment out storages from the INSTALLED_APPS
change STATIC_URL to '/assets/'
comment out the entire storages section (save for part of INSTALLED_APPS and STATIC_URL)
add this to the nginx config:
location /assets/ { root /var/www/karmaworld/karmaworld/; }

page looks good on localhost and jquery/javascript is actually loading.

sethwoodworth · 2013-12-10T05:03:41Z

Did you try:
http://stackoverflow.com/questions/10854095/boto-exception-s3responseerror-s3responseerror-403-forbidden

It might be a permissions error on the root of the static directory on s3.

On Mon, Dec 9, 2013 at 9:09 PM, Bryan Bonvallet notifications@github.comwrote:

re: workaround.

comment out everything about static_s3 from imports

comment out storages from the INSTALLED_APPS

change STATIC_URL to '/assets/'

comment out the entire storages section (save for part of
INSTALLED_APPS and STATIC_URL)

add this to the nginx config: location /assets/ { root
/var/www/karmaworld/karmaworld/; }

page looks good on localhost and jquery/javascript is actually loading.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/192#issuecomment-30193617
.

btbonval · 2013-12-10T09:30:18Z

Bolo is related to Django access, but this problem is experienced directly
in the browser trying to read the link as opposed to Django doing it on the
backend.

Also ... no access to S3 perms :(
On Dec 10, 2013 12:03 AM, "Seth Woodworth" notifications@github.com wrote:

Did you try:

http://stackoverflow.com/questions/10854095/boto-exception-s3responseerror-s3responseerror-403-forbidden

It might be a permissions error on the root of the static directory on s3.

On Mon, Dec 9, 2013 at 9:09 PM, Bryan Bonvallet notifications@github.comwrote:

re: workaround.

comment out everything about static_s3 from imports

comment out storages from the INSTALLED_APPS

change STATIC_URL to '/assets/'

comment out the entire storages section (save for part of
INSTALLED_APPS and STATIC_URL)

add this to the nginx config: location /assets/ { root
/var/www/karmaworld/karmaworld/; }

page looks good on localhost and jquery/javascript is actually loading.

—
Reply to this email directly or view it on GitHub<
https://github.com/FinalsClub/karmaworld/issues/192#issuecomment-30193617>

.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/192#issuecomment-30199893
.

btbonval · 2013-12-11T22:21:35Z

Nuked VM and reinitialized it using first_deploy(). Now the remote S3 links work exactly as they would be expected to. The HTML in my browser says this (which it did when the ticket was originally made):

  <link rel="stylesheet" type="text/css" media="all" href="https://s3.amazonaws.com/karma-beta/css/fontface/fontface.css">
  <link rel="stylesheet" type="text/css" media="all" href="https://s3.amazonaws.com/karma-beta/vendor/foundation-4.2.3/css/foundation.css">
  <link rel="stylesheet" type="text/css" media="all" href="https://s3.amazonaws.com/karma-beta/css/global.css">
  <link rel="stylesheet" type="text/css" media="all" href="https://s3.amazonaws.com/karma-beta/css/media.css">
  <link rel="stylesheet" type="text/css" media="all" href="https://s3.amazonaws.com/karma-beta/css/lightbox.css">
  <link rel="stylesheet" type="text/css" media="all" href="https://s3.amazonaws.com/karma-beta/ajaxuploader/css/fileuploader.css">

But now the CSS and JS actually loads from S3 as if there weren't no problem.

More Alice in Wonderland craziness. Let's say it was the caterpillar getting my VM all smoky with his hookah. Closing ticket.

btbonval · 2014-01-16T05:37:43Z

This is a problem again. It mysteriously solved itself last time, but I think we missed a step.

We have not setup our S3 for static hosting at all! I think it's a fluke that we use it as we do. I looked over the original instructions, which didn't say what to do with the S3 setup, only how to make Django push to the S3.
#65 (comment)

I'm convinced we need to set our S3 for static hosting. There's a button. It says "You can host your static website entirely on Amazon S3. Once you enable your bucket for static website hosting, all your content is accessible to web browsers via the Amazon S3 website endpoint for your bucket."

There are three options on our buckets:

Do not enable website hosting
Enable website hosting
Redirect all requests to another hostname

Guess which one we've selected for all our buckets? "Do not enable website hosting."

If we do enable website hosting, the URL will look like this:
http://karma-beta.s3-website-us-east-1.amazonaws.com/

This should be an easy thing to fix. Check that button. Replace our STATIC_URL with the above.

btbonval · 2014-01-16T22:54:09Z

Turns out each object does have its own permissions. not sure why I didn't see that before, it used to just say "details" when I selected an object, but now I see "permissions."

Each object in the bucket has "Open/Download" permission which must be assigned to Everyone.

I wonder if those permissions are there because I enabled Static S3 hosting? Let's disable and find out. Nope. Somehow I missed it the last go through.

Here's the S3 Policy Generator: http://awspolicygen.s3.amazonaws.com/policygen.html

According to this, Principal "AWS": "." should be all anonymous users:
http://docs.aws.amazon.com/IAM/latest/UserGuide/AccessPolicyLanguage_ElementDescriptions.html#Principal

Here's the Policy I generated for Everyone to GetObject on all objects in karma-beta's S3:

{
  "Id": "Policy1389912543493",
  "Statement": [
    {
      "Sid": "Stmt1389912533433",
      "Action": [
        "s3:GetObject"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::karma-beta",
      "Principal": {"AWS": "*.*"}
    }
  ]
}

That won't save on the S3 bucket, says the principal is invalid.

Unrelated, this might be worth looking into to prevent other sites from hosting directly out of our S3 bucket.
http://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html

btbonval · 2014-01-16T22:56:28Z

There has got to be a way to let Everyone Open/Download without setting it on every single file.

btbonval · 2014-01-16T23:00:40Z

Selecting a directory shows only Details, no Permissions. Checking more than one object shows only Details, no Permissions. No wonder I missed it. Permissions are only available on a single object, one at a time.
http://docs.aws.amazon.com/AmazonS3/latest/UG/EditingObjectProperties.html

btbonval · 2014-01-16T23:03:51Z

Alright well there's no nice way to batch this through S3's interface, but the problem is clearly individual file permissions. Solution: go through each file, one by one, and add Everyone Read/Download.

This good enough for this ticket.

btbonval mentioned this issue Dec 6, 2013

use first_deploy to prepare a VM with Django #173

Closed

btbonval added a commit that referenced this issue Dec 11, 2013

copying workaround from #192 to address part of the problem in #171

d85c5ca

btbonval mentioned this issue Dec 11, 2013

s3 API seemingly required for offline dev work #171

Closed

btbonval closed this as completed Dec 11, 2013

btbonval mentioned this issue Jan 16, 2014

Import MIT Notes #68

Closed

btbonval reopened this Jan 16, 2014

ghost assigned btbonval Jan 16, 2014

btbonval closed this as completed Jan 16, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

external services (S3 specifically) responds 403 Forbidden to VM hosted HTML #192

external services (S3 specifically) responds 403 Forbidden to VM hosted HTML #192

btbonval commented Dec 6, 2013

btbonval commented Dec 6, 2013

sethwoodworth commented Dec 9, 2013

btbonval commented Dec 9, 2013

btbonval commented Dec 9, 2013

btbonval commented Dec 9, 2013

btbonval commented Dec 9, 2013

btbonval commented Dec 10, 2013

sethwoodworth commented Dec 10, 2013

btbonval commented Dec 10, 2013

btbonval commented Dec 11, 2013

btbonval commented Jan 16, 2014

btbonval commented Jan 16, 2014

btbonval commented Jan 16, 2014

btbonval commented Jan 16, 2014

btbonval commented Jan 16, 2014

external services (S3 specifically) responds 403 Forbidden to VM hosted HTML #192

external services (S3 specifically) responds 403 Forbidden to VM hosted HTML #192

Comments

btbonval commented Dec 6, 2013

btbonval commented Dec 6, 2013

sethwoodworth commented Dec 9, 2013

btbonval commented Dec 9, 2013

btbonval commented Dec 9, 2013

btbonval commented Dec 9, 2013

btbonval commented Dec 9, 2013

btbonval commented Dec 10, 2013

sethwoodworth commented Dec 10, 2013

btbonval commented Dec 10, 2013

btbonval commented Dec 11, 2013

btbonval commented Jan 16, 2014

btbonval commented Jan 16, 2014

btbonval commented Jan 16, 2014

btbonval commented Jan 16, 2014

btbonval commented Jan 16, 2014