Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Camo can't handle the `application/octet-stream` mimetype #5596

Open
johnlinp opened this issue Mar 20, 2019 · 11 comments

Comments

@johnlinp
Copy link

commented Mar 20, 2019

Describe the bug
Images with external links in README.md failed to display.

Expected behavior
Images with external links in README.md should display normally.

To Reproduce
There is a gif image in my README.md with an external link to https://github.com/johnlinp/duosql/raw/master/images/duosql-demo.gif.

However, the image shown in my PyPI project page is rewritten and linked to a weird url: https://warehouse-camo.cmh1.psfhosted.org/fb671586f80a719fd21568d008eed338e97ad5f1/68747470733a2f2f6769746875622e636f6d2f6a6f686e6c696e702f64756f73716c2f7261772f6d61737465722f696d616765732f64756f73716c2d64656d6f2e676966.

@di

This comment has been minimized.

Copy link
Member

commented Mar 20, 2019

Hi @johnlinp, the image you're trying to link to is https://github.com/johnlinp/duosql/raw/master/images/duosql-demo.gif, however this producing a 302 redirect that our image proxy doesn't seem to like:

$ http https://github.com/johnlinp/duosql/raw/master/images/duosql-demo.gif
HTTP/1.1 302 Found
Access-Control-Allow-Origin: https://render.githubusercontent.com
Age: 69
Cache-Control: no-cache
Content-Length: 145
Content-Security-Policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com
Content-Type: text/html; charset=utf-8
Date: Wed, 20 Mar 2019 18:17:19 GMT
Expect-CT: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"
Location: https://raw.githubusercontent.com/johnlinp/duosql/master/images/duosql-demo.gif
Server: GitHub.com
Status: 302 Found
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
Vary: X-PJAX
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-GitHub-Request-Id: FE52:75E8:29B59:4A3CC:5C9283F4
X-XSS-Protection: 1; mode=block

Can you try using the actual URL at
https://raw.githubusercontent.com/johnlinp/duosql/master/images/duosql-demo.gif instead?

johnlinp added a commit to johnlinp/duosql that referenced this issue Mar 21, 2019

@di

This comment has been minimized.

Copy link
Member

commented Mar 21, 2019

So, the "weird url" is our image proxy -- this is expected.

The issue is that the Content-Type for this image is application/octet-stream, which it seems our proxy can't handle. Instead, it should be image/gif. It produces this error in Camo:

Non-Image content-type returned 'application/octet-stream': https://raw.githubusercontent.com/johnlinp/duosql/master/images/duosql-demo.gif

You'll need to host the image elsewhere in order for this to work. Interestingly, if you upload the image in a comment, like this:

duosql-demo

The link is https://user-images.githubusercontent.com/294415/54768413-1d17f900-4bcd-11e9-8611-3370f5929ce5.gif, which has the image/gif content type:

$ http https://user-images.githubusercontent.com/294415/54768413-1d17f900-4bcd-11e9-8611-3370f5929ce5.gif --headers
HTTP/1.1 200 OK
Accept-Ranges: bytes
Accept-Ranges: bytes
Age: 0
Cache-Control: max-age=2592000
Connection: keep-alive
Content-Length: 6382153
Content-Type: image/gif
Date: Thu, 21 Mar 2019 16:33:57 GMT
ETag: "931d554bec7010ce8f472b114b3af474"
Last-Modified: Thu, 21 Mar 2019 16:33:15 GMT
Server: GitHub Cloud
Timing-Allow-Origin: https://github.com
Via: 1.1 varnish
X-Cache: MISS
X-Cache-Hits: 0
X-Fastly-Request-ID: 2b582bdef495d7a333f77d0d6d31dc88836d3205
X-Served-By: cache-dfw18637-DFW
X-Timer: S1553186037.042500,VS0,VE450

And which should work with our proxy.

@di di changed the title Images with external links in README.md failed to display Camo can't handle the `application/octet-stream` mimetype Mar 21, 2019

@di di added the bug 🐛 label Mar 21, 2019

@johnlinp

This comment has been minimized.

Copy link
Author

commented Mar 22, 2019

Thank you for the analysis. I've submitted a ticket to GitHub for why the Content-Type of my gif image is mistaken to application/octet-stream. Let's see if they have any explanation.

@johnlinp

This comment has been minimized.

Copy link
Author

commented Apr 17, 2019

Hi @di,

GitHub replied:

It seems we serve gif files under 5MB via raw.githubusercontent. Any files over this size are served as downloadable octet-stream files.

It seems that there is nothing I can do but shrinking the gif image.

@jamadden

This comment has been minimized.

Copy link
Member

commented Apr 17, 2019

@johnlinp If you're satisfied in the resolution, could you close this ticket please?

@johnlinp

This comment has been minimized.

Copy link
Author

commented Apr 17, 2019

Hi @jamadden,

I don't think I can close this issue. Since the title says "Camo can't handle the application/octet-stream mimetype" and it still can't right now, I think we should keep it open until it can handle application/octet-stream mimetype.

@jamadden

This comment has been minimized.

Copy link
Member

commented Apr 17, 2019

It's not clear to me that an image proxy should handle non-image MIME types. Part of the image proxy's job involves security, and to that end, that explicitly includes "Restrict[ing] proxied images content-types to a whitelist".

@johnlinp

This comment has been minimized.

Copy link
Author

commented Apr 17, 2019

Sure. If you think it doesn't make sense to handle non-image mime type, we should definitely close this issue. Thanks.

@di

This comment has been minimized.

Copy link
Member

commented Apr 17, 2019

I think this is probably a valid issue, and at the very least we should explore whether we can add this functionality to Camo. We will almost certainly have other users that encounter this.

@jamadden

This comment has been minimized.

Copy link
Member

commented Apr 17, 2019

Does Camo do other checks to verify images, besides the content type? That is, if the content type whitelist is expanded to include application/octet-stream, is there anything from stopping Mr. Evil from including an "image" reference in his project pointing to https://evil.example.com/badstuff/bad-data? (An exe or vbs or xls...) If not, the question is then, is that exploitable in any new way? I'm not an expert but I'll take a quick first pass at contributing to that conversation.

Based on a quick test of available browsers (safari, chrome, firefox), if an <img> tag has a source that returns a type of application/octet-stream, the filename is inspected to attempt to decode the image. (e.g, <img src="foo"> does nothing when /foo is served as application/octet-stream (even though it is fetched), but <img src="foo.gif"> does display even when /foo.gif is served as application/octet-stream. That's not exhaustive, and doesn't include older browsers, or other ways of referencing images (if any are allowed through the renderer).

Security wise, I could imagine potential issues (overflows and the like) with browsers mishandling malformed input, but those are exactly the same issues that would result from malformed input with a recognized content type, so that's not new.

Navigating directly to the URL, either foo or foo.gif, when served with application/octet-stream, caused all the browsers to download the file, or prompt if I wanted to open it. In its default settings, Safari automatically opened foo.gif --- even when foo.gif wasn't actually a GIF, leading to errors, and more potential malformed input handling, this time on the part of arbitrary programs. The Content-Security-Policy header doesn't seem to make a difference. Since we don't currently allow octet-stream, that's new behaviour, albeit a corner case. That's potentially exploitable; is it more exploitable than a link directly to evil.example.com? Maybe, because psfhosted.org is a domain of higher trust?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.