Discussion: serving content when resource is not meant to be accessed directly

[elarlang]

Problem: there is no requirement which says, that API responses must be in JSON or XML format.

If they are not, and those return some content of file for example, then with direct request to the API response, the content may be executed as HTML, XML and/or JavaScript. Problem is valid for not authenticated API access or when session is cookie-based.

If an API returns XML output with Content-Type: text/xml and if an attacker can fully control the content, it's already possible to execute JavaScript from that situation with direct request.

Previously there was requirement 14.4.2, which was covering this kind of situations - serve them as attachment = no execution problem:

V14.4.2 Verify that all API responses contain a Content-Disposition: attachment; filename="api.json" header (or other appropriate filename for the content type).

There was issue #721 to discuss the meaning for that, but requirement was deleted via issue #1004.

Like in file section there is requirement:

V12.5.2 Verify that direct requests to uploaded files will never be executed as HTML/JavaScript content.

I think something similar must be for API responses.

Or in general - everything, which is not intended to serve with direct request, can have Content-Disposition: attachment header and solve a lot of content execution problems.

No proposals from my side yet, I was not supporting removing 14.4.2 for the reasons above, so now opened for discussions - what is the correct defense in this kind of situation and is it covered by ASVS? If not, we should fill the cap.

ping @NeilMadden @jmanico @Bankde

[jmanico]

If an API returns XML output with Content-Type: text/xml and if an attacker can fully control the content, it's already possible to execute JavaScript from that situation with direct request.

It's the user interface's job to protect against that based on how the XML or JSON is being embedded in a UI. This is not the job of a server.

If I missed something can you give me a more specific example?

PS: THANK YOU for opening a new issue and moving the content here. I appreciate this Elar!

[jmanico]

Dear @NeilMadden and @Bankde can you take a look at this issue and provide comments, please?

[Bankde]

I think he mean directly access the URL; e.g. attacker sending API link to the victim.
This is the real possible case for API that incorrectly returns Content-Type: text/html. I'm not familiar with text/xml though. Is that also a case? I'm bit busy right now so will test it later.

[elarlang]

If Content-Type is text/html but content itself is JSON, then it's fixable with just proper Content-Type: application/json.

If you can make request directly to URL where API is serving XML content with Content-Type: text/xml it will execute as XML. And from XML you can execute JavaScript as well, if you have full control over served content.

An example:

<?xml version="1.0"?>
<html:html xmlns:html='http://www.w3.org/1999/xhtml'>
<html:script>
alert(1);
</html:script>
</html:html>

And if API is serving some other content, which is not XML or JSON? Then what?

[Bankde]

I have reviewed this issue a bit and here is my opinion.

Current browsers consider XML mime type to be active content (thus executing Javascript/XSS). There are also other contents that fall into this categories. This link has very good summary on the content-types with XSS issue: https://github.com/BlackFan/content-type-research/blob/master/XSS.md

I'm not sure I understand this correctly
https://datatracker.ietf.org/doc/html/rfc7303#section-10

XML MIME entities contain information that may be parsed and further
   processed by the recipient.

It also stated that text/xml may also include processing instructions:

An XML document labeled as application/xml or text/xml, or with a
   '+xml' media type, might contain namespace declarations, stylesheet-
   linking processing instructions (PIs), schema information, or other
   declarations that might be used to suggest how the document is to be
   processed.

I understand that the RFC does not explicitly state about the browsers but overall it looks like XML being executed is correct by the design (or until browsers decide to change this behavior). If this is true, then the responsibility of preventing vulnerabilities (not only XSS, but also many others) should normally fall into the input validation and encoding categories.

I'm not confident in XML topic though. I will need more time for the research later.

[elarlang]

First, let's not get stuck only to JSON and XML answers from an API - we don't have any requirement for such limitation.

I understand that the RFC does not explicitly state about the browsers but overall it looks like XML being executed is correct by the design (or until browsers decide to change this behavior). If this is true, then the responsibility of preventing vulnerabilities (not only XSS, but also many others) should normally fall into the input validation and encoding categories.

That is the same like to say "XSS is input validation problem" because execution HTML / JavaScript is correct by design. It is not input validation, it is output encoding problem. And if encoding is not the solution, then serving content correctly (Content-Type + as attachment if needed) is solution.

The problem here is, in this example, XML response from an API is not meant to be executed in browser as a response for direct request to URL of an API - it's meant to be loaded by browser via user interface (using JavaScript).

[Bankde]

It is not input validation, it is output encoding problem.

We might both have the same idea but just from different perspectives. I personally think it is both. We have to ensure that user input do not cross into the domain of script/HTML structure. If we concatenate the user input directly into the Javascript content (don't do this) then output encoding may not be enough.

My current breakdown idea is like this:

• User can control the whole XML content for the API, then application may be doing something wrong.
• User control the whole XML content as upload/downloadable file, application should serve back as an attachment. No issue here.
• User can control some fields' values, encoding sounds like a solution. Preventing not only XSS but also poisoning and malformed XML structure. No issue here.
• User can control both fields and values such that field="html:script xmlns:html..." and value="alert(1);", this is the only issue I could address. Either validation or content-type+attachment may be the solution?
• Anything else I'm missing?

The problem here is, in this example, XML response from an API is not meant to be executed in browser as a response for direct request to URL of an API

I get your point. And unlike json, this XML complex thing makes my head spin.
Content-Type + as attachment looks good as a solution (not yet tested though); however, I'm not sure it is the standard mitigation to this issue. Does Content-Type: text/plain also accomplish the same thing? It basically says to the browser "Do not execute this" and let JS handles the rest.

[Bankde]

By the way, I completely agree with @elarlang concern. While I don't have an exact idea of recommendation, I propose to add some general idea like this into the requirements:

Verify and choose proper Content-Type for API responses that the browsers do not execute them as an active content (XSS) when directly access through the API URLs.

This will cover not only json but also XML and others as well and let the complex details to the developers to decide according to their applications.

[elarlang]

Content-Type text/plain for XML content probably solves this example, but I call it dirty hack.

Verify and choose proper Content-Type for API responses that the browsers do not execute them as an active content (XSS) when directly access through the API URLs.

What is the reason that you want/need to show answer from an API in browser? (inline vs attachment).

In practice, what Content-Type someone need to use when serving XML (including SVG)? PDF? any other text? any other binary?
application/octet-stream could be actually solution here, but the effect is the same like forcing download the content with previously removed requirement. At the moment I have not tested it, maybe browsers actually still want to render the content.

My idea is that Content-Type must match with served content, otherwise (in theory) you may cause some other problems.

Choosing correct Content-Type is (and will be covered) with V14.4.1.

Solution for current problem should be: to avoid response from an API being rendered as active content, serve it as attachment.

... and this is what deleted requirement V14.4.2 covered.

Why I got stuck in #721 was:

• should "force download" be only for API
• should "force download" be for everything which is not mean to be served with direct HTTP request, but as source for HTML/JavaScript/CSS/... content.

If you don't need to navigate to this URL, it should be served as attachment and a lot of problems could be solved.

[Bankde]

Content-Type text/plain for XML content probably solves this example, but I call it dirty hack.

That's true haha. (Forgive me if I sound like a troll), the point is I feel like "API response as an attachment" is also a dirty hack more or less.

What is the reason that you want/need to show answer from an API in browser? (inline vs attachment).

Not much, some developers love to see it visualize though. That's the reason why browsers support json beautifier.

... and this is what deleted requirement V14.4.2 covered.

It is a positive side-effect but not the main concern of V14.4.2 though. In my opinion, deleting that one when it's deprecated then propose a new accurate requirement that matches you concern is a better call.

Solution for current problem should be: to avoid response from an API being rendered as active content, serve it as attachment.

The problem is that while it solves problem for XML, it improves nothing for JSON. Not that I turn your idea down, I just feel that it's also a hacky solution and I don't want the new requirement to effect JSON API application too much. Should it be specifically for XML? or L3 for JSON?
How about this?

Verify and choose proper Content-Type or serve as an attachment for API responses so that the browsers do not execute them as an active content (XSS) when directly access through the API URLs.

I don't want it to be duplicate or sound like a new V14.4.1 though. But I don't want it to be too specific or a hindrance to the applications that do not have this XSS issue either. You may change it as you wish.

[elarlang]

point is I feel like "API response as an attachment" is also a dirty hack more or less.

Disagree. Serving content as attachment when you don't have any business logic need to watch it directly it's correct solution. You can not see any piece of content "out of context" (not only XSS problem)

Not much, some developers love to see it visualize though. That's the reason why browsers support json beautifier.

"Some developers may like to see visualized JSON beautifier" is not really valid argument to remove some security requirement.

In my opinion, deleting that one when it's deprecated then propose a new accurate requirement that matches you concern is a better call.

Can you please clarify - what exactly is deprecated?

The problem is that while it solves problem for XML,

Like I wrote many times, don't get stuck with XML and JSON examples. Till we don't have requirement to limit API output only for JSON and XML, we need to take in account, it can be whatever.

it improves nothing for JSON.

That's correct and I have never said anything against that but it is valid ONLY if you have Content-Type set correctly. But if you don't have and an API returns JSON but Content-Type is text/html, then serving response as attachment saves the situation. Wrong Content-Type is covered with 14.4.1, but gives one extra layer of defense.

I just feel that it's also a hacky solution and I don't want the new requirement to effect JSON API application too much.

Disagree. Arguments? How does it affect JSON API? Just this "developers want to see" situation or something else?

[Bankde]

Apologize. My head is so spinning over so many issues tangling together @_@

Disagree. Serving content as attachment when you don't have any business logic need to watch it directly it's correct solution. You can not see any piece of content "out of context" (not only XSS problem)

Ah I get your perspective now. It does make sense.

Can you please clarify - what exactly is deprecated?
Disagree. Arguments? How does it affect JSON API? Just this "developers want to see" situation or something else?

At first, I wouldn't want a change that affect too much, against the norm, and with little improvement. But that argument was from my own mistake and confusion that this will be a new requirement; this requirement is actually been here for quite some time so there should not be any issue to developers if we bring it back.

Another mistake is that I thought it exists solely to mitigate RFD and mime type confusion (which original commit stated) so I called a remove too early as I saw that most of RFD and mime type confusion is deprecated and handled by other requirements. However, from this discussion, it does also prevent browser execution on non-business-logic-related content too.

For the level, I originally preferred it to be L3. I'm still inclined that proper encoding/validation can prevent XML XSS and this is just a defense in depth. However, I find the idea to be similar to CSP so I'm ok if it's L1, L2, L3 (same level as CSP).

Thank you for tackling with my head. I'm still opened to the discussion but I need to rest for today.

[ThunderSon]

Allow me to chime in.

APIs are the main orchestrator for the type that gets returned. What the issue should clearly say is that "The client should not parse the returned response as they please it to be", MIME confusion is a direct issue here, executing JSON, running a JPG file as a script, etc.

This is related to anything being returned to ANY client, and not to browsers only. It's similar with how TLS can be configured to enforce the server config over the client one.

The API if it returns XML, should clearly say "This is XML", do what you would do with XML. If the backend is worried about the client behaviour, then it should clearly secure its response. XML is supposed to execute, and should for all I know if it's sent as is.
CDATA is there to protect against that and load content as text.

Maybe something to the lines of "Be aware of the execution paths of your responses" which is still vague, but I am trying to get this a bit closer to how APIs are and should work. Going for undercuts just make security look cheap and provide a false sense of security.

[elarlang]

Discussed this topic with @ThunderSon and I provided some more food for brain.

TL;DR

• for taking away JavaScript execution you can use Content-Security-Policy: sandbox
• for taking away any content execution in browser you can use Content-Disposition: attachment

Target: only resources which are not meant to be accessed directly but user/browser can do it - non-autenticated users or users in application which uses cookie-based sessions.

Direct access - no extra defense

Test	Content-Type	Content	HTML/XML executed	JavaScript executed
link	text/html	HTML	Yes	Yes
link	text/html	JSON	Yes	Yes
link	application/json	JSON	No	No
link	text/xml	XHTML	Yes	Yes
link	image/svg+xml	SVG	Yes	Yes

Direct access - defense Content-Disposition: attachment

Test	Content-Type	Content	HTML/XML executed	JavaScript executed
link	text/html	HTML	No	No
link	text/html	JSON	No	No
link	application/json	JSON	No	No
link	text/xml	XHTML	No	No
link	image/svg+xml	SVG	No	No

Direct access - defense Content-Security-Policy: sandbox

Test	Content-Type	Content	HTML/XML executed	JavaScript executed
link	text/html	HTML	Yes	No
link	text/html	JSON	Yes	No
link	application/json	JSON	No	No
link	text/xml	XHTML	Yes	No
link	image/svg+xml	SVG	Yes	No

[Bankde]

That's very nice summary and comparison with CSP. Thank you.

Thought from me:

• There is still some small use from this header for a Defense in Depth that is not completely deprecated by browsers yet.
• The requirement was originally existing before we deleted it (a few days ago), so there should not be any issue in reverting it.
• So overall, I'm ok if we bring the requirement back.

Lastly, I also like this idea. I'm interested if there is any reference (as a security requirement/practice) to this?

Serving content as attachment when you don't have any business logic need to watch it directly it's correct solution. You can not see any piece of content "out of context" (not only XSS problem)

[elarlang]

Some more options to achieve "do not render content out of context in browser":

• for API - for old-style XHR request you can check, is there XMLHttpRequest header, and if there is not, then you should not give response.
  • Problem, fetch API do not send this header, but you can set your own custom header to solve it (like serving content with direct access is not issue, when API needs Authorized header)
• use Sec-Fetch-Mode and check on the server side, was it navigation request or not.
  • not ready as only solution yet, as Firefox, IE ans Safari do not support it, global support 68% (https://caniuse.com/mdn-http_headers_sec-fetch-mode)

The point is - there are many ways to achieve goal do not serve or render response content on direct access to resource, which is not intended for that.

The reason, why #721 stayed opened, was that purpose for 14.4.2 was not clear and that is why it got (too fast) deleted as well. So just bringing back this requirement with the same text is not the best solution and instead of "technical requirement" maybe we should ask "what vector must be taken away".

Direction for that requirement should contain goals:

• if there is direct request to resource which is not meant to be navigation (including responses from API)
  • is not rendered and/or executed in browser as HTML, XML or JavaScript
    • using serving as Content-Disposition: attachment
    • using sandboxing with Content-Security-Policy: sandbox
  • OR serving response is not provided
    • as request is not initialized by application (not authorized request)

At the moment it seems, that technically the most bullet-proof solution could be to bring back previous 14.4.2, removing API part from it and adding description, what goal this requirement have.

So, opened for discussion and arguments if there is some better alternatives.

[ThunderSon]

I am tilted more towards the CSP solution:

1. Pushes for the adoption of CSP
2. If something is to look bad, it feels better contained in the browser
3. I am not a fan of websites that download content when being traversed (no one is)

The benefit of using Content-Disposition is against the HTML rendering. That is still going to happen when the user opens the file. What we just did is mainly add an additional step.

The last defense option looks like using the Origin header. If it's coming from the API domain itself, then don't provide a response. Not a big fan, but can be an extra layer since the browsers are firm on this header.

[Bankde]

Hi, I don't want this to go stale.
This Content-Disposition is currently still in the current released version of ASVS. We should decide it soon whether to bring it back (and/or rephrase) or confirm of removal so the change can be minimal.

I'm ok with either way.
or how about rephrase into something more general?

Verify that security controls are in place that browsers do not render or execute the API responses (and other resources not meant for navigation) from the direct URL access.

I'm not a fan of adding specific implementations into the general guidelines; they are usually fragile. Developers are free to add them in their project as pleased though.

[elarlang]

One more random argument, CSP vs attachment but now against attachment - if user is forced to download some attacker controlled content, it is better for application but worse for user.

Let's say application is serving HTML content and serving it as attachment. If user opens it in own machine, all the scripts inside HTML can be now executed without Content-Security-Policy limitations. It can not cause directly any damage for the application, but can use victim's browser to make some other attack.

[elarlang]

I'm not a fan of adding specific implementations into the general guidelines; they are usually fragile. Developers are free to add them in their project as pleased though.

It's only true when you have multiple ways to achieve the same goal - like with CSRF. If we want to keep this requirement in V14.4 category, it should direct clearly what kind of configuration options it covers.

[Bankde]

can use victim's browser to make some other attack

Thank you for bringing that up. I realize we may not want something with possible negative effect in the guidelines so...
how about this?

Verify that API responses contains `Content-Security-Policy: sandbox` or other security controls so that the browsers do not render when directly accessed from the URL..

This way, we are enforcing the CSP (not yet perfect but no downside) while still open to other solutions (that devs have to take care by themselves)

[ThunderSon]

This just came in today:
https://portswigger.net/daily-swig/epub-vulnerabilities-electronic-reading-systems-riddled-with-browser-like-flaws

Exactly what we are discussing, if we want to consider the endpoint returning the content to be our API at hand.

[NeilMadden]

A bit late to this party (long weekend in the UK). I am generally in favour of Content-Security-Policy: sandbox over Content-Disposition as I think it better reflects what we actually want to achieve and I agree with @ThunderSon 's comment from a few days ago.

To add to what @elarlang says about running a local file may result in CSP bypass, it looks like the origin of a file: URL is somewhat murky. I suspect most browsers would push it into an opaque origin much like CSP: sandbox, but this doesn't seem guaranteed.

[NeilMadden]

I'm also happy with wording like @Bankde suggests.

[elarlang]

First big decision for wording is - do we limit it for API or we going to build general "do not allow browser to render responses if they are not intended to be rendered by the application itself"?

[Bankde]

First big decision for wording is - do we limit it for API or we going to build general

I'd love it to cover all the cases.
However, I'm not sure whether this some-parts-of general case may be already covered by other requirements? For example, V12.5.2 has covered the downloadable files. Then it might be reasonable to have separate requirement for API endpoints so it's easier to follow/audit?

V12.5.2 Verify that direct requests to uploaded files will never be executed as HTML/JavaScript content.

[jmanico]

Just a note on 12.5.2

1. User authored HTML is common, you just need to clean that content via a HTML sanitizer before displaying to another user.
2. Even user authored JavaScript can be displayed to another user safely via iFrame sandboxing like https://jsfiddle.net/.

[Bankde]

How about this?
As far as we know, downloadable content is handled by other reqs. So we specify this to be an API requirement. This will make the initial proposal simple and easy to provide with specific configurations. We can raise a new issue/req later if we find areas that are not covered by download/upload or API.

My previous proposal still stands and feel free to adjust/criticize/edit/reword any way you like.

Verify that API responses contains Content-Security-Policy: sandbox or other security controls so that the browsers do not render when directly accessed from the URL.

[elarlang]

@elarlang I am still not sure what your are proposing to add here.

Re-read it all and #1009 (comment) is still valid opinion, goal and proposal.

[Sjord]

If resources are only meant to be used with JavaScript requests, a possible solution would be for the client to set a specific header (X-Requested-With: XMLHttpRequest) and then only serving the response if that header is present. This would prevent users from browsing to that resource directly.

Another solution is to set a content-type to text/plain or application/octet-stream. That way, the HTML won't be interpreted as HTML. However, I don't really like this since then the content type is incorrect, and this can make it harder to work with these resources in legitimate use cases.

[elarlang]

If resources are only meant to be used with JavaScript requests, a possible solution would be for the client to set a specific header (X-Requested-With: XMLHttpRequest) and then only serving the response if that header is present. This would prevent users from browsing to that resource directly.

It is one possible way of detecting it, but nowadays I would take direction to Sec-Fetch-* headers. The requirement must be abstract enough to not limit to any solution but must just describe the goal.

Another solution is to set a content-type to text/plain or application/octet-stream. That way, the HTML won't be interpreted as HTML. However, I don't really like this since then the content type is incorrect, and this can make it harder to work with these resources in legitimate use cases.

This is technically incorrect and against X-Content-Type-Options: nosniff requirement, if the header is set (like requirement V14.4.4 asks), then with incorrect Content-Type for <script> sources are not loaded by browsers.

[jmanico]

Any request defense that requires certain non-session headers can be directly evaded.

[tghosth]

@elarlang I am still not sure what your are proposing to add here.

Re-read it all and #1009 (comment) is still valid opinion, goal and proposal.

So can you come up with a proposed requirement wording @elarlang because I am still not sure what we are asking developers to do.

[elarlang]

So, based on #1009 (comment)

I tried to put my ideas here (wording needs some work):
Verify that if a client navigates to a resource (template, API response) which are not meant to be accessed directly, the application have defense (not serving the response, serving as an attachment or sandboxed content) to avoid rendering the response in browser or showing content and functionality out of context.

[elarlang]

@tghosth - does the last comment makes it more clear for you?

[tghosth]

Ok so how about:

Verify that security controls are in place (e.g. Content-Security-Policy: sandbox, Content-Disposition: attachment) to prevent browsers from rendering content or functionality in HTTP responses in an incorrect context such as when an API is called directly rather than via an XmlHttpRequest.

[elarlang]

2 things to improve:

• your proposal does not cover "do not serve content at all" which would be my first recommendation
• XmlHttpRequest is a bit oldschool, there is also fetch. But I would not limitate the requirement to API context, it applies the same way to every resource.

[tghosth]

So how about this:

Verify that security controls are in place (e.g. not serving the content unless headers indicate it is the correct context, Content-Security-Policy: sandbox, Content-Disposition: attachment, etc) to prevent browsers from rendering content or functionality in HTTP responses in an incorrect context. For exampel when an API or other resource is loaded directly rather than via an XmlHttpRequest/fetch.

[elarlang]

I tried to change to order, fix typo end remove the last part:

Verify that to prevent browsers from rendering content or functionality in HTTP responses in an incorrect context, security controls are in place (e.g. not serving the content unless headers indicate it is the correct context, Content-Security-Policy: sandbox, Content-Disposition: attachment, etc). For example when an API or other resource is loaded directly.

[elarlang]

The requirement got in as 50.5.3 as it was proposed:

#	Description	L1	L2	L3	CWE
50.5.3	[ADDED, DEPRECATES 14.4.2] Verify that to prevent browsers from rendering content or functionality in HTTP responses in an incorrect context, security controls are in place (e.g. not serving the content unless headers indicate it is the correct context, Content-Security-Policy: sandbox, Content-Disposition: attachment, etc). For example when an API or other resource is loaded directly.	✓	✓	✓

@tghosth - do you have any additional recommendations to this wording or we can close this?

[tghosth]

Makes sense to me