[2.0] Introduce adapter interface and implementations

[franzliedke]

As discussed in #3, this PR starts by adding a HTTP adapter interface and a basic implementation using PHP's global functions a la headers_list(), header_remove() and header().

This also opens the door for introducing a simple adapter implementation that simply collects all headers and returns them as a string - for use in tests, which - I believe - was the only use of the headersAsString mode.

To prevent more tampering with actual HTTP responses than necessary, I also introduced a new HeaderBag class as a middleman that is added to / removed from before actually sending / removing all headers at once. Turns out, this is more or less what is already done with the internal $this->headers array, so I will remove that array in a next PR once the HeaderBag can completely fulfill its role.

Hope you like it. :)

[franzliedke]

Whoops, I think I messed up the difference between add and replace here. add should still add the header if it already exists, just add another equally-named header...

[aidantwoods]

Changes are looking good, thanks for the great work! 😃

(Just a few tiny queries 😉):

1. Can you preserve capitalisation in the header name by implementing something similar to this? (

   SecureHeaders/src/SecureHeaders.php

   Lines 441 to 445 in 36e81d4

   	$this->headers[$name] = array(
   	'name' =>
   	$capitalisedName,
   	'value' =>
   	$value,

   )

   Where the key of the array is always lowercase, but the original casing is contained as a sub-item along with the value?

2. Could you add type assertions for functions that take scalars that are not type-hintable (and would break if given the wrong type)? Pre PHP7 only arrays or an instance of a class are type-hintable, so things like strings, integers, booleans etc... are all "non-type-hintable".
   I've created a function that backports the type of error that you'd get from a type error (

   SecureHeaders/src/SecureHeaders.php

   Lines 2213 to 2270 in 36e81d4

   	private function assertTypes(array $typeList, array $argNums = null)
   	{
   	$i = 0;
   	$n = count($typeList);
   	foreach ($typeList as $type => $vars)
   	{
   	if (is_array($vars)) $n += count($vars) - 1;
   	}
   	if ( ! isset($argNums)) $argNums = range(1, $n);
   	$backtrace = debug_backtrace();
   	$caller = $backtrace[1];
   	foreach ($typeList as $type => $vars)
   	{
   	$type = strtolower($type);
   	$type = preg_replace(
   	array(
   	'/bool(?=$|[\|])/',
   	'/int(?=$|[\|])/'
   	),
   	array(
   	'boolean',
   	'integer'
   	),
   	$type
   	);
   	foreach ($vars as $var)
   	{
   	$allowedTypes = array_merge(
   	array('NULL'),
   	explode('|', $type)
   	);
   	if ( ! in_array(($varType = gettype($var)), $allowedTypes))
   	{
   	$typeError
   	= new SecureHeadersTypeError(
   	'Argument '.$argNums[$i].' passed to '
   	.__CLASS__."::${caller['function']}() must be of"
   	." the type $type, $varType given in "
   	."${caller['file']} on line ${caller['line']}"
   	);
   	$typeError->passHeaders($this);
   	throw $typeError;
   	}
   	$i++;
   	}
   	}
   	}

   ). Perhaps this function should be moved somewhere that's more accessible?

   The gist of the syntax can probably be summed up in these examples:

   public function removeCSPSource($directive, $source, $reportOnly = null)
       {
           $this->assertTypes(array('string' => array($directive, $source)));

    public function hpkp(
           $pins,
           $maxAge = null,
           $subdomains = null,
           $reportUri = null,
           $reportOnly = null
       ) {
           $this->assertTypes(
               array(
                   'string|array' => array($pins),
                   'int|string' => array($maxAge),
                   'string' => array($reportUri)
               ),
               array(1, 2, 4)
           );

   For the second example, just stick an | between allowed types (if multiple) – gives a similar feeling to working with bitwise operators. The second arg also gives an array listing the argument numbers (since in this case they weren't sequential starting from 1).

   Basically both arguments of assertTypes should be arrays, the first containing (unique) keys mapping to an array of the arguments that should be of this type. If an argument(s) can satisfy multiple types then stick a pipe in-between the allowed types in the key. The second array should list the argument numbers (for your function) if the argument variables weren't given in order (1,2,...,n). (Argument numbers important for a meaningful error message).

3. The function getHeaderAliases will also need modifying this update (possibly removing). Its only purpose is really to let removeHeader return something useful about whether the header existed or not. getHeaderAliases is almost the same as your has function (it checks SecureHeaders' internal list and PHPs list at call time)

4. I'm not sure that header($headerString, false); can be replaced by $this->headerBag->add('Set-Cookie', $headerString); – since the Set-Cookie header likely need to be set more than once (set for each cookie), can the headerBag be adjusted to allow more than one header to be set with the same name?

Thanks again for all those changes so far!

[franzliedke]

Thanks for the detailed feedback!

I have stumbled across most of these already and was working on a follow-up PR after sending this one.

1. Can you explain why you decided to preserve capitalization? As far as I know, headers are not case-sensitive, so I went with the approach that I knew from PSR-7 (lower-casing everything).
2. I will extract the type assertions to its own class.
3. Should not be necessary if lower-casing everything is fine. By the way, why do you check both the internal list and the headers that PHP has stored? If I read the code correctly, the ones from headers_list() are already part of the internal headers array.
4. Yep, that was an oversight. I will push a fix later today.

[aidantwoods]

1. RE capitalisation, this is actually PSR7 compliant 😉
   http://www.php-fig.org/psr/psr-7/#http-headers

   Despite that headers may be retrieved case-insensitively, the original case MUST be preserved by the implementation, in particular when retrieved with getHeaders().

2. Awesome!

3. Ah yes, I forgot to mention about the construct function! importHeaders gets called as part of the ->done call (so at init SecureHeaders doesn't actually have all the headers loaded into ->headers). I chose this method partly because it lets you define an extension and populate the construct function with your own config options (without having to reimplement code). And partly because I'd have to do an ->importHeaders at ->done anyway to account for any headers set to PHPs list between init and ->done. Because headers could be set between init and ->done, I'd have to implement something like ->getHeaderAliases to check both lists anyway for anything that needs to know about all headers at call time during config (i.e. before ->done).

   Going back to the construct function, obviously instance creations can't really be moved into the main class body. So perhaps I will have to change the wiki to advise that parent::__construct() is called on any extensions of the class in the 2.0 release.

4. Thanks! 😄

[aidantwoods]

Also, just RE the capitalisation again – some clients (like IE11 😞) have been known to break when Header-Names-Dont-Look-Like-This. That's kind of why I made correctHeaderName default to on – as well as asthetic reasons ;-)

[franzliedke]

Hmm, extracting the type checking method is kind of tricky, as it is coupled to the SecureHeaders - apparently it tries to send all headers that were sent so far, when displaying an exception.

This is not needed, in my humble opinion. If the type checks fail, the program should just fail completely, we don't need to try to get some of the behavior that we wanted. After all, we are trying to mirror PHP 7 here, right?

[aidantwoods]

Not "needed" as far as mirroring PHP7 – though IMO sending the headers that we know about is better than a failure causing headers not to be sent at all. (An enhancement on top of PHP 7 if you like). Also, since this class is really all about headers – having a misconfiguration prevent all headers being sent seems a bit, "suicidal" (lack of a better word).

If things like CSP don't get sent then it could open the page up to script execution – supposing that an attacker could somehow force the exception. Not a perfect solution I know (ideally people should keep user input away from headers), but given PHP will serialise arrays and strings in $_POST for example – it's not infeasible to imagine that an attacker could leverage this control over variable types to generate type exceptions.

Would it be completely unreasonable to suggest that SecureHeaders to pass its own instance along to the HeaderBag maybe? (Similar to what happens when a type error is thrown?).

[franzliedke]

We're talking about a type error. I would argue that's a much worse thing to an app/website than a few headers not being sent.

[aidantwoods]

Well that's why we throw the exception – though the exception shouldn't become a vector for disabling CSP (for example), IMO. We're definitely not talking about ignoring the error – we're just sending everything that was okay up until that point. (If the response is completely lacking in headers, that can put the user in danger).

[aidantwoods]

Incidentally by sending the headers from __toString() in SecureHeadersTypeError the headers only get sent in the case that the exception is uncaught. If the dev catches the type error then everything carries on as normal 😃

[franzliedke]

A method called __toString() should not have side effects like that.

Again, these errors are meant to show when the developer makes an error - preventing this exception from being thrown at all is the developer's job. I don't agree that we are talking about a real vector here.

[franzliedke]

P.S.: And, if anything, the problem about such a "vector" would be that the site breaks down, not that some headers do or do not get sent.

[aidantwoods]

Hmm... I'm not sure – I think the application responding in the most helpful way should be the goal? If the type error is left uncaught then the dev pretty clearly hasn't accounted for it – if we send the headers at this point then we get behaviour that is close to encountering an exception after using the built in header function (everything that has already been configured is sent).

I do agree that it is the devs fault, and they should catch the exception, but I think we can at least try to honour the developer's intent for things that were previously configured correctly?

RE __toString(), it's a magic method that gets called if the class is casted to a string (not sure casting a class to a string should have any "expected" behaviour – other than returning a string). Certainly nothing should just attempt to cast an instance of an unknown class to a completely different type and expect it to work?

From the docs:

The __toString() method allows a class to decide how it will react when it is treated like a string. > For example, what echo $obj; will print. This method must return a string, as otherwise a fatal E_RECOVERABLE_ERROR level error is emitted.

[franzliedke]

Ha, that's the nature of PHP. ;)

IMO, the key point here is can these exceptions happen surprisingly in production? And given where the assertTypes() method is used, I don't see why a runtime value should have a different type from the values passed in during testing. So, I'm saying, these exception should already tell developers they're doing something wrong when developing.

[aidantwoods]

😉

I'd say that yes, it's possible one of these exceptions could happen surprisingly in production (especially on public scoped functions).

Even just as a mechanism to cover self-inflicted failures I think it's valuable. If I push an update that casts a variable to the wrong type in a place I didn't test for – I'd rather the application generated an exception, than the application generated an exception and someone's CSP doesn't get deployed!

(remember that SecureHeaders does things like cookie upgrades too – would rather these things happened even in the event of a type error, than throw an error that allows cookies to go out without the secure attribute).

[franzliedke]

Okay, I just pushed two commits that extract the type assertion method into its own static helper class, and uses it from the HeaderBag as well. This also means I

If you can't live without it, it will have to be added back on top of these classes. Again, I'd strongly advocate against it. It violates separations of concerns, I would say it's unexpected, it uses the global PHP methods (so would not work well with adapters), and it is trying to be smart in cases where the developer has messed up (so badly that they should notice in development). And in any case, trying to convert any weirdly-typed input into expected types seems like more in line with classic dynamic typing as beloved and hated in PHP.

And I'd still like to see that example of where a developer might get surprised by a sudden wrong type, for any of these methods. ;)

As for your other feedback, I'll try to get it done in the next few days.

[franzliedke]

Okay, casing of headers and the add/replace thing is now fixed.

What's left is now dealing with getHeaderAliases(). Can you come up with a test case that helps me implement this feature?

[aidantwoods]

Okay, I just pushed two commits that extract the type assertion method into its own static helper class, and uses it from the HeaderBag as well.

Okay, casing of headers and the add/replace thing is now fixed.

Awesome! 😃 Changes looking great there! Thanks for all the hard work.

And I'd still like to see that example of where a developer might get surprised by a sudden wrong type, for any of these methods. ;)

Okay ;p

setcookie('auth', 'supersecretauthenticationstring');

$headers = new SecureHeaders;

$headers->csp('default', 'none');
$headers->cspNonce('script');

$page = $_GET['redirect'];
$headers->header('Location', $page);

$headers->done();

$ curl -sI localhost/?redirect=a
--------------------------------
HTTP/1.1 302 Found
Location: a
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Frame-Options: Deny
Content-Security-Policy: default-src 'none'; script-src 'nonce-eUthX8wh+Y5MQXrNCuZb5MmLMvwfSO7XMisyLNpk';
Set-Cookie: auth=supersecretauthenticationstring; Secure; HttpOnly
Content-Type: text/html; charset=UTF-8

$ curl -sI localhost/?redirect%5B%5D=a
--------------------------------------
HTTP/1.1 500 Internal Server Error
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Frame-Options: Deny
Content-Security-Policy: default-src 'none'; script-src 'nonce-hkzy9P8+OCyexj7bU5pGrNihH91t1Ji7mTSPt/x/';
Set-Cookie: auth=supersecretauthenticationstring; Secure; HttpOnly
Connection: close
Content-Type: text/html; charset=UTF-8

Right now, already configured headers (like CSP), and automatic actions tied to ->done (like cookie upgrades based on name) are still able to be sent and completed.

If we take away this special exception handling, then you'd get an output like this:

$ curl -sI localhost/?redirect%5B%5D=a
--------------------------------------
HTTP/1.1 500 Internal Server Error
Set-Cookie: auth=supersecretauthenticationstring
Connection: close
Content-Type: text/html; charset=UTF-8

Basically a free pass for an attacker on the network/one that can find XSS to do what they like with the page/cookies.

Given that the type in this case is dictated by the user (thanks PHP 😉) – I can't see it being an obvious one to spot in development (bar a pentest – in which case it should be obvious).

Perhaps we can make this functionality optional to solve both our concerns? Regarding whether it should default to on/off, I can open up another issue perhaps and we can see if anyone from #3 has any input?

P.S. RE

it uses the global PHP methods (so would not work well with adapters)

there's no reason that the built in header has to be used in this handling, could just as easily be passed off to the configured adapter right?

I do take your point that it would be unexpected, so perhaps configuration options would be a must if we were to keep this (though wouldn't an uncaught exception be just as unexpected?).

[aidantwoods]

I know StringHttpAdapter isn't really meant to ever retrieve headers as an object, but just to keep this as a good implementation example for others, I guess this should be return new HeaderBag($this->headers);, rather than a blank list of headers?

[franzliedke]

No, not really, as getHeaders() is called before $this->headers is ever populated.

[aidantwoods]

No guarantee that explode would return two values, and that space is optional but not ignored if present. Unfortunately I think that's slightly too conditional for a nice one liner, would be pleased if wrong though!

[aidantwoods]

Much like PHPs type hinting – that type assertion function will allow null to be passed (so you can see if the user left the default value in, without having to hard code check for the default). SecureHeaders defaults a header value to null if it isn't specified, so it's plausible that the value may be of null type (which should also set a header with no value).

[aidantwoods]

Both bugs fixed here were brought over from my code of course, just explaining the change here ;-)

[aidantwoods]

This avoids the headerBag receiving headers that it already has when headers are sent back to it (though the httpAdapter should still (as it currently does) remove headers from its header source before sending it new ones – for the same duplication reasons).

[aidantwoods]

@franzliedke

What's left is now dealing with getHeaderAliases(). Can you come up with a test case that helps me implement this feature?

I've replaced getHeaderAliases with headerExists. getHeaderAliases was added before headers were stored case insensitively, so the concept of aliases when removing made a bit more sense ;p

Test case for headerExists would be setting a header with the HttpAdapter (via sendHeaders) and checking the bool response on ->removeHeader for headers that are known to exist/known not to. This to see if it can gauge information about the current headers in the HttpAdapter's header source at call time (so before a header import is run).

[franzliedke]

To me, it more and more looks like the better way to fix this trouble is by being liberal in what we accept: don't raise exceptions on invalid types, but convert them to the appropriate types where possible. In the example you posted, this would result in the string 'Array' - which obviously doesn't make sense, but would be the result of a programmer error anyway, so it's hard to expect good results anyway. ;)

As for headerExists() and getHeaderAliases(), things seem okay now. Or is anything missing from this getting merged?

My plan is still to replace $this->headers with the header bag - this will simplify the code drastically. But it's also yet another big change, so I'd prefer to do that in a separate pull request, building on top of this one. After that, I would also like to replace the done method with something along the lines of apply(AdapterInterface $adapter) - that would make all of this less stateful, and it would mean we don't need to inject any adapter implementation into the constructor.

tl;dr This is done, right?

[aidantwoods]

My plan is still to replace $this->headers with the header bag - this will simplify the code drastically.

👍 would love to see this done – I think the only thing that is really left to implement is the header value deconstruction (and changing the references in code to use the header bag, of course).

After that, I would also like to replace the done method with something along the lines of apply(AdapterInterface $adapter)

I was thinking about something along those lines too actually – would be useful if you wanted to say, use the string adapter to get the list of headers, and send them off to a framework/PHPs list.

it would mean we don't need to inject any adapter implementation into the constructor

Yup! That may be a good change – though I suppose if we don't have the adapter implemented in the constructor, then it would make it impossible for SecureHeaders to probe the HttpAdapter's source at configuration time (so headerExists() won't be able give a useful return value). Though maybe the return value isn't that useful anyway (since it is possible for a header not to be found at call time, but still removed – since removeHeader will queue the header for removal). Maybe I'll just make the behaviour identical to removeCookie, where void is returned, but the behaviour is probably clearer.

To me, it more and more looks like the better way to fix this trouble is by being liberal in what we accept: don't raise exceptions on invalid types

Hmm, I did consider doing that – though I kind of already do accept as many types as I can where possible e.g.

SecureHeaders/SecureHeaders.php

Lines 593 to 603 in 180913f

	public function hsts(
	$maxAge = 31536000,
	$subdomains = false,
	$preload = false
	) {
	$this->assertTypes(array('int|string' => array($maxAge)));
	$this->hsts['max-age'] = $maxAge;
	$this->hsts['subdomains'] = ($subdomains == true);
	$this->hsts['preload'] = ($preload == true);
	}

int or string is valid for the max-age value, and (something you'll see mostly everywhere) booleans are just loosely casted with no restriction on type.

For types where nothing reasonable can be inferred, if we don't throw an exception what is the alternative? Do we cast (leading to a strange result that may be hard for the programmer to track down the cause of), or do we silently fail (making it hard to track down why a function isn't doing anything)?

I'd love to just check types myself and blame the programmer for anything bad that happens when they use the wrong one, but there are more than a few legitimate situations where PHP is particularly unhelpful with the so called "type-juggling", e.g. '0e123' == '0' returns true because these two strings are compared as integers (which cast to 0) when using a loose comparison (types are "juggled" even when they are the same). But then '0e123' == '' returns false, even though if both strings are casted as integers you get 0 again.

TL;DR Yup, I think this is done. I think I'd still like to add the custom exceptions back in, but that's maybe best left for later once the HttpAdapter implementation is finalised, and perhaps sending the headers on exception should default to off?

Give me a couple hours (should be able to get some final testing done within then) and will merge 😄