Web UI returns `400 Bad Request` if there is an empty header value #4917

sorenstoutner · 2021-04-13T15:37:30Z

Before you add a new report, we ask you kindly to acknowledge the following:

I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
I have searched the existing issues and I am convinced that mine is new.

The Disable HTTP_REFERER enforcement check option in System > Settings > Enforcement does not make the web interface work when no referer header is sent. Using a browser that does not send the referer header causes a 400 Bad Request error.

I am new to OPNsense from a longtime user of pfSense. As such, my experience is with a new 21.1.4 installation yesterday. There are forum posts that indicate this problem began with the 21.1.1 release.

https://forum.opnsense.org/index.php?topic=21437.msg100640#msg100640

https://forum.opnsense.org/index.php?topic=21521.msg101158#msg101158

In the first blog post, the user indicates the problem manifests with Safari on iOS, but not with other browsers.

In my case, I am the developer of Privacy Browser, which does not send the referer header, even for same-domain requests. Because the referer header is security theater and has significant privacy implications, there is no way for a user to enable it in Privacy Browser. However, if I edit the code to send the referer header, then Privacy Browser can load the website just fine (as long as JavaScript is enabled, which is to be expected).

The text was updated successfully, but these errors were encountered:

AdSchellevis · 2021-04-13T15:49:19Z

It's quite unlikely that skipping the check leads to a 400, maybe there's some other issue underneath, but when checked it only prevents OPNsense to spawn an error about mismatches:

core/src/etc/inc/auth.inc

Lines 111 to 183 in 3f3ebd9

    
           if (function_exists("display_error_form") && !isset($config['system']['webgui']['nohttpreferercheck'])) { 
        
               if (isset($_SERVER['HTTP_REFERER'])) { 
        
                   if (file_exists('/tmp/setupwizard_lastreferrer')) { 
        
                       if ($_SERVER['HTTP_REFERER'] == file_get_contents('/tmp/setupwizard_lastreferrer')) { 
        
                           unlink('/tmp/setupwizard_lastreferrer'); 
        
                           header("Refresh: 1; url=index.php"); 
        
                           echo "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\"\n        \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">"; 
        
                           echo "<html><head><title>" . gettext("Redirecting...") . "</title></head><body>" . gettext("Redirecting to the dashboard...") . "</body></html>"; 
        
                           exit; 
        
                       } 
        
                   } 
        
                   $found_host = false; 
        
                   $referrer_host = parse_url($_SERVER['HTTP_REFERER'], PHP_URL_HOST); 
        
                   $referrer_host = str_replace(array("[", "]"), "", $referrer_host); 
        
                   if ($referrer_host) { 
        
                       if ( 
        
                           strcasecmp($referrer_host, $config['system']['hostname'] . "." . $config['system']['domain']) == 0 || 
        
                           strcasecmp($referrer_host, $config['system']['hostname']) == 0 
        
                       ) { 
        
                           $found_host = true; 
        
                       } 
        
                       if (!empty($config['system']['webgui']['althostnames']) && !$found_host) { 
        
                           $althosts = explode(" ", $config['system']['webgui']['althostnames']); 
        
                           foreach ($althosts as $ah) { 
        
                               if (strcasecmp($referrer_host, $ah) == 0) { 
        
                                   $found_host = true; 
        
                                   break; 
        
                               } 
        
                           } 
        
                       } 
        
                       if (isset($config['dyndnses']['dyndns']) && !$found_host) { 
        
                           foreach ($config['dyndnses']['dyndns'] as $dyndns) { 
        
                               if (strcasecmp($dyndns['host'], $referrer_host) == 0) { 
        
                                   $found_host = true; 
        
                                   break; 
        
                               } 
        
                           } 
        
                       } 
        
                       if (isset($config['dnsupdates']['dnsupdate']) && !$found_host) { 
        
                           foreach ($config['dnsupdates']['dnsupdate'] as $rfc2136) { 
        
                               if (strcasecmp($rfc2136['host'], $referrer_host) == 0) { 
        
                                   $found_host = true; 
        
                                   break; 
        
                               } 
        
                           } 
        
                       } 
        
                       if (!$found_host) { 
        
                           $found_host = isAuthLocalIP($referrer_host); 
        
                           if ($referrer_host == "127.0.0.1" || $referrer_host == "localhost") { 
        
                               // allow SSH port forwarded connections and links from localhost 
        
                               $found_host = true; 
        
                           } 
        
                       } 
        
                   } 
        
                   if ($found_host == false) { 
        
                       if (!security_checks_disabled()) { 
        
                           display_error_form('501', sprintf( 
        
                               gettext('The HTTP_REFERER "%s" does not match the predefined settings. You can disable this check if needed under System: Settings: Administration.'), 
        
                               html_safe($_SERVER['HTTP_REFERER']) 
        
                           )); 
        
                           exit; 
        
                       } 
        
                       $security_passed = false; 
        
                   } 
        
               } else { 
        
                   $security_passed = false; 
        
               } 
        
           }

If there's an error on the dashboard, it might shed more light on the issue.

sorenstoutner · 2021-04-13T15:59:02Z

My guess is that it is indeed some issue underneath, possibly with a system package inherited from upstream. I do not see any errors on the dashboard that would be helpful in troubleshooting.

You can test using Privacy Browser on Android if you like, or using Safari on iOS according to the forum post. If you do use Privacy Browser, you will need to enable JavaScript (see the guide inside the app for instructions). As I indicated in the original post, if I make one change in the code to enable the referer header, then it works fine.

sorenstoutner · 2021-04-13T16:06:13Z

If it is helpful for you, the 400 Bad Request error is also shown when the referer header is sent but JavaScript is disabled. This could indicate that something involved in the processing of the JavaScript added a dependency on the referer header with the 21.1.1 release.

kulikov-a · 2021-04-13T19:28:34Z

hi!
can confirm (if related): when debugging is enabled on lighttpd and accessing GUI from safari new tab on iPhone it produces an entry in the log: lighttpd[24119] | (request.c.359) invalid header value -> 400
looks like upstream issue for me at first glance: lighttpd/lighttpd1.4@262561f

fichtner · 2021-04-14T06:35:38Z

/CC the one and only @gstrauss :)

gstrauss · 2021-04-14T07:14:56Z

ISTR recently having a conversation somewhere about why a client sending an empty request header field-value (after removing leading and trailing whitespace) was poor behavior.

gstrauss · 2021-04-14T07:36:50Z

@kulikov-a found it, above. lighttpd/lighttpd1.4@262561f is on lighttpd master, but not yet part of an official release.

In discussion on IRC in #lighttpd, I pointed to https://redmine.stoutner.com/issues/37 where one poster wrote

Using a null value is probably not the best approach. Keys without values are considered malformed headers by some web servers leading to HTTP 400 errors.

The lighttpd patch in lighttpd/lighttpd1.4@262561f will be part of the next lighttpd release, lighttpd 1.4.60, likely released some time in the next few months, and ignores empty (non-HTTP/2-pseudo) request headers instead of rejecting them.

gstrauss · 2021-04-14T07:49:24Z

@sorenstoutner, while you may argue that ABNF in RFC 7230 for field values does allow an empty field-value, I can separately argue that servers may treat strange behavior as an attack and may choose to reject such requests.

The requests from Privacy Browser would be less suspicious if the Referer header were either omitted, or non-empty. However, if you choose to maintain such behavior in Privacy Browser, that might be used to fingerprint and reject the usage of Privacy Browser. Something to consider WRT your posts in https://redmine.stoutner.com/issues/37

sorenstoutner · 2021-04-14T14:39:12Z

Just to be clear, Privacy Browser does not send an empty Referer header. Privacy Browser sends no Referer header. You can see this at https://www.stoutner.com/the-referer-header/.

Due to limitations in Android's System WebView, Privacy Browser currently sends an empty X-Requested-With header, which is what is being discussed in https://redmine.stoutner.com/issues/37. This header will be entirely removed in the 4.x series of Privacy Browser with the release of Privacy WebView.

When I tested sending the Referer header the change also populated the X-Requested-With header again, so that was the root of the problem. It is my fault for not separating the two during testing. (Also, I had forgotten that some websites don't like empty headers, despite previously discussing it in a bug report, as this is the first time I have experienced the issue in my own browsing.) I do not know what the root of the problem is with Safari. Perhaps they are sending an empty header as well, or perhaps it is something else.

A web server can do whatever they like. Trust me when I say that Privacy Browser already does a lot of things much more invasive than sending an empty header, and that is only going to increase in the future. For example, wait until the default becomes to send no user agent. ;) However, I would consider it silly for a web server to think itself "under attack" because there is a header with no value. There are a lot of real attacks in the world that web servers need to protect themselves against, and this isn't one of them..

OPNsense-bot · 2021-10-10T14:05:09Z

This issue has been automatically timed-out (after 180 days of inactivity).

For more information about the policies for this repository,
please read https://github.com/opnsense/core/blob/master/CONTRIBUTING.md for further details.

If someone wants to step up and work on this issue,
just let us know, so we can reopen the issue and assign an owner to it.

gstrauss · 2021-10-10T23:22:57Z

The patch in lighttpd for HTTP/2 request processing to ignore empty HTTP request headers (if not HTTP/2 pseudo-headers) was posted for reference at https://redmine.lighttpd.net/boards/2/topics/9720 and committed to lighttpd in https://git.lighttpd.net/lighttpd/lighttpd1.4/commit/262561fae1aad08550870efa4234c6c5659bd680

The patch is part of lighttpd 1.4.60, released 3 Oct

AdSchellevis added the support Community support label Apr 13, 2021

sorenstoutner changed the title ~~Disabling Referer Header Currently Broken~~ Disabling Referer Header Enforcement Currently Broken Apr 13, 2021

sorenstoutner changed the title ~~Disabling Referer Header Enforcement Currently Broken~~ Web UI returns 400 Bad Request if there is an empty header value Apr 14, 2021

gstrauss mentioned this issue Jun 13, 2021

Don't set X-XSRF-TOKEN when the user isn't logged in umputun/remark42#1043

Merged

OPNsense-bot closed this as completed Oct 10, 2021

OPNsense-bot added the help wanted Contributor missing / timeout label Oct 10, 2021

fichtner added upstream Third party issue and removed help wanted Contributor missing / timeout support Community support labels Oct 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Web UI returns `400 Bad Request` if there is an empty header value #4917

Web UI returns `400 Bad Request` if there is an empty header value #4917

sorenstoutner commented Apr 13, 2021

AdSchellevis commented Apr 13, 2021

sorenstoutner commented Apr 13, 2021

sorenstoutner commented Apr 13, 2021

kulikov-a commented Apr 13, 2021 •

edited

Loading

fichtner commented Apr 14, 2021

gstrauss commented Apr 14, 2021

gstrauss commented Apr 14, 2021 •

edited

Loading

gstrauss commented Apr 14, 2021

sorenstoutner commented Apr 14, 2021 •

edited

Loading

OPNsense-bot commented Oct 10, 2021

gstrauss commented Oct 10, 2021

Web UI returns 400 Bad Request if there is an empty header value #4917

Web UI returns 400 Bad Request if there is an empty header value #4917

Comments

sorenstoutner commented Apr 13, 2021

AdSchellevis commented Apr 13, 2021

sorenstoutner commented Apr 13, 2021

sorenstoutner commented Apr 13, 2021

kulikov-a commented Apr 13, 2021 • edited Loading

fichtner commented Apr 14, 2021

gstrauss commented Apr 14, 2021

gstrauss commented Apr 14, 2021 • edited Loading

gstrauss commented Apr 14, 2021

sorenstoutner commented Apr 14, 2021 • edited Loading

OPNsense-bot commented Oct 10, 2021

gstrauss commented Oct 10, 2021

Web UI returns `400 Bad Request` if there is an empty header value #4917

Web UI returns `400 Bad Request` if there is an empty header value #4917

kulikov-a commented Apr 13, 2021 •

edited

Loading

gstrauss commented Apr 14, 2021 •

edited

Loading

sorenstoutner commented Apr 14, 2021 •

edited

Loading