Skip to content
This repository has been archived by the owner on Sep 7, 2023. It is now read-only.

HTTP headers #715

Closed
dalf opened this issue Oct 2, 2016 · 7 comments · Fixed by #2295
Closed

HTTP headers #715

dalf opened this issue Oct 2, 2016 · 7 comments · Fixed by #2295

Comments

@dalf
Copy link
Contributor

dalf commented Oct 2, 2016

searx includes contents from others websites. The contents are escaped, trimed and all, but there is always a risk. The security could be improve using HTTP headers.

Some headers that could be added :

X-Frame-Options: SAMEORIGIN
when there is the image proxy (break the included videos, in this case a more precise).
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options

X-Content-Security-Policy "default-src 'self'; img-src 'self'; style-src 'self' 'unsafe-inline'; font-src 'self'; script-src 'self' 'unsafe-inline'; connect-src 'self';"
to be adjust when there are images, videos
https://content-security-policy.com/
https://report-uri.io/

X-Content-Type-Options: nosniff
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options

X-XSS-Protection "1; mode=block"
https://blogs.msdn.microsoft.com/ieinternals/2011/01/31/controlling-the-xss-filter/

X-Download-Options: noopen
https://msdn.microsoft.com/fr-fr/library/jj542450(v=vs.85).aspx#noOpenDirective

more or less out of topic
X-Robots-Tag: noindex, nofollow
http://robots-txt.com/x-robots-tag/
avoid to be indexed and then spam (useless for that purpose ?)

HSTS header ( see #62 )
https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security
https://hstspreload.appspot.com/

Should nginx / apache send the headers or should the python code should send them ? in case of X-Content-Security-Policy and X-Frame-Options the values can change according to the search options.

@dalf dalf added the security label Oct 2, 2016
@cy8aer
Copy link
Contributor

cy8aer commented Oct 6, 2016

IMHO it would be better to have the web servers create the headers if you want to have additional headers (like key pinning stuff). But because Apache/mod_uwsgi does not work with header add I hard code my headers in webapp.py by now which is not very easy because I need to edit webapp.py with every git pull. Is there an mod_proxy_uwsgi description for searx? mod_proxy_uwsgi makes web server side headers possible.

@dalf
Copy link
Contributor Author

dalf commented Oct 22, 2016

A solution about hard coded headers in webapp is settings.yml. But it would be messy to have header inside settings.yml and inside the HTTP server.
I agree it should be in the web server, so it is related to the installation.

@cy8aer
Copy link
Contributor

cy8aer commented Nov 2, 2016

after reverse proxying my uwsgi <-> apache connection by http all my headers are in apache now. So updating is easier now because there is no webapp.py patching anymore. Headers in apache is also good if you are intercepting by filtron (if it works) because then you do not have problems with loosing headers.

@asciimoo
Copy link
Member

It would be good to have decent defaults for http headers, but I'm not sure if the searx webapp is the right place for these headers. What if someone wants to change it for some reason? Should we add a configuration section for HTTP headers? Also, there is a possibility that adding new default headers may break someones existing configuration (CSP).

As a backward compatible solution, I'd suggest to do not turn these headers on by default, but allow admins to turn it on from settings.yml.

What do you think?

@dalf
Copy link
Contributor Author

dalf commented Oct 26, 2020

I'm afraid that off by default means "it doesn't exist" most of the time: if these headers would break something we would have feedback, but there are mostly seat belts (except the Cache-Control), so most of the time even turn off everything will be look good (same for a wrong TLS configuration).

But as you said, if some headers are enabled by default, it may conflict with some existing configurations.

The only way I see is a two phase activation:

  • add some headers, disabled by default, display a warning (where ?) that they will be enabled at one point in the future (to be defined in relative or absolute time).
  • enabled them by default later (in a new release ?).

Can be related to searx/searx-instances#70 :

  • searx-stats2 can check the headers (if there are multiple values for the same header, or if there is some missing headers according to the good practice).
  • we can provide an API to check an existing instance : a script can call this API (in the installation script for example). But there are a lot of external tutorials which described how to install searx : they won't be updated.

Does it worth it ? I'm not sure where to put the cursor.


Looking at https://github.com/searx/searx-docker/blob/master/Caddyfile (not updated since last year, may be it requires some updates ? may be some values are broken ?).

CSP

documentation: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy

value in searx-docker: https://github.com/searx/searx-docker/blob/7ad4d4903d0ce41de4094d5fe0fecda156b41eaa/Caddyfile#L83

The value of this header depends on the searx code (tile.openstreetmap.org hostname etc...)

I think a default value would be good : currently we can't change this value without issue.

X-XSS-Protection

The header seems deprecated (replaced by CSP) : https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection

To keep the backward compatibility for some old browsers, we can keep this header.

X-Frame-Options

The header seems deprecated (replaced by CSP) : https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options

To keep the backward compatibility for some old browsers, we can keep this header.

X-Content-Type-Options

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options

Not deprecated, seems good whatever the usage context

Feature-Policy

https://developer.mozilla.org/en-US/docs/Web/HTTP/Feature_Policy

It is a seat belt: searx doesn't use these features, if for some reason there is an XSS, at least this header will block usage of these features.

Here, my take is HTTP/2 allows to send a reference to some previous headers so it doesn't cost anything to add this header.

Referrer-Policy

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referrer-Policy

Seems good, it is redundant with the templates :

<meta name="referrer" content="no-referrer">

X-Robots-Tag

According to https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers :
The header is effectively equivalent to <meta name="robots" content="...">.

Cache-Control

Related to

I think a default value would be good at least for the static files ?

I'm not sure what to do for the index and preferences page ( searx-docker set the value of "no-cache, no-store", but some instances doesn't set this value.

@asciimoo
Copy link
Member

Good points. What about adding the headers that most probably can't cause backward incompatibility and creating a warning for adding the rest in the future?

@dalf
Copy link
Contributor Author

dalf commented Oct 28, 2020

Except Cache-Control for the /index and /preferences pages, I would declare all these headers in settings.yml , section server.headers for example.

The main issue is the duplicate headers: the reverse proxy most probably will add or replace some headers too. Replace should not be a problem (but it produces an expect configuration), add may create a wrong / unsecure configuration.

Adding headers in settings.yml is related to #1209 : settings.yml won't be updated blindly if it contains some specific configuration (like the secret key, the engine tokens, etc...)

@kvch kvch closed this as completed in #2295 Nov 8, 2020
v1.0.0 automation moved this from Milestone #2 - Backend to Done Nov 8, 2020
kvch added a commit that referenced this issue Nov 8, 2020
[enh] add default http headers - closes #715
mikeri pushed a commit to mikeri/searx that referenced this issue Mar 7, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
No open projects
v1.0.0
  
Done
Development

Successfully merging a pull request may close this issue.

3 participants