2.3.0 returns wrong content-type for built-in javascript libraries

[veksh]

Bug Report

I've installed 2.3.0 in a VM with port 9090 forwarded to localhost, and
web interface does not work as expected: no javascript libraries are loaded (2.2.1 is OK).
On closer inspection, error console has 14 errors like

Refused to execute script from 'http://localhost:9090/static/vendor/js/jquery.min.js?v=290d71791a507a5057b9a099c9d48703d86dc941' because its MIME type ('text/x-js') is not executable, and strict MIME type checking is enabled.

so most probably Content-Type header for built-in javascript libraries is wrong.

Command-line is

/usr/local/bin/prometheus --config.file=/usr/local/etc/prometheus.yml --storage.tsdb.path=/var/lib/prometheus/data/ --web.console.templates=/usr/local/share/prometheus/console --web.console.libraries=/usr/local/share/prometheus/console_libraries --log.level=debug

Environment

• System information:

  Linux 3.0.101-108.38-default x86_64

• Prometheus version:

  prometheus, version 2.3.0 (branch: HEAD, revision: 290d717)
  build user: root@d539e167976a
  build date: 20180607-08:46:54
  go version: go1.10.2

• Prometheus configuration file:

global:
  scrape_interval:     15s
  evaluation_interval: 15s
  scrape_timeout:      10s
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']

[krasi-georgiev]

that is probably because of this commit
c207920

[fabxc]

@krasi-georgiev any further info how that commit causes this?

[brian-brazil]

It's likely the nosniff.

[fabxc]

Okay, what was the motivation for adding those headers? Were we exposed to any security risk?

[brian-brazil]

General security hardening was the motivation.

[fabxc]

Yes, but what are the risks those headers are fixing? I personally haven't seen them before and don't know what they do and how they are causing this issue. I'm all for hardening, but would be good to understand what we are hardening against and how.

[hasso]

While reverting security headers commit temporarily is OK as hotfix, the real problem is the Content-Type. text/x-js and similar ones are obsolete for more than 10 years and application/javascript must be used.

https://tools.ietf.org/html/rfc4329

[amorken]

I tried to reproduce this, but I am not able to given the Prometheus versions and Go versions mentioned in the issue. Built from a clean git checkout, run locally, nothing in between.

Startup and startup log:

level=info ts=2018-06-26T08:54:22.22102435Z caller=main.go:222 msg="Starting Prometheus" version="(version=2.3.0, branch=HEAD, revision=290d71791a507a5057b9a099c9d48703d86dc941)"
level=info ts=2018-06-26T08:54:22.221073818Z caller=main.go:223 build_context="(go=go1.10.2, user=anders@gaffa, date=20180626-08:52:49)"
level=info ts=2018-06-26T08:54:22.221090754Z caller=main.go:224 host_details="(Linux 4.15.0-22-lowlatency #24~16.04.1-Ubuntu SMP PREEMPT Fri May 18 10:50:50 UTC 2018 x86_64 gaffa (none))"
level=info ts=2018-06-26T08:54:22.22110483Z caller=main.go:225 fd_limits="(soft=65535, hard=65535)"
level=info ts=2018-06-26T08:54:22.221590661Z caller=web.go:426 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2018-06-26T08:54:22.221587546Z caller=main.go:514 msg="Starting TSDB ..."
level=info ts=2018-06-26T08:54:22.22508954Z caller=main.go:524 msg="TSDB started"
level=info ts=2018-06-26T08:54:22.225120019Z caller=main.go:603 msg="Loading configuration file" filename=prometheus.yml
level=info ts=2018-06-26T08:54:22.225182115Z caller=main.go:500 msg="Server is ready to receive web requests."

Test request with header response:

anders@gaffa:~$ curl -si 'localhost:9090/static/vendor/js/jquery.min.js?v=290d71791a507a5057b9a099c9d48703d86dc941'|head -11
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 86671
Content-Security-Policy: frame-ancestors 'self'
Content-Type: application/javascript
Last-Modified: Fri, 11 May 2018 16:21:47 GMT
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block
Date: Tue, 26 Jun 2018 08:57:06 GMT

As we see, the Content-Type header is present and correct.

I tested with go1.10.1 and go1.10.3 too, no change.

Testing v2.3.1 with go1.10.3:

anders@gaffa:~/gocode/src/github.com/prometheus/prometheus$ ./prometheus --web.console.libraries=$PWD/console_libraries --web.console.templates=/home/anders/gocode/src/github.com/prometheus/prometheus/consoles/
level=info ts=2018-06-26T09:10:30.470347174Z caller=main.go:222 msg="Starting Prometheus" version="(version=2.3.1, branch=HEAD, revision=188ca45bd85ce843071e768d855722a9d9dabe03)"
level=info ts=2018-06-26T09:10:30.470426199Z caller=main.go:223 build_context="(go=go1.10.3, user=anders@gaffa, date=20180626-09:06:46)"
level=info ts=2018-06-26T09:10:30.470461491Z caller=main.go:224 host_details="(Linux 4.15.0-22-lowlatency #24~16.04.1-Ubuntu SMP PREEMPT Fri May 18 10:50:50 UTC 2018 x86_64 gaffa (none))"
level=info ts=2018-06-26T09:10:30.470491136Z caller=main.go:225 fd_limits="(soft=65535, hard=65535)"
level=info ts=2018-06-26T09:10:30.471432619Z caller=web.go:415 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2018-06-26T09:10:30.471432004Z caller=main.go:514 msg="Starting TSDB ..."
level=info ts=2018-06-26T09:10:30.476634786Z caller=main.go:524 msg="TSDB started"
level=info ts=2018-06-26T09:10:30.47667456Z caller=main.go:603 msg="Loading configuration file" filename=prometheus.yml
level=info ts=2018-06-26T09:10:30.476750787Z caller=main.go:500 msg="Server is ready to receive web requests."

[...]

anders@gaffa:~$ curl -si 'localhost:9090/static/vendor/js/jquery.min.js?v=290d71791a507a5057b9a099c9d48703d86dc941'|head -7
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 86671
Content-Type: application/javascript
Last-Modified: Fri, 11 May 2018 16:21:47 GMT
Date: Tue, 26 Jun 2018 09:10:42 GMT

The security headers have been removed as referenced previously in the ticket, and the Content-Type header remains correct.
Am I missing some repro steps, or is something odd in the original environment?

@veksh, could you please share some more information about how you perform the forwarding from your client environment to your VM, and perhaps also describe your client setup, i.e. browser name and version?

Thanks,
-AndersM

[veksh]

With current 2.3.1 release (default config, just unpacked and started) I'm seeing different results on different platforms:

1. centos 7: application/javascript

[alex@v-centos7 ~]$ curl -si 'localhost:9090/static/vendor/js/jquery.min.js?v=290d71791a507a5057b9a099c9d48703d86dc941'|head -7
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 86671
Content-Type: application/x-javascript
Last-Modified: Fri, 11 May 2018 16:21:47 GMT
Date: Tue, 26 Jun 2018 10:03:27 GMT
[alex@v-centos7 ~]$ grep PRETTY_NAME /etc/os-release
PRETTY_NAME="CentOS Linux 7 (Core)"
[alex@v-centos7 ~]$ uname -a
Linux v-centos7.m1.maxidom.ru 3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

2. suse linux enterprise 11.4: text/x-js

alex@v-test:~$ curl -si 'localhost:9090/static/vendor/js/jquery.min.js?v=290d71791a507a5057b9a099c9d48703d86dc941'|head -7
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 86671
Content-Type: text/x-js; charset=utf-8
Last-Modified: Fri, 11 May 2018 16:21:47 GMT
Date: Tue, 26 Jun 2018 10:04:38 GMT
alex@v-test:~$ grep PRETTY_NAME /etc/os-release
PRETTY_NAME="SUSE Linux Enterprise Server 11 SP4"
alex@v-test:~$ uname -a
Linux v-test 3.0.101-108.38-default #1 SMP Mon Apr 16 12:16:19 UTC 2018 (3dac46e) x86_64 x86_64 x86_64 GNU/Linux

W/o security headers Google Chrome is OK with both.

[amorken]

Ah. Right, thanks, @veksh, those differences between distros are great data points.
For the record, my testing was performed on Ubuntu 16.04.

Prometheus uses https://golang.org/pkg/net/http/#ServeContent for static files. ServeContent uses https://golang.org/pkg/mime/#TypeByExtension - which in turn references /etc/mime.types.

On Ubuntu this file maps .js to application/json - I'm betting CentOS and SLES contain mappings that correspond with your observations - or possibly no mapping on CentOS, as go's mime type package contains a default mapping from .js to application/x-javascript

Both of those distros are kinda ancient, which sort of explains why these mappings are not up to date with today's conventions.

[veksh]

On SLES /etc/mime.types correctly maps js to application/javascript:

alex@v-test:~$ grep javascript /etc/mime.types 
application/javascript js

and there are no Apache mappings on that host

alex@v-test:~$ ls -l /etc/apache2/
total 4
drwxr-xr-x 2 root root 4096 Feb  9  2015 conf.d
alex@v-test:~$ ls -l /etc/apache/
ls: cannot access /etc/apache/: No such file or directory

[amorken]

@veksh I suspect that if you grep for "text/x-js" instead of "javascript" you may find that the /etc/mime.types contains a mapping for that type to "js" later in the file.

Go's mime type loader simply scans the input files in order and adds things to a map, so later entries win. See https://golang.org/src/mime/type_unix.go and https://golang.org/src/mime/type.go for specifics

If you comment out the "text/x-js" entry you may get the behavior you expect.

[veksh]

@amorken you are absolutely right -- there are 2 entries for "js" in /etc/mime.types, second one is wrong and after editing it out Content-Type for js became application/javascript. Thanks!

[brian-brazil]

I'm not sure we should be depending on external (and apparently outdated) resources for something like this. Is there a better way to do this?

[amorken]

Well, we can "simply" provide the MIME types ourselves: amorken@fc6b6bf

It's not beautiful, but it should be functional. I can send a PR over and include the now-reverted c207920 commit too if there's interest.

-@m

[brian-brazil]

Is it just the static endpoint that's affected, or is this something we need to set more broadly?

[amorken]

Just the static endpoint. It is the only usage of http.ServeContent in the Prometheus tree, and that's where the automatic selection of Content-Type based on file extension happens.