update default image versions

[pgier]

Prometheus image should default to latest version in compatibility
matrix. Also updates Alertmanager and Thanos to latest stable versions.

[yeya24]

I already opened a similar PR #2901. But the CI always fails.

[pgier]

Ah sorry, I didn't see that one!

[lilic]

@yeya24 I think its because of the ordering of your vars. If you don't mind closing yours, as this one fixes a few more things. Hope thats okay? :) Thanks!

[pgier]

@lilic I don't think it's the ordering, since if that was the issue it would probably fail early during build, wouldn't it?

[yeya24]

@lilic Sure, I will close mine.

[lilic]

Yes, its a global var not inline defined, strange. 🤷‍♀

[pgier]

Looks like the same test failure we saw before, I'll try to investigate further.

[pgier]

I tracked the issue down to prometheus v2.8.0 and higher, and seems to be caused by a change in the error handling in the scrape client (prometheus/prometheus#5182). The issue is that the certificates are not available in the pod when the test is running, and in Prometheus 2.7.2 and lower, the invalid cert actually causes Prometheus to crash with nil pointer.

level=error ts=2020-02-04T22:23:17.800304565Z caller=scrape.go:147 component="scrape manager" scrape_pool=allns-y-promarbitraryfsacc-allowed-tls-file-q577gu-0/test/0 msg="Error creating HTTP client"
 err="unable to use specified CA cert /etc/ca-certificates/example-ca.pem: open /etc/ca-certificates/example-ca.pem: no such file or directory"               
panic: runtime error: invalid memory address or nil pointer dereference                                                                                       
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x669c12]                                                                                       
                                                                               
goroutine 416 [running]:                                                                                                                                      
net/http.(*Client).deadline(0x0, 0xc00009cf30, 0x40bb8f, 0xc000af5fb0)                                                                                        
        /usr/local/go/src/net/http/client.go:187 +0x22                                                                                                        
net/http.(*Client).do(0x0, 0xc000b55100, 0x0, 0x0, 0x0)                                                                                                       
        /usr/local/go/src/net/http/client.go:527 +0xab                                                                                                               
net/http.(*Client).Do(0x0, 0xc000b55100, 0x23, 0xc00099df70, 0x9)                                                                                             
        /usr/local/go/src/net/http/client.go:509 +0x35                                                                                                        
github.com/prometheus/prometheus/scrape.(*targetScraper).scrape(0xc000b03890, 0x1fd4a80, 0xc00007bb00, 0x1fb2780, 0xc0004d29a0, 0x0, 0x0, 0x0, 0x0)                           
        /app/scrape/scrape.go:471 +0x111                              
github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc000b07500, 0x6fc23ac00, 0x2540be400, 0x0) 
        /app/scrape/scrape.go:813 +0x487
created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
        /app/scrape/scrape.go:336 +0x45d 

And this allows the test to succeed for some reason. Prometheus 2.8.0 and later prints the error but keeps running, but I guess it's not able to scrape and the test just times out.

Anyway, this seems to be a problem in our tests and not an issue with Prometheus.

[brancz]

Could we double check whether an empty file would have the same result? I think it’s just the existence of a file that is referenced.

[pgier]

@brancz Looks like it needs a real certificate, the test fails with a timeout when using an empty file, and the Prometheus logs print an error.

level=error ts=2020-02-07T13:17:00.063Z caller=manager.go:123 component="scrape manager" msg="error creating new scrape pool" err="error creating HTTP client: unable to use specified CA cert /etc/ca-certificates/cert.pem" scrape_pool=allns-y-promarbitraryfsacc-allowed-tls-file-q5c261-0/test/0

[lilic]

thanks! lgtm