Alert posting does not seem to follow HTTP redirects

[quolix]

I try running prometheus on Mesos, where jobs get assigned dynamic ports. I have an http redirector that redirects requests to specific hostnames to the correct hosts and ports on Mesos via 303/307/308. I want to use that to make prometheus POST its alerts to the alert manager.

However it seems that the redirect is never followed, indicated by tcpdump and netstat not showing any TCP traffic to the actual alert manager port.

[juliusv]

Haven't looked deeply into this yet, but I'm wondering whether it has to do with this (redirects upon POST being problematic in HTTP in general because POST is not an idempotent operation): http://programmers.stackexchange.com/questions/99894/why-doesnt-http-have-post-redirect

[juliusv]

Indeed, it looks like when Go is following redirects and the method is POST or PUT, it does subsequent redirected requests via GET: https://github.com/golang/go/blob/master/src/net/http/client.go#L353-L355

Not sure why you don't see any followup request (not even GET) in tcpdump, but it seems that redirecting POSTs doesn't really work well in general. Would it be possible to use a method of redirection which is transparent to the client?

Reading the code, it seems it might be possible to change Go's default behavior here by setting a custom CheckRedirect method on the HTTP client, but I'm kind of wary setting that for everyone...

[quolix]

Yes, that was my reading of the client.go awell.

However, the spec seems to indicate that a 307 must not result in a changed method:
http://www.heise.de/netze/rfc/rfcs/rfc7231.shtml#page-58

Bug in client.go then?

[juliusv]

Interesting pointer. Yeah, the code does not make any differentiation for StatusTemporaryRedirect (307), so that looks buggy to me as well.

I found a bug reported about this already against Go, but the status is unclear - I left a comment: golang/go#7912

[juliusv]

(That still doesn't explain why there was no followup request at all - I guess you verified that your redirector returns 307 with the appropriate headers?)

[quolix]

Both curl and Firefox/Chrome are happy with the redirector's responses.

[juliusv]

@quolix FYI, golang/go#7912 has been fixed and the fix will be in Go 1.8. So once that's out and Prometheus uses Go 1.8, then this should work.

[brian-brazil]

We're on Go 1.8 now.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.