Skip to content

/prometheus

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Promql won't time-out queries in matrixSelector #4288

Closed
jacksontj opened this Issue Jun 19, 2018 · 11 comments

Comments

Assignees
No one assigned
Labels
component/promql kind/bug
Projects
None yet
Milestone
No milestone
3 participants
@jacksontj @brian-brazil @juliusv
@jacksontj
Copy link
Contributor

jacksontj commented Jun 19, 2018 •
edited

Bug Report

What did you do?
Someone sent a large query such as {__name__=~".+"}[5d] to our prometheus host.

What did you expect to see?
I expect the query to timeout before completing, while consuming some memory and CPU along the way.

What did you see instead? Under which circumstances?
The query runs until it exhausts all available memory on the host and OOM killer get is. I have set it up in a repro env locally and the query has been running for ~60m (its still going) even though I have the timeout set to the default 2m.

Environment

  • Prometheus version:

2.2

jacksontj added a commit to jacksontj/prometheus that referenced this issue Jun 19, 2018

@jacksontj
Check for timeout in each iteration of matrixSelector 
Fixes prometheus#4288
9ea87a8
@brian-brazil

This comment has been minimized.

Sign in to view
Copy link
Member

brian-brazil commented Jun 19, 2018

Thanks for the report. I'm aware of this, and it's something I intend to cleanup now that the big PromQL change is in.

@brian-brazil brian-brazil added kind/bug component/promql labels Jun 19, 2018

@jacksontj jacksontj referenced this issue Jun 19, 2018

Closed

Promql won't time-out queries in expandSeriesSet #4289

@juliusv

This comment has been minimized.

Sign in to view
Copy link
Member

juliusv commented Jun 19, 2018

@brian-brazil Something like 29294b2? Or probably you were already thinking of something more holistic?
@jacksontj

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

jacksontj commented Jun 19, 2018 •
edited

@juliusv that would cover #4289 but not this issue. This matrixSelector is after the expandSeriesSet is called.
@juliusv

This comment has been minimized.

Sign in to view
Copy link
Member

juliusv commented Jun 19, 2018

@jacksontj Have you verified somehow that the bulk of the time (and memory) here is spent after the expansion of the series set for the matrix selector (

prometheus/promql/engine.go

Line 511 in 404abe0

n.series, err = expandSeriesSet(set)
), in the actual evaluation?
@juliusv

This comment has been minimized.

Sign in to view
Copy link
Member

juliusv commented Jun 19, 2018

(of course there should still be timeout handling in the other pieces of the engine)
@jacksontj

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

jacksontj commented Jun 19, 2018

In the example I have both spots are very time intensive (at least on 2.2). Meaning it doesn't time out in either place (I can see it spend lots of time in both). I'm verifying master right now on my test host -- should have an answer in ~5-10m (prom takes FOREVER to start/stop)
@juliusv

This comment has been minimized.

Sign in to view
Copy link
Member

juliusv commented Jun 19, 2018

Yup, makes sense.

(prom takes FOREVER to start/stop)

Whoa, how come?
@jacksontj

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

jacksontj commented Jun 19, 2018

From the little looking I've done its TSDB-- we have ~5TB of data on this box. I'll probably spend some time looking into that in the future, but for another issue/pr :)
@juliusv

This comment has been minimized.

Sign in to view
Copy link
Member

juliusv commented Jun 19, 2018

Oh yeah, I would assume it's the TSDB, but which part would be interesting. Anyway, let's leave that one for another issue :)
@brian-brazil

This comment has been minimized.

Sign in to view
Copy link
Member

brian-brazil commented Jun 19, 2018

Something like 29294b2?

No, it should be in the range vector function evaluation. There's math to be done to see how often we're required to do this, and it turns out due to edge cases (really long durations) to be at every time series.

@jacksontj jacksontj referenced this issue Jun 21, 2018

Merged

Check for timeout in each iteration of matrixSelector #4300

jacksontj added a commit to jacksontj/prometheus that referenced this issue Jun 21, 2018

@jacksontj
Check for timeout in each iteration of matrixSelector 
Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>

Fixes prometheus#4288
6f091ca

@brian-brazil brian-brazil closed this in #4300 Jun 21, 2018

brian-brazil added a commit that referenced this issue Jun 21, 2018

@jacksontj @brian-brazil
Check for timeout in each iteration of matrixSelector (#4300) 
Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>

Fixes #4288
Loading status checks…
a6dace8

jacksontj added a commit to jacksontj/prometheus that referenced this issue Jun 28, 2018

@jacksontj
Check for timeout in each iteration of matrixSelector (prometheus#4300) 
Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>

Fixes prometheus#4288
c95b795

brian-brazil added a commit that referenced this issue Jul 11, 2018

@jacksontj @brian-brazil
Check for timeout in each iteration of matrixSelector (#4300) 
Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>

Fixes #4288
86239ee

mknapphrt added a commit to mknapphrt/prometheus that referenced this issue Jul 26, 2018

@mknapphrt
Return whatever data is available when there is a failed remote read 
Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>

Bubble up errors to promql from populating iterators (prometheus#4136)

This changes the Walk/Inspect API inside the promql package to bubble
up errors. This is done by having the inspector return an error (instead
of a bool) and then bubbling that up in the Walk. This way if any error
is encountered in the Walk() the walk will stop and return the error.
This avoids issues where errors from the Querier where being ignored
(causing incorrect promql evaluation).

Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>

Fixes prometheus#4136

*: cut v2.3.0

Signed-off-by: Fabian Reinartz <freinartz@google.com>

Update changelog

Signed-off-by: Fabian Reinartz <freinartz@google.com>

limit size of POST requests against remote read endpoint (prometheus#4239)

This commit fixes a denial-of-service issue of the remote
read endpoint. It limits the size of the POST request body
to 32 MB such that clients cannot write arbitrary amounts
of data to the server memory.

Fixes prometheus#4238

Signed-off-by: Andreas Auernhammer <aead@mail.de>

Update example console template for node exporter 0.16.0 (prometheus#4208)

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

Makefile: update .PHONY target (prometheus#4234)

Makefile: update .PHONY target

* Move .PHONY declarations near their targets

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

Add prompb/README (prometheus#4222)

Signed-off-by: Henri DF <henridf@gmail.com>

discovery/file: fix logging (prometheus#4178)

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

web: remove security headers

Signed-off-by: Fabian Reinartz <freinartz@google.com>

config: set target group source index during unmarshalling (prometheus#4245)

* config: set target group source index during unmarshalling

Fixes issue prometheus#4214 where the scrape pool is unnecessarily reloaded for a
config reload where the config hasn't changed.  Previously, the discovery
manager changed the static config after loading which caused the in-memory
config to differ from a freshly reloaded config.

Signed-off-by: Paul Gier <pgier@redhat.com>

* [issue prometheus#4214] Test that static targets are not modified by discovery manager

Signed-off-by: Paul Gier <pgier@redhat.com>

Log the line when failing a PromQL test. (prometheus#4272)

Signed-off-by: Alin Sinpalean <alin.sinpalean@gmail.com>

web: restore old path prefix behavior

Signed-off-by: Fabian Reinartz <freinartz@google.com>

kubernetes_sd: fix namespace filtering (prometheus#4273)

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

fix minor issues in custom SD example (prometheus#4278)

Signed-off-by: Callum Styan <callumstyan@gmail.com>

federation: nil pointer deference when using remove read

```
level=error ts=2018-06-13T07:19:04.515149169Z caller=stdlib.go:89 component=web caller="http: panic serving [::1" msg="]:56202: runtime error: invalid memory address or nil pointer dereference"
level=error ts=2018-06-13T07:19:04.516199547Z caller=stdlib.go:89 component=web caller="http: panic serving [::1" msg="]:56204: runtime error: invalid memory address or nil pointer dereference"
level=error ts=2018-06-13T07:19:04.51717692Z caller=stdlib.go:89 component=web caller="http: panic serving [::1" msg="]:56206: runtime error: invalid memory address or nil pointer dereference"
level=error ts=2018-06-13T07:19:04.564952878Z caller=stdlib.go:89 component=web caller="http: panic serving [::1" msg="]:56208: runtime error: invalid memory address or nil pointer dereference"
level=error ts=2018-06-13T07:19:04.566575791Z caller=stdlib.go:89 component=web caller="http: panic serving [::1" msg="]:56210: runtime error: invalid memory address or nil pointer dereference"
level=error ts=2018-06-13T07:19:04.567106063Z caller=stdlib.go:89 component=web caller="http: panic serving [::1" msg="]:56212: runtime error: invalid memory address or nil pointer dereference"
```

When remove read is enabled, federation will call `q.Select(nil, mset...)`
which will break remote reads because it currently doesn't handle empty
SelectParams.

Signed-off-by: Corentin Chary <c.chary@criteo.com>

Extend API tests to cover remote read API.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Review feedback.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

spelling.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

web: restore old path prefix behavior

Signed-off-by: Fabian Reinartz <freinartz@google.com>

kubernetes_sd: fix namespace filtering (prometheus#4273)

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

Avoid infinite loop on duplicate NaN values. (prometheus#4275)

Fixes prometheus#4254

NaNs don't equal themselves, so a duplicate NaN would
always hit the break statement and never get popped.

We should not be returning multiple data point for the same
timestamp, so don't compare values at all.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

config: set target group source index during unmarshalling (prometheus#4245)

* config: set target group source index during unmarshalling

Fixes issue prometheus#4214 where the scrape pool is unnecessarily reloaded for a
config reload where the config hasn't changed.  Previously, the discovery
manager changed the static config after loading which caused the in-memory
config to differ from a freshly reloaded config.

Signed-off-by: Paul Gier <pgier@redhat.com>

* [issue prometheus#4214] Test that static targets are not modified by discovery manager

Signed-off-by: Paul Gier <pgier@redhat.com>

discovery/file: fix logging (prometheus#4178)

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

federation: nil pointer deference when using remove read

```
level=error ts=2018-06-13T07:19:04.515149169Z caller=stdlib.go:89 component=web caller="http: panic serving [::1" msg="]:56202: runtime error: invalid memory address or nil pointer dereference"
level=error ts=2018-06-13T07:19:04.516199547Z caller=stdlib.go:89 component=web caller="http: panic serving [::1" msg="]:56204: runtime error: invalid memory address or nil pointer dereference"
level=error ts=2018-06-13T07:19:04.51717692Z caller=stdlib.go:89 component=web caller="http: panic serving [::1" msg="]:56206: runtime error: invalid memory address or nil pointer dereference"
level=error ts=2018-06-13T07:19:04.564952878Z caller=stdlib.go:89 component=web caller="http: panic serving [::1" msg="]:56208: runtime error: invalid memory address or nil pointer dereference"
level=error ts=2018-06-13T07:19:04.566575791Z caller=stdlib.go:89 component=web caller="http: panic serving [::1" msg="]:56210: runtime error: invalid memory address or nil pointer dereference"
level=error ts=2018-06-13T07:19:04.567106063Z caller=stdlib.go:89 component=web caller="http: panic serving [::1" msg="]:56212: runtime error: invalid memory address or nil pointer dereference"
```

When remove read is enabled, federation will call `q.Select(nil, mset...)`
which will break remote reads because it currently doesn't handle empty
SelectParams.

Signed-off-by: Corentin Chary <c.chary@criteo.com>

Review feedback.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

spelling.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Release 2.3.1

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

Timeout if populating iterators takes too long (prometheus#4291)

Right now promql won't time out a request if populating the iterators
takes a long time.

Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>

Fixes prometheus#4289

return error exit status in prometheus cli (prometheus#4296)

Signed-off-by: mikeykhalil <mikeyfkhalil@gmail.com>

Check for timeout in each iteration of matrixSelector (prometheus#4300)

Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>

Fixes prometheus#4288

Make TestUpdate() do some work (prometheus#4306)

Previously it would set no preconditions and check no postconditions,
as the `groups` member was empty.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

Add "omitempty" to some SD config YAML field tags (prometheus#4338)

Especially for Kubernetes SD, this fixes a bug where the rendered
configuration says "api_server: null", which when read back is not
interpreted as an un-set API server (thus the default is not applied).

Signed-off-by: Julius Volz <julius.volz@gmail.com>

travis: remove testing with go 1.x

Travis and CircleCI should use the same Go version(s).

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

Reduce CircleCI duplication (prometheus#4335)

Reduce the duplication of per-project specifics in the CircleCI config.
* Add docker repo variable, default to docker hub.
* Add make targets for docker push and tag latest.

Signed-off-by: Ben Kochie <superq@gmail.com>

fix the TestManagerReloadNoChange test (prometheus#4267)

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>

Reorder startup and shutdown to prevent panics. (prometheus#4321)

Start rule manager only after tsdb and config is loaded.
Stop rule manager before tsdb to avoid writing to closed storage.
Wait for any in-progress reloads to complete before shutting
down rule manager, so that rule manager doesn't get updated after
being shut down.

Remove incorrect comment around shutting down query enginge.
Log when config reload is completed.

Fixes prometheus#4133
Fixes prometheus#4262

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

discovery/kubernetes/ingress: add more tests

Signed-off-by: Dmitry Bashkatov <dbashkatov@gmail.com>

discovery/kubernetes/ingress: fix scheme discovery (Closes prometheus#4327)

Signed-off-by: Dmitry Bashkatov <dbashkatov@gmail.com>

discovery/kubernetes/ingress: remove unnecessary check

Signed-off-by: Dmitry Bashkatov <dbashkatov@gmail.com>

Fix markup in example. (prometheus#4351)

Signed-off-by: Marcin Owsiany <marcin@owsiany.pl>

fix the zookeper race (prometheus#4355)

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>

docs: added undocumented step api parameter format (prometheus#4360)

Update vendoring for tsdb (prometheus#4369)

This pulls in tsdb PRs 330 344 348 353 354 356

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

k8s SD: Fix "schema" -> "scheme" typo (prometheus#4371)

Signed-off-by: Julius Volz <julius.volz@gmail.com>

Fix missing 'msg' in remote storage adapter main.go .Log info message (prometheus#4377)

Signed-off-by: Peter Gallerani <peter.gallerani@gmail.com>

Don't forget to register query_duration_seconds{slice="queue_time"} (prometheus#4381)

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

docs: fix OpenStack SD for the hypervisor role

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

discovery/openstack: remove unneeded assignment

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

Bubble up errors to promql from populating iterators (prometheus#4136)

This changes the Walk/Inspect API inside the promql package to bubble
up errors. This is done by having the inspector return an error (instead
of a bool) and then bubbling that up in the Walk. This way if any error
is encountered in the Walk() the walk will stop and return the error.
This avoids issues where errors from the Querier where being ignored
(causing incorrect promql evaluation).

Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>

Fixes prometheus#4136

Timeout if populating iterators takes too long (prometheus#4291)

Right now promql won't time out a request if populating the iterators
takes a long time.

Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>

Fixes prometheus#4289

Check for timeout in each iteration of matrixSelector (prometheus#4300)

Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>

Fixes prometheus#4288

fix the zookeper race (prometheus#4355)

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>

return error exit status in prometheus cli (prometheus#4296)

Signed-off-by: mikeykhalil <mikeyfkhalil@gmail.com>

Reorder startup and shutdown to prevent panics. (prometheus#4321)

Start rule manager only after tsdb and config is loaded.
Stop rule manager before tsdb to avoid writing to closed storage.
Wait for any in-progress reloads to complete before shutting
down rule manager, so that rule manager doesn't get updated after
being shut down.

Remove incorrect comment around shutting down query enginge.
Log when config reload is completed.

Fixes prometheus#4133
Fixes prometheus#4262

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

Update vendoring for tsdb (prometheus#4369)

This pulls in tsdb PRs 330 344 348 353 354 356

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

Release 2.3.2

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

rules: Minor naming/comment cleanups (prometheus#4328)

Signed-off-by: Julius Volz <julius.volz@gmail.com>

Optimize PromQL aggregations (prometheus#4248)

* Compute hash of label subsets without creating a LabelSet first.

Signed-off-by: Alin Sinpalean <alin.sinpalean@gmail.com>

Add offset to selectParams (prometheus#4226)

* Add Start/End to SelectParams
* Make remote read use the new selectParams for start/end

This commit will continue sending the start/end time of the remote read
query as the overarching promql time and the specific range of data that
the query is intersted in receiving a response to is now part of the
ReadHints (upstream discussion in prometheus#4226).

* Remove unused vendored code

The genproto.sh script was updated, but the code wasn't regenerated.
This simply removes the vendored deps that are no longer part of the
codegen output.

Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>

Forbid rule-abiding robots from indexing. (prometheus#4266)

* Resolves github issue prometheus#4257

Signed-off-by: Martin Lee <martin@billforward.net>

Discovery consul service meta (prometheus#4280)

* Upgrade Consul client
* Add ServiceMeta to the labels in ConsulSD

Signed-off-by: Romain Baugue <romain.baugue@elwinar.com>

Fix some (valid) lint errors (prometheus#4287)

Signed-off-by: Julius Volz <julius.volz@gmail.com>

Update vendoring of Prometheus Go client (prometheus#4283)

This is to pickup changes from
prometheus/client_golang#414. It leads to
better error output in promtool.

Signed-off-by: Sneha Inguva <singuva@digitalocean.com>

Simplify BufferedSeriesIterator usage (prometheus#4294)

* Allow for BufferedSeriesIterator instances to be created without an underlying iterator, to simplify their usage.

Signed-off-by: Alin Sinpalean <alin.sinpalean@gmail.com>

Saner defaults and metrics for remote-write (prometheus#4279)

* Rename queueCapacity to shardCapacity
* Saner defaults for remote write
* Reduce allocs on retries

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

Update autorest vedoring (prometheus#4147)

Signed-off-by: bege13mot <bege13mot@gmail.com>

Update aws-sdk-go (prometheus#4153)

Signed-off-by: bege13mot <bege13mot@gmail.com>

add unused pointslices to the pool (prometheus#4363)

Signed-off-by: Tony Lee <tl@hudson-trading.com>

Add 3 commands in `promtool` for getting debug information from prometheus server (prometheus#4247)

`debug all` - all information
`debug metrics` - metrics  information
`debug pprof` - profiling  information

the final result is compressed in a `tar.gz` file

Signed-off-by: chyeh <chyeh.taiwan@gmail.com>

main: Improve / clean up error messages (prometheus#4286)

Signed-off-by: Julius Volz <julius.volz@gmail.com>

Document internal Prometheus server architecture (prometheus#4295)

* Document internal Prometheus server architecture

Signed-off-by: Julius Volz <julius.volz@gmail.com>

* Review fixups

Signed-off-by: Julius Volz <julius.volz@gmail.com>

promtool: add command for querying series (prometheus#4308)

Signed-off-by: Shubheksha Jalan <jshubheksha@gmail.com>

Add missing import to promtool, fix build (prometheus#4395)

Sorry, I used GitHub's web-based merge-conflict-resolution editor on
prometheus#4308 and it didn't show me
test errors afterwards, but maybe they didn't run again or I should have
waited or something.

Signed-off-by: Julius Volz <julius.volz@gmail.com>

EC2 Discovery: Allow to set a custom endpoint (prometheus#4333)

Allowing to set a custom endpoint makes it easy to monitor targets on non AWS providers with EC2 compliant APIs.

Signed-off-by: Jannick Fahlbusch <git@jf-projects.de>

Reuse (copy) overlapping matrix samples between range evaluation steps (prometheus#4315)

* Reuse (copy) overlapping matrix samples between range evaluation steps.

Signed-off-by: Alin Sinpalean <alin.sinpalean@gmail.com>

Expose Group.CopyState() (prometheus#4304)

This makes the `rules` package more useful to projects that use
Prometheus as a library.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

add query labels command to promtool (prometheus#4346)

Signed-off-by: Daisy T <daisyts@gmx.com>

web: add named anchors for each rule group (prometheus#4130)

* web: add named anchors for each rule group

Signed-off-by: Adam Shannon <adamkshannon@gmail.com>

Update internal architecture diagram (prometheus#4398)

Signed-off-by: Julius Volz <julius.volz@gmail.com>

Only add LookbackDelta to vector selectors (prometheus#4399)

Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>

Related to prometheus#4226

add prefix "common-" to make target names

This allows rules to be overridden with warnings about conflicting
target names.

Signed-off-by: Paul Gier <pgier@redhat.com>

expose log.level for promlog for remote_storage_adapter (prometheus#4195)

* expose log.level for promlog for remote_storage_adapter

Signed-off-by: sipian <cs15btech11019@iith.ac.in>

* replace flag description

Signed-off-by: Harsh Agarwal <cs15btech11019@iith.ac.in>

go-bindata debug clarification (prometheus#4411)

Signed-off-by: Stafford Williams <stafford.williams@gmail.com>

discovery/ec2: Maintain order of subnet_id label

Signed-off-by: José Martínez <xosemp@gmail.com>

discovery/ec2: Add primary_subnet_id label

Signed-off-by: José Martínez <xosemp@gmail.com>

Don't import testing in code which is imported from non-test code. (prometheus#4400)

It polutes the flags.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Log errors encountered when marshalling and writing responses.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Review feedback.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Review feedback.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Review feedback.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Update method name in rules template, fix rendering (prometheus#4416)

Fixes prometheus#4407

Signed-off-by: Julius Volz <julius.volz@gmail.com>

Fix typo (prometheus#4423)

Signed-off-by: Henri DF <henridf@gmail.com>

Send "Accept-Encoding" header in read request (prometheus#4421)

We should be doing this since we only accept Snappy-encoded responses.

Signed-off-by: Henri DF <henridf@gmail.com>

Handle a remote read error and return other results, add remote error as extra field in api response.

Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>

Removed some code from other project

Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>
367ca6c

gouthamve added a commit to gouthamve/prometheus that referenced this issue Aug 1, 2018

@jacksontj @gouthamve
Check for timeout in each iteration of matrixSelector (prometheus#4300) 
Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>

Fixes prometheus#4288
15f6491

bobmshannon pushed a commit to bobmshannon/prometheus that referenced this issue Nov 19, 2018

Bob Shannon GitHub Enterprise
Upgrade Prometheus to 2.3.2 (prometheus#20) 
* Bubble up errors to promql from populating iterators (prometheus#4136)

This changes the Walk/Inspect API inside the promql package to bubble
up errors. This is done by having the inspector return an error (instead
of a bool) and then bubbling that up in the Walk. This way if any error
is encountered in the Walk() the walk will stop and return the error.
This avoids issues where errors from the Querier where being ignored
(causing incorrect promql evaluation).

Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>

Fixes prometheus#4136

* Timeout if populating iterators takes too long (prometheus#4291)

Right now promql won't time out a request if populating the iterators
takes a long time.

Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>

Fixes prometheus#4289

* Check for timeout in each iteration of matrixSelector (prometheus#4300)

Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>

Fixes prometheus#4288

* fix the zookeper race (prometheus#4355)

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>

* Merge pull request prometheus#4329 from nailgun/4327-ingress-discovery-issue

discovery/kubernetes/ingress: fix TLS discovery

* return error exit status in prometheus cli (prometheus#4296)

Signed-off-by: mikeykhalil <mikeyfkhalil@gmail.com>

* Reorder startup and shutdown to prevent panics. (prometheus#4321)

Start rule manager only after tsdb and config is loaded.
Stop rule manager before tsdb to avoid writing to closed storage.
Wait for any in-progress reloads to complete before shutting
down rule manager, so that rule manager doesn't get updated after
being shut down.

Remove incorrect comment around shutting down query enginge.
Log when config reload is completed.

Fixes prometheus#4133
Fixes prometheus#4262

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Update vendoring for tsdb (prometheus#4369)

This pulls in tsdb PRs 330 344 348 353 354 356

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Release 2.3.2

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
696ee49
@lock

This comment has been minimized.

Sign in to view
Copy link

lock bot commented Mar 22, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 22, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.