Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Views don't search on all indexes of a stream #6828

Closed
dploeger opened this issue Nov 19, 2019 · 4 comments · Fixed by #6957
Closed

Views don't search on all indexes of a stream #6828

dploeger opened this issue Nov 19, 2019 · 4 comments · Fixed by #6957
Labels
Milestone

Comments

@dploeger
Copy link

@dploeger dploeger commented Nov 19, 2019

When using views to display a graph and selecting a specific stream, the graph seems to stop at the point in time, where the stream's index was rotated.

Expected Behavior

A graph in the view should display all data across all indices of a selected stream.

Current Behavior

The graph only displays data from where the rotation was done.

image

When no stream is selected, the complete graph is shown

Steps to Reproduce (for bugs)

  1. Create an index that has a rotation (for example: daily rotation)
  2. Collect data past that rotation point
  3. Create a view showing a graph from that data

Your Environment

  • Graylog Version: 3.1.2
  • Elasticsearch Version: 5.6.16
  • MongoDB Version: 4.0.12
  • Operating System: macOS 10.15.1 (19B88)
  • Browser version: 78.0.3904.97
@dploeger dploeger added the bug label Nov 19, 2019
@kroepke

This comment has been minimized.

Copy link
Member

@kroepke kroepke commented Nov 25, 2019

This sounds like the index ranges aren't up to date for that index set.
Can you double check whether this problem continues to be there after choosing "Recalcuate index ranges" from the Maintenance menu on the affected index range page?

@dploeger

This comment has been minimized.

Copy link
Author

@dploeger dploeger commented Nov 25, 2019

It does, sadly. Recalculating didn't do anything.

@l0calhost

This comment has been minimized.

Copy link

@l0calhost l0calhost commented Dec 6, 2019

I can confirm this for

  • Graylog 3.1.3
  • ES 5.6.16
dennisoelkers added a commit that referenced this issue Dec 9, 2019
When rotating an index, the following details are calculated and stored
in an index range:

  * the smallest timestamp (oldest document)
  * the largest timestamp (youngest document)
  * the set of stream ids contained in documents in this index

For the latter, a terms query is used. Unfortunately, no specific size
was set for it, so it used the default size of 10. This meant that all
index ranges calculated before did not contain more than 10 stream ids.

In the current search, this did not lead to imminent problems. The logic
used to determine the indices which are used for the search was lenient
enough to only consider either the set of stream ids containing the
current stream id or the index being managed by the index set of the
stream.

The new search is stricter and expects the set of stream ids to contain
the current stream being searched for. It only checks if the index set
of the stream manages the index if it is the deflector.

This change is now calculating the stream id set for a give index
correctly, by specifying a `size` parameter for the terms aggregation
that is hopefully high enough (`Integer.MAX_VALUE`) to handle all cases.

Fixes #6828.
@bernd bernd added this to the 3.1.4 milestone Dec 10, 2019
@bernd bernd closed this in #6957 Dec 10, 2019
bernd added a commit that referenced this issue Dec 10, 2019
…x. (#6957)

When rotating an index, the following details are calculated and stored
in an index range:

  * the smallest timestamp (oldest document)
  * the largest timestamp (youngest document)
  * the set of stream ids contained in documents in this index

For the latter, a terms query is used. Unfortunately, no specific size
was set for it, so it used the default size of 10. This meant that all
index ranges calculated before did not contain more than 10 stream ids.

In the current search, this did not lead to imminent problems. The logic
used to determine the indices which are used for the search was lenient
enough to only consider either the set of stream ids containing the
current stream id or the index being managed by the index set of the
stream.

The new search is stricter and expects the set of stream ids to contain
the current stream being searched for. It only checks if the index set
of the stream manages the index if it is the deflector.

This change is now calculating the stream id set for a give index
correctly, by specifying a `size` parameter for the terms aggregation
that is hopefully high enough (`Integer.MAX_VALUE`) to handle all cases.

Fixes #6828.
bernd added a commit that referenced this issue Dec 10, 2019
…x. (#6957)

When rotating an index, the following details are calculated and stored
in an index range:

  * the smallest timestamp (oldest document)
  * the largest timestamp (youngest document)
  * the set of stream ids contained in documents in this index

For the latter, a terms query is used. Unfortunately, no specific size
was set for it, so it used the default size of 10. This meant that all
index ranges calculated before did not contain more than 10 stream ids.

In the current search, this did not lead to imminent problems. The logic
used to determine the indices which are used for the search was lenient
enough to only consider either the set of stream ids containing the
current stream id or the index being managed by the index set of the
stream.

The new search is stricter and expects the set of stream ids to contain
the current stream being searched for. It only checks if the index set
of the stream manages the index if it is the deflector.

This change is now calculating the stream id set for a give index
correctly, by specifying a `size` parameter for the terms aggregation
that is hopefully high enough (`Integer.MAX_VALUE`) to handle all cases.

Fixes #6828.

(cherry picked from commit 8e7560b)
@bernd

This comment has been minimized.

Copy link
Member

@bernd bernd commented Dec 10, 2019

@dploeger @l0calhost Thank you for the report and information. A fix for this will be included in the upcoming 3.1.4 bug fix release.

Please make sure to read the release notes because it requires a manual step to recalculate the index ranges. (there was a bug in calculating them)

bernd added a commit that referenced this issue Dec 10, 2019
…x. (#6957) (#6967)

When rotating an index, the following details are calculated and stored
in an index range:

  * the smallest timestamp (oldest document)
  * the largest timestamp (youngest document)
  * the set of stream ids contained in documents in this index

For the latter, a terms query is used. Unfortunately, no specific size
was set for it, so it used the default size of 10. This meant that all
index ranges calculated before did not contain more than 10 stream ids.

In the current search, this did not lead to imminent problems. The logic
used to determine the indices which are used for the search was lenient
enough to only consider either the set of stream ids containing the
current stream id or the index being managed by the index set of the
stream.

The new search is stricter and expects the set of stream ids to contain
the current stream being searched for. It only checks if the index set
of the stream manages the index if it is the deflector.

This change is now calculating the stream id set for a give index
correctly, by specifying a `size` parameter for the terms aggregation
that is hopefully high enough (`Integer.MAX_VALUE`) to handle all cases.

Fixes #6828.

(cherry picked from commit 8e7560b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.