Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.
Sign upRefactor index ranges handling #1274
Conversation
joschi
added some commits
Jun 25, 2015
joschi
added this to the 1.2.0 milestone
Jun 30, 2015
joschi
added some commits
Jun 30, 2015
joschi
added
elasticsearch
infrastructure
ready-for-review
labels
Jun 30, 2015
joschi
added some commits
Jul 1, 2015
kroepke
reviewed
Jul 1, 2015
| @Override | ||
| public SortedSet<IndexRange> findAll() { | ||
| final SearchRequest request = client.prepareSearch() |
This comment has been minimized.
This comment has been minimized.
kroepke
Jul 1, 2015
Member
I think this search also needs to take into account the graylog index prefix.
Otherwise we could pull in indices that could be part of another graylog deployment writing to the same cluster. (Unlikely but that would trigger very fun bugs
This comment has been minimized.
This comment has been minimized.
joschi
Jul 1, 2015
Contributor
Good catch. This should actually be added to all methods making queries to Elasticsearch.
kroepke
reviewed
Jul 1, 2015
| @Singleton | ||
| public class IndexMapping { | ||
| public static final String TYPE_MESSAGE = "message"; | ||
| public static final String TYPE_META = "graylog_meta"; |
This comment has been minimized.
This comment has been minimized.
kroepke
Jul 1, 2015
Member
should this maybe be called something more specific?
if we ever decide to store other metadata about the index the name would maybe be a little misleading?
also, since we control the types in the index, i think we can drop the graylog_ prefix.
perhaps indexrange ?
This comment has been minimized.
This comment has been minimized.
kroepke
reviewed
Jul 1, 2015
| } | ||
| @Override | ||
| public void destroyAll() { |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
kroepke
reviewed
Jul 1, 2015
| } | ||
| @Override | ||
| public void destroy(String index) { |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
kroepke
reviewed
Jul 1, 2015
| public abstract String indexName(); | ||
| @JsonProperty | ||
| public abstract DateTime begin(); |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
joschi
Jul 1, 2015
Contributor
"start" would be the same as "end" (it was misnamed from the start – pun intended) and I think that would be much more irritating than just changing the format.
This comment has been minimized.
This comment has been minimized.
kroepke
reviewed
Jul 1, 2015
| @@ -108,7 +93,9 @@ public void execute() { | ||
| } | ||
| try { | ||
| ranges.add(calculateRange(index)); | ||
| final IndexRange indexRange = indexRangeService.calculateRange(index); | ||
| indexRangeService.save(indexRange); |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
joschi
added some commits
Jul 2, 2015
This comment has been minimized.
This comment has been minimized.
|
lgtm |
added a commit
that referenced
this pull request
Jul 7, 2015
kroepke
merged commit 3bcee89
into
master
Jul 7, 2015
kroepke
deleted the
index-ranges-refactoring
branch
Jul 7, 2015
joschi
referenced this pull request
Jul 8, 2015
Closed
Buffer overflow with integer arithmetics in index calculation excludes index from search due not to be closed #1293
joschi
referenced this pull request
Jul 20, 2015
Closed
Save indices metadata into separate index #1311
This was referenced Aug 5, 2015
This comment has been minimized.
This comment has been minimized.
drewmmiranda
commented
Sep 9, 2015
|
Is 1.2 rc2 meant to fully address this? I'm still getting blank searches for specific time ranges until IIrecalculate. |
This comment has been minimized.
This comment has been minimized.
|
@drewmmiranda You need to recalculate index ranges once (although the old ones should've been migrated on the first start of Graylog 1.2.0, see #1378). Do you get the blank searches while running Graylog 1.2.0 or just after the upgrade from an earlier version? |
This comment has been minimized.
This comment has been minimized.
drewmmiranda
commented
Sep 9, 2015
|
Running v1.2.0-rc.2. This is correct. Same behavior as before. Relative and keyword ranges work correctly. Absolute ranges do not. |
This comment has been minimized.
This comment has been minimized.
|
@drewmmiranda Please open a new ticket and elaborate on what "Relative and keyword ranges work correctly. Absolute ranges do not." means, e. g. describe test cases or what results you expect and what the actual results are. |
drewmmiranda
referenced this pull request
Sep 9, 2015
Closed
Index ranges handling still not working correctly, no data returned for absolute search times #1409
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
joschi commentedJun 30, 2015
This PR refactors the index ranges calculation and handling in Graylog. It moves the actual persistence from MongoDB ("index_ranges" collection) into Elasticsearch ("graylog_meta" type) so that this information is stored along the Elasticsearch indices they refer to.
Additionally the index ranges now not only contain the (misnamed) "start" attribute but actually the whole index range with begin (oldest message) and end (most recent message) which can be used to optimize query requests by more precisely restricting those to the actually relevant indices.