New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor index ranges handling #1274

Merged
merged 31 commits into from Jul 7, 2015

Conversation

Projects
None yet
3 participants
@joschi
Contributor

joschi commented Jun 30, 2015

This PR refactors the index ranges calculation and handling in Graylog. It moves the actual persistence from MongoDB ("index_ranges" collection) into Elasticsearch ("graylog_meta" type) so that this information is stored along the Elasticsearch indices they refer to.

Additionally the index ranges now not only contain the (misnamed) "start" attribute but actually the whole index range with begin (oldest message) and end (most recent message) which can be used to optimize query requests by more precisely restricting those to the actually relevant indices.

@joschi joschi added this to the 1.2.0 milestone Jun 30, 2015

@Override
public SortedSet<IndexRange> findAll() {
final SearchRequest request = client.prepareSearch()

This comment has been minimized.

@kroepke

kroepke Jul 1, 2015

Member

I think this search also needs to take into account the graylog index prefix.
Otherwise we could pull in indices that could be part of another graylog deployment writing to the same cluster. (Unlikely but that would trigger very fun bugs 🎅 )

This comment has been minimized.

@joschi

joschi Jul 1, 2015

Contributor

Good catch. This should actually be added to all methods making queries to Elasticsearch.

@Singleton
public class IndexMapping {
public static final String TYPE_MESSAGE = "message";
public static final String TYPE_META = "graylog_meta";

This comment has been minimized.

@kroepke

kroepke Jul 1, 2015

Member

should this maybe be called something more specific?
if we ever decide to store other metadata about the index the name would maybe be a little misleading?
also, since we control the types in the index, i think we can drop the graylog_ prefix.

perhaps indexrange ?

This comment has been minimized.

@joschi

joschi Jul 1, 2015

Contributor

Good point. I'll rename it.

}
@Override
public void destroyAll() {

This comment has been minimized.

@kroepke

kroepke Jul 1, 2015

Member

Is this method used outside of tests?

This comment has been minimized.

@joschi

joschi Jul 1, 2015

Contributor

No, it's a remainder of the incremental refactoring.

}
@Override
public void destroy(String index) {

This comment has been minimized.

@kroepke

kroepke Jul 1, 2015

Member

Is this method used outside of tests?

This comment has been minimized.

@joschi

joschi Jul 1, 2015

Contributor

Nope, also a remainder of the incremental refactoring.

public abstract String indexName();
@JsonProperty
public abstract DateTime begin();

This comment has been minimized.

@kroepke

kroepke Jul 1, 2015

Member

This changes public API, can we either keep it as start() or alias it?

This comment has been minimized.

@joschi

joschi Jul 1, 2015

Contributor

"start" would be the same as "end" (it was misnamed from the start – pun intended) and I think that would be much more irritating than just changing the format.

This comment has been minimized.

@kroepke

kroepke Jul 7, 2015

Member

Ok, then let's somehow make this clear in the release notes?

@@ -108,7 +93,9 @@ public void execute() {
}
try {
ranges.add(calculateRange(index));
final IndexRange indexRange = indexRangeService.calculateRange(index);
indexRangeService.save(indexRange);

This comment has been minimized.

@kroepke

kroepke Jul 1, 2015

Member

so much 👍 for not destroying all and then adding them back!

This comment has been minimized.

@joschi

joschi Jul 1, 2015

Contributor

High Five!

@kroepke

This comment has been minimized.

Member

kroepke commented Jul 7, 2015

lgtm

kroepke added a commit that referenced this pull request Jul 7, 2015

@kroepke kroepke merged commit 3bcee89 into master Jul 7, 2015

2 checks passed

ci Jenkins build graylog2-server-integration-pr 10 has succeeded
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@kroepke kroepke deleted the index-ranges-refactoring branch Jul 7, 2015

@drewmmiranda

This comment has been minimized.

drewmmiranda commented Sep 9, 2015

Is 1.2 rc2 meant to fully address this? I'm still getting blank searches for specific time ranges until IIrecalculate.

@joschi

This comment has been minimized.

Contributor

joschi commented Sep 9, 2015

@drewmmiranda You need to recalculate index ranges once (although the old ones should've been migrated on the first start of Graylog 1.2.0, see #1378).

Do you get the blank searches while running Graylog 1.2.0 or just after the upgrade from an earlier version?

@drewmmiranda

This comment has been minimized.

drewmmiranda commented Sep 9, 2015

Running v1.2.0-rc.2. This is correct. Same behavior as before. Relative and keyword ranges work correctly. Absolute ranges do not.

2015-09-08 13:46:56,218 INFO : org.graylog2.periodical.IndexRotationThread - Deflector index <gl2cluster_84> should be rotated, Pointing deflector to new index now!
2015-09-08 13:46:56,218 INFO : org.graylog2.indexer.Deflector - Cycling deflector to next index now.
2015-09-08 13:46:56,236 INFO : org.graylog2.indexer.Deflector - Cycling from <gl2cluster_84> to <gl2cluster_85>
2015-09-08 13:46:56,236 INFO : org.graylog2.indexer.Deflector - Creating index target <gl2cluster_85>...
2015-09-08 13:46:56,629 INFO : org.graylog2.indexer.Deflector - Waiting for index allocation of <gl2cluster_85>
2015-09-08 13:46:56,630 INFO : org.graylog2.indexer.Deflector - Done!
2015-09-08 13:46:56,630 INFO : org.graylog2.indexer.Deflector - Pointing deflector to new target index....
2015-09-08 13:46:56,651 INFO : org.graylog2.system.jobs.SystemJobManager - Submitted SystemJob <faebbea0-5659-11e5-92a1-0050568e386c> [org.graylog2.indexer.SetIndexReadOnlyJob]
2015-09-08 13:46:56,651 INFO : org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob - Calculating ranges for index gl2cluster_84.
2015-09-08 13:46:56,651 INFO : org.graylog2.system.jobs.SystemJobManager - Submitted SystemJob <faebe5b0-5659-11e5-92a1-0050568e386c> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob]
2015-09-08 13:46:56,651 INFO : org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob - Calculating ranges for index gl2cluster_85.
2015-09-08 13:46:56,652 INFO : org.graylog2.system.jobs.SystemJobManager - Submitted SystemJob <faebe5b1-5659-11e5-92a1-0050568e386c> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob]
2015-09-08 13:46:56,652 INFO : org.graylog2.indexer.Deflector - Done!
2015-09-08 13:46:56,659 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_85] in [8ms].
2015-09-08 13:46:56,670 INFO : org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob - Created ranges for index gl2cluster_85.
2015-09-08 13:46:56,670 INFO : org.graylog2.system.jobs.SystemJobManager - SystemJob <faebe5b1-5659-11e5-92a1-0050568e386c> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob] finished in 18ms.
2015-09-08 13:46:57,918 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_84] in [1267ms].
2015-09-08 13:46:57,921 INFO : org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob - Created ranges for index gl2cluster_84.
2015-09-08 13:46:57,921 INFO : org.graylog2.system.jobs.SystemJobManager - SystemJob <faebe5b0-5659-11e5-92a1-0050568e386c> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob] finished in 1269ms.
2015-09-08 13:47:26,651 INFO : org.graylog2.indexer.SetIndexReadOnlyJob - Flushing old index <gl2cluster_84>.
2015-09-08 13:47:26,706 INFO : org.graylog2.indexer.SetIndexReadOnlyJob - Setting old index <gl2cluster_84> to read-only.
2015-09-08 13:47:26,742 INFO : org.graylog2.system.jobs.SystemJobManager - Submitted SystemJob <0cdb4451-565a-11e5-92a1-0050568e386c> [org.graylog2.indexer.indices.jobs.OptimizeIndexJob]
2015-09-08 13:47:26,742 INFO : org.graylog2.system.jobs.SystemJobManager - SystemJob <faebbea0-5659-11e5-92a1-0050568e386c> [org.graylog2.indexer.SetIndexReadOnlyJob] finished in 90ms.
2015-09-08 13:47:26,742 INFO : org.graylog2.indexer.indices.jobs.OptimizeIndexJob - Optimizing index <gl2cluster_84>.
2015-09-08 13:50:56,187 INFO : org.graylog2.periodical.IndexRetentionThread - Number of indices (54) higher than limit (53). Running retention for 1 indices.
2015-09-08 13:50:56,232 INFO : org.graylog2.periodical.IndexRetentionThread - Running retention strategy [org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy] for index <gl2cluster_32>
2015-09-08 13:50:56,232 INFO : org.graylog2.plugin.indexer.retention.RetentionStrategy - Strategy is deleting.
2015-09-08 13:50:56,316 INFO : org.graylog2.plugin.indexer.retention.RetentionStrategy - Finished index retention strategy [org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy] for index <gl2cluster_32> in 83ms.
2015-09-08 13:50:56,317 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Re-calculating index ranges.
2015-09-08 13:50:56,317 INFO : org.graylog2.system.jobs.SystemJobManager - Submitted SystemJob <89c5dfc0-565a-11e5-92a1-0050568e386c> [org.graylog2.indexer.ranges.RebuildIndexRangesJob]
2015-09-08 13:50:57,324 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_33] in [980ms].
2015-09-08 13:50:58,153 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_34] in [698ms].
2015-09-08 13:50:59,551 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_37] in [765ms].
2015-09-08 13:51:00,299 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_38] in [637ms].
2015-09-08 13:51:01,340 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_35] in [917ms].
2015-09-08 13:51:02,273 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_36] in [834ms].
2015-09-08 13:51:03,650 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_78] in [927ms].
2015-09-08 13:51:04,797 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_77] in [1003ms].
2015-09-08 13:51:05,907 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_39] in [1006ms].
2015-09-08 13:51:07,593 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_76] in [1546ms].
2015-09-08 13:51:09,192 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_75] in [1412ms].
2015-09-08 13:51:10,525 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_79] in [1158ms].
2015-09-08 13:51:11,776 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_70] in [1109ms].
2015-09-08 13:51:12,899 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_73] in [1027ms].
2015-09-08 13:51:14,098 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_74] in [1073ms].
2015-09-08 13:51:15,517 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_71] in [1219ms].
2015-09-08 13:51:16,444 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_72] in [818ms].
2015-09-08 13:51:17,461 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_65] in [920ms].
2015-09-08 13:51:18,897 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_64] in [1051ms].
2015-09-08 13:51:20,107 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_67] in [1095ms].
2015-09-08 13:51:21,595 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_66] in [1221ms].
2015-09-08 13:51:22,512 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_69] in [791ms].
2015-09-08 13:51:24,565 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_68] in [1912ms].
2015-09-08 13:51:25,734 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_60] in [1085ms].
2015-09-08 13:51:27,063 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_61] in [1254ms].
2015-09-08 13:51:28,624 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_62] in [1310ms].
2015-09-08 13:51:30,150 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_63] in [1437ms].
2015-09-08 13:51:31,597 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_59] in [1340ms].
2015-09-08 13:51:32,814 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_58] in [1112ms].
2015-09-08 13:51:34,065 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_57] in [1161ms].
2015-09-08 13:51:35,119 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_56] in [975ms].
2015-09-08 13:51:36,119 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_55] in [923ms].
2015-09-08 13:51:37,364 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_54] in [1086ms].
2015-09-08 13:51:38,749 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_53] in [1207ms].
2015-09-08 13:51:39,740 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_51] in [915ms].
2015-09-08 13:51:40,909 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_52] in [1081ms].
2015-09-08 13:51:42,103 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_50] in [1094ms].
2015-09-08 13:51:42,289 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_85] in [33ms].
2015-09-08 13:51:44,540 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_47] in [2243ms].
2015-09-08 13:51:46,522 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_84] in [1851ms].
2015-09-08 13:51:48,522 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_46] in [1837ms].
2015-09-08 13:51:49,502 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_83] in [882ms].
2015-09-08 13:51:50,746 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_49] in [1148ms].
2015-09-08 13:51:51,742 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_82] in [852ms].
2015-09-08 13:51:53,276 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_48] in [1390ms].
2015-09-08 13:51:54,376 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_81] in [1019ms].
2015-09-08 13:51:55,548 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_43] in [966ms].
2015-09-08 13:51:56,900 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_80] in [1268ms].
2015-09-08 13:51:57,972 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_42] in [994ms].
2015-09-08 13:52:00,476 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_45] in [2138ms].
2015-09-08 13:52:01,664 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_44] in [1051ms].
2015-09-08 13:52:02,805 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_40] in [1001ms].
2015-09-08 13:52:03,845 INFO : org.graylog2.indexer.ranges.EsIndexRangeService - Calculated range of [gl2cluster_41] in [662ms].
2015-09-08 13:52:03,976 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Done calculating index ranges for 53 indices. Took 67632ms.
2015-09-08 13:52:03,977 INFO : org.graylog2.system.jobs.SystemJobManager - SystemJob <89c5dfc0-565a-11e5-92a1-0050568e386c> [org.graylog2.indexer.ranges.RebuildIndexRangesJob] finished in 67660ms.
2015-09-08 13:54:03,690 INFO : org.graylog2.system.jobs.SystemJobManager - SystemJob <0cdb4451-565a-11e5-92a1-0050568e386c> [org.graylog2.indexer.indices.jobs.OptimizeIndexJob] finished in 396948ms.
@joschi

This comment has been minimized.

Contributor

joschi commented Sep 9, 2015

@drewmmiranda Please open a new ticket and elaborate on what "Relative and keyword ranges work correctly. Absolute ranges do not." means, e. g. describe test cases or what results you expect and what the actual results are.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment