New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move index ranges back into MongoDB #1493

Merged
merged 6 commits into from Oct 23, 2015

Conversation

Projects
None yet
3 participants
@joschi
Contributor

joschi commented Oct 19, 2015

This PR moves index ranges back from Elasticsearch into MongoDB (while keeping the new format with additional information).

The IndexRangesMigrationPeriodical has been extended to now migrate old index ranges from MongoDB (Graylog 1.1.x and earlier) and the current index ranges from Elasticsearch into MongoDB and will only run the migration once.

EsIndexRangeService is now deprecated and shouldn't be used anymore (except for the migration of
old index range information from Elasticsearch in ´IndexRangesMigrationPeriodical`).

Fixes #1465

@joschi joschi added this to the 1.2.2 milestone Oct 19, 2015

Implement MongoIndexRangeService and make it new default IndexRangeSe…
…rvice

Index range information is now stored in MongoDB again, but in the format being used in Elasticsearch
and not in the legacy format (which only included partial information).

The IndexRangesMigrationPeriodical has been extended to now migrate old index ranges from MongoDB
(Graylog 1.1.x and earlier) and the current index ranges from Elasticsearch into MongoDB and will
only run the migration once.

EsIndexRangeService is now deprecated and shouldn't be used anymore (except for the migration of
old index range information from Elasticsearch in IndexRangesMigrationPeriodical).

Fixes #1465

@joschi joschi force-pushed the issue-1465 branch from 3170386 to 22486d4 Oct 19, 2015

@bernd

This comment has been minimized.

Member

bernd commented Oct 20, 2015

I went from a 1.1.6 setup to a setup with this PR applied and got the following error during startup.

2015-10-20 17:47:34,041 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running.
2015-10-20 17:47:34,045 INFO : org.graylog2.shared.inputs.InputStateListener - Input [Random HTTP message generator/5626607fc83058425d47e050] is now STARTING
2015-10-20 17:47:34,051 INFO : org.graylog2.shared.inputs.InputStateListener - Input [Random HTTP message generator/5626607fc83058425d47e050] is now RUNNING
2015-10-20 17:47:34,651 INFO : org.graylog2.periodical.IndexRangesMigrationPeriodical - Migrating index range from MongoDB: MongoIndexRange{id=56266094c83058425d47e06f, indexName=graylog2_0, begin=1970-01-01T00:00:00.000Z, end=2015-10-20T15:41:07.000Z, calculatedAt=2015-10-20T15:41:08.000Z, calculationDuration=35}
2015-10-20 17:47:34,654 INFO : org.graylog2.periodical.IndexRangesMigrationPeriodical - Migrating index range from MongoDB: MongoIndexRange{id=562660a1c83058425d47e084, indexName=graylog2_1, begin=1970-01-01T00:00:00.000Z, end=2015-10-20T15:41:19.000Z, calculatedAt=2015-10-20T15:41:21.000Z, calculationDuration=19}
2015-10-20 17:47:34,655 INFO : org.graylog2.periodical.IndexRangesMigrationPeriodical - Migrating index range from MongoDB: MongoIndexRange{id=562660abc83058425d47e095, indexName=graylog2_2, begin=1970-01-01T00:00:00.000Z, end=2015-10-20T15:41:30.000Z, calculatedAt=2015-10-20T15:41:31.000Z, calculationDuration=15}
2015-10-20 17:47:34,656 INFO : org.graylog2.periodical.IndexRangesMigrationPeriodical - Migrating index range from MongoDB: MongoIndexRange{id=562660b3c83058425d47e0a9, indexName=graylog2_3, begin=1970-01-01T00:00:00.000Z, end=2015-10-20T15:41:37.000Z, calculatedAt=2015-10-20T15:41:39.000Z, calculationDuration=21}
2015-10-20 17:47:34,657 INFO : org.graylog2.periodical.IndexRangesMigrationPeriodical - Migrating index range from MongoDB: MongoIndexRange{id=562660b3c83058425d47e0a7, indexName=graylog2_4, begin=1970-01-01T00:00:00.000Z, end=2015-10-20T15:41:39.000Z, calculatedAt=1970-01-01T00:00:00.000Z, calculationDuration=0}
2015-10-20 17:47:34,660 WARN : com.google.common.cache.LocalCache - Exception thrown during refresh
java.util.concurrent.ExecutionException: org.graylog2.database.NotFoundException: Couldn't find index range for index graylog2_0
    at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
    at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
    at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
    at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137)
    at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2348)
    at com.google.common.cache.LocalCache$Segment$1.run(LocalCache.java:2331)
    at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
    at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
    at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:101)
    at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:170)
    at com.google.common.cache.LocalCache$Segment.loadAsync(LocalCache.java:2326)
    at com.google.common.cache.LocalCache$Segment.refresh(LocalCache.java:2389)
    at com.google.common.cache.LocalCache.refresh(LocalCache.java:4090)
    at com.google.common.cache.LocalCache$LocalLoadingCache.refresh(LocalCache.java:4843)
    at org.graylog2.indexer.ranges.EsIndexRangeService.handleIndexRangeUpdate(EsIndexRangeService.java:218)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.google.common.eventbus.EventSubscriber.handleEvent(EventSubscriber.java:74)
    at com.google.common.eventbus.EventBus.dispatch(EventBus.java:322)
    at com.google.common.eventbus.AsyncEventBus.access$001(AsyncEventBus.java:34)
    at com.google.common.eventbus.AsyncEventBus$1.run(AsyncEventBus.java:117)
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.graylog2.database.NotFoundException: Couldn't find index range for index graylog2_0
    at org.graylog2.indexer.ranges.EsIndexRangeService.loadIndexRange(EsIndexRangeService.java:132)
    at org.graylog2.indexer.ranges.EsIndexRangeService.access$000(EsIndexRangeService.java:61)
    at org.graylog2.indexer.ranges.EsIndexRangeService$1.load(EsIndexRangeService.java:83)
    at org.graylog2.indexer.ranges.EsIndexRangeService$1.load(EsIndexRangeService.java:80)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.google.common.cache.LocalCache$Segment.loadAsync(LocalCache.java:2325)
    ... 16 more
2015-10-20 17:47:34,671 WARN : com.google.common.cache.LocalCache - Exception thrown during refresh
java.util.concurrent.ExecutionException: org.graylog2.database.NotFoundException: Couldn't find index range for index graylog2_1
    at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
    at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
    at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
    at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137)
    at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2348)
    at com.google.common.cache.LocalCache$Segment$1.run(LocalCache.java:2331)
    at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
    at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
    at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:101)
    at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:170)
    at com.google.common.cache.LocalCache$Segment.loadAsync(LocalCache.java:2326)
    at com.google.common.cache.LocalCache$Segment.refresh(LocalCache.java:2389)
    at com.google.common.cache.LocalCache.refresh(LocalCache.java:4090)
    at com.google.common.cache.LocalCache$LocalLoadingCache.refresh(LocalCache.java:4843)
    at org.graylog2.indexer.ranges.EsIndexRangeService.handleIndexRangeUpdate(EsIndexRangeService.java:218)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.google.common.eventbus.EventSubscriber.handleEvent(EventSubscriber.java:74)
    at com.google.common.eventbus.EventBus.dispatch(EventBus.java:322)
    at com.google.common.eventbus.AsyncEventBus.access$001(AsyncEventBus.java:34)
    at com.google.common.eventbus.AsyncEventBus$1.run(AsyncEventBus.java:117)
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.graylog2.database.NotFoundException: Couldn't find index range for index graylog2_1
    at org.graylog2.indexer.ranges.EsIndexRangeService.loadIndexRange(EsIndexRangeService.java:132)
    at org.graylog2.indexer.ranges.EsIndexRangeService.access$000(EsIndexRangeService.java:61)
    at org.graylog2.indexer.ranges.EsIndexRangeService$1.load(EsIndexRangeService.java:83)
    at org.graylog2.indexer.ranges.EsIndexRangeService$1.load(EsIndexRangeService.java:80)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.google.common.cache.LocalCache$Segment.loadAsync(LocalCache.java:2325)
    ... 16 more
@joschi

This comment has been minimized.

Contributor

joschi commented Oct 21, 2015

    at org.graylog2.indexer.ranges.EsIndexRangeService.handleIndexRangeUpdate(EsIndexRangeService.java:218)

D'oh!

@kroepke kroepke added the 0 - Backlog label Oct 21, 2015

@bernd

This comment has been minimized.

Member

bernd commented Oct 21, 2015

The previous issue is gone, thanks!

I still see the following during the migration. Harmless?

2015-10-21 10:59:01,094 INFO : org.graylog2.periodical.IndexRangesMigrationPeriodical - Migrating index range from MongoDB: MongoIndexRange{id=56275366c8301c720ebcff8e, indexName=graylog2_0, begin=1970-01-01T00:00:00.000Z, end=2015-10-21T08:57:09.000Z, calculatedAt=2015-10-21T08:57:10.000Z, calculationDuration=56}
2015-10-21 10:59:01,098 INFO : org.graylog2.periodical.IndexRangesMigrationPeriodical - Migrating index range from MongoDB: MongoIndexRange{id=56275375c8301c720ebcffa6, indexName=graylog2_1, begin=1970-01-01T00:00:00.000Z, end=2015-10-21T08:57:23.000Z, calculatedAt=2015-10-21T08:57:25.000Z, calculationDuration=19}
2015-10-21 10:59:01,101 INFO : org.graylog2.periodical.IndexRangesMigrationPeriodical - Migrating index range from MongoDB: MongoIndexRange{id=56275384c8301c720ebcffbc, indexName=graylog2_2, begin=1970-01-01T00:00:00.000Z, end=2015-10-21T08:57:39.000Z, calculatedAt=2015-10-21T08:57:40.000Z, calculationDuration=17}
2015-10-21 10:59:01,102 INFO : org.graylog2.periodical.IndexRangesMigrationPeriodical - Migrating index range from MongoDB: MongoIndexRange{id=5627538cc8301c720ebcffd1, indexName=graylog2_3, begin=1970-01-01T00:00:00.000Z, end=2015-10-21T08:57:48.000Z, calculatedAt=2015-10-21T08:57:48.000Z, calculationDuration=7}
2015-10-21 10:59:01,102 INFO : org.graylog2.periodical.IndexRangesMigrationPeriodical - Migrating index range from MongoDB: MongoIndexRange{id=5627538cc8301c720ebcffcf, indexName=graylog2_4, begin=1970-01-01T00:00:00.000Z, end=2015-10-21T08:57:48.000Z, calculatedAt=1970-01-01T00:00:00.000Z, calculationDuration=0}
2015-10-21 10:59:01,117 WARN : org.graylog2.indexer.ranges.EsIndexRangeService - Couldn't load index range for index graylog2_1
2015-10-21 10:59:01,119 WARN : org.graylog2.indexer.ranges.EsIndexRangeService - Couldn't load index range for index graylog2_0
2015-10-21 10:59:01,126 WARN : org.graylog2.indexer.ranges.EsIndexRangeService - Couldn't load index range for index graylog2_3
2015-10-21 10:59:01,127 WARN : org.graylog2.indexer.ranges.EsIndexRangeService - Couldn't load index range for index graylog2_2
2015-10-21 10:59:01,129 WARN : org.graylog2.indexer.ranges.EsIndexRangeService - Couldn't load index range for index graylog2_4
@bernd

This comment has been minimized.

Member

bernd commented Oct 21, 2015

I am getting this WARN with a clean MongoDB and Elasticsearch.

2015-10-21 19:06:12,940 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running.
2015-10-21 19:06:14,934 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Enabling session validation scheduler...
2015-10-21 19:06:18,136 INFO : org.graylog2.indexer.Deflector - Did not find an deflector alias. Setting one up now.
2015-10-21 19:06:18,138 INFO : org.graylog2.indexer.Deflector - There is no index target to point to. Creating one now.
2015-10-21 19:06:18,139 INFO : org.graylog2.indexer.Deflector - Cycling deflector to next index now.
2015-10-21 19:06:18,140 INFO : org.graylog2.indexer.Deflector - Cycling from <none> to <graylog2_0>
2015-10-21 19:06:18,140 INFO : org.graylog2.indexer.Deflector - Creating index target <graylog2_0>...
2015-10-21 19:06:18,640 INFO : org.graylog2.indexer.Deflector - Waiting for index allocation of <graylog2_0>
2015-10-21 19:06:18,763 INFO : org.graylog2.indexer.Deflector - Done!
2015-10-21 19:06:18,763 INFO : org.graylog2.indexer.Deflector - Pointing deflector to new target index....
2015-10-21 19:06:18,811 INFO : org.graylog2.system.jobs.SystemJobManager - Submitted SystemJob <0bd9e0b0-7816-11e5-b0d1-7c7a91a75dc3> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob]
2015-10-21 19:06:18,811 INFO : org.graylog2.indexer.Deflector - Done!
2015-10-21 19:06:18,811 INFO : org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob - Calculating ranges for index graylog2_0.
2015-10-21 19:06:18,937 INFO : org.graylog2.indexer.ranges.MongoIndexRangeService - Calculated range of [graylog2_0] in [125ms].
2015-10-21 19:06:18,942 INFO : org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob - Created ranges for index graylog2_0.
2015-10-21 19:06:18,943 INFO : org.graylog2.system.jobs.SystemJobManager - SystemJob <0bd9e0b0-7816-11e5-b0d1-7c7a91a75dc3> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob] finished in 131ms.
2015-10-21 19:06:23,262 WARN : org.graylog2.indexer.ranges.EsIndexRangeService - Couldn't load index range for index graylog2_0
@bernd

This comment has been minimized.

Member

bernd commented Oct 23, 2015

LGTM 👍

bernd added a commit that referenced this pull request Oct 23, 2015

Merge pull request #1493 from Graylog2/issue-1465
Move index ranges back into MongoDB

@bernd bernd merged commit d434405 into 1.2 Oct 23, 2015

3 checks passed

ci Jenkins build graylog2-server-integration-pr 302 has succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@bernd bernd deleted the issue-1465 branch Oct 23, 2015

bernd added a commit that referenced this pull request Oct 30, 2015

Merge pull request #1493 from Graylog2/issue-1465
Move index ranges back into MongoDB
(cherry picked from commit d434405)

bernd added a commit that referenced this pull request Oct 30, 2015

Merge pull request #1493 from Graylog2/issue-1465
Move index ranges back into MongoDB
(cherry picked from commit d434405)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment