New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
IndexSet default setting #3209
IndexSet default setting #3209
Conversation
Can we simply use a |
@bernd Idea 3: Store the default index set ID in cluster config. This would make queries for that more expensive, but in how many places do we really have to check if an index set is the default index set? |
4339bb9
to
073bcf3
Compare
This is ready for review now. |
@@ -38,6 +38,13 @@ | |||
Optional<IndexSetConfig> get(String id); | |||
|
|||
/** | |||
* Retrieve the default index set. | |||
* | |||
* @return A filled {@link Optional} with the default index set, an empty {@link Optional} if there is no default. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a situation where there is no default index set, which is not a serious bug?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The default index set should be there. If it's not then
- Content pack imports that contain streams will fail
- The old indices REST API endpoints without an index set parameter will fail
- The
V20161116172200_CreateDefaultStreamMigration
will fail
If we change that to not returning an optional and throwing an error when the default index set is absent, we also need a migration to migrate the default flag from the index_sets
collection to the cluster config object.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that would be preferable to handling an Optional<T>
in this case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
✅
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, damn. Forgot to change the return type of IndexSetService#getDefault()
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adjusted the interfaces.
I added a migration to create a |
Doesn't do anything yet.
…setting This removes the `default` field from the IndexSetConfig database object and creates a DefaultIndexSetConfig object in the cluster config instead. Using a cluster config object makes it easier to ensure that only one index set is the default. There is no migration to convert the `default` flag to a DefaultIndexSetConfig in the cluster config because this only happened to people using alphas and betas and the user is able to just set the default index set via the API or in the UI. (UI upcoming) Also implement the IndexSetsResource#setDefault() method to actually set the default in cluster config.
4b5d15e
to
b9fc166
Compare
Rebased against master. |
@@ -529,6 +529,12 @@ | |||
<scope>test</scope> | |||
</dependency> | |||
<dependency> | |||
<groupId>com.github.fakemongo</groupId> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a duplicate:
graylog2-server/graylog-project-parent/pom.xml
Lines 531 to 548 in b9fc166
<dependency> | |
<groupId>com.github.fakemongo</groupId> | |
<artifactId>fongo</artifactId> | |
<version>${fongo.version}</version> | |
<scope>test</scope> | |
</dependency> | |
<dependency> | |
<groupId>org.jukito</groupId> | |
<artifactId>jukito</artifactId> | |
<version>${jukito.version}</version> | |
<scope>test</scope> | |
</dependency> | |
<dependency> | |
<groupId>com.github.fakemongo</groupId> | |
<artifactId>fongo</artifactId> | |
<version>${fongo.version}</version> | |
<scope>test</scope> | |
</dependency> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lol, for some reason I couldn't import the FongoRule class before adding this. Must have been something else.
Since the default index must be there, we do not return an Optional but throw an IllegalStateException if the default index set cannot be found.
indexSet.cycle(); | ||
indexSet.cycle(); | ||
} catch (IllegalStateException e) { | ||
throw new NotFoundException("Default index set not found"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would throw an InternalServerErrorException
here (or simply let the IllegalStateException
bubble up which results in the same) because it's actually an error in the Graylog system, not a wrong request by the client.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
✅
final IndexSet indexSet = indexSetRegistry.getDefault(); | ||
return DeflectorSummary.create(indexSet.isUp(), indexSet.getCurrentActualTargetIndex()); | ||
} catch (IllegalStateException e) { | ||
throw new NotFoundException("Default index set not found"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would throw an InternalServerErrorException
here (or simply let the IllegalStateException
bubble up which results in the same) because it's actually an error in the Graylog system, not a wrong request by the client.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
✅
An index set can be marked as
default
inIndexSetConfig
. This cannot be changed by the user at the moment, but should be. We have to ensure that only one index set is the default!Since we are using MongoDB, there are no cross-document atomic operations or transactions so we have to take care of that ourselves.
Idea 1: Use a timestamp
We are currently using a boolean
default
field in theindex_sets
database collection to mark the default index set. Because of the lack of transactions this is not really safe to update for several documents at once. (settrue
on the new default, setfalse
on all others)As an alternative to the boolean flag we could use a timestamp field like
default_since
or similar that will be set to the current time when a "set default index set" request comes in. To determine the default we have to load the index set with the newest timestamp. By using a unique index on the timestamp field, we would avoid duplicate timestamps.Idea 2: Coordinate on master
Another idea was to publish an event on the cluster event bus to set the default index set. All nodes subscribe to the event but only the master node will actually handle it. Since only one node should be master and the events are not processed in parallel, this should avoid interleaving writes to the index sets collection. It doesn't guarantee any order though.
This is more complicated than the timestamp idea beacuse it involves the cluster event bus and needs an event handler. It also makes the request to set the default index set async so we have to deal with that in the UI. It also fails when there is more than one active master node.
Other Ideas?
Please let me know if you have different, better ideas. Let's discuss!