Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Return cached segments stats if include_unloaded_segments is true #39698

Merged
merged 5 commits into from Mar 20, 2019
Merged

Return cached segments stats if include_unloaded_segments is true #39698

merged 5 commits into from Mar 20, 2019

Conversation

s1monw
Copy link
Contributor

@s1monw s1monw commented Mar 5, 2019

Today we don't return segments stats for closed indices which makes it
hard to tell how much memory such an index would require. With this change
we return the statistics if requested by setting include_unloaded_segments to
true on the rest request.

Relates to #39512

@s1monw
Return cached segments stats if include_unloaded_segments is true
a39ad3a 
Today we don't return segments stats for closed indices which makes it
hard to tell how much memory such an index would require. With this change
we return the statistics if requested by setting `include_unloaded_segments` to
true on the rest request.

Relates to elastic#39512/
@s1monw s1monw added >enhancement :Distributed/Engine Anything around managing Lucene and the Translog in an open shard. v8.0.0 v7.2.0 labels Mar 5, 2019
@s1monw s1monw requested review from tlrx and ywelsch March 5, 2019 13:48
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

tlrx
tlrx approved these changes Mar 7, 2019
View reviewed changes
Copy link
Member

@tlrx tlrx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

ywelsch
ywelsch suggested changes Mar 7, 2019
View reviewed changes
Copy link
Contributor

@ywelsch ywelsch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps also add a REST test showing that these stats are properly working for closed replicated indices? We can also do that as a follow-up if you want to keep this an engine-level change only.

server/src/main/java/org/elasticsearch/index/engine/NoOpEngine.java
}
return stats;
} else {
return super.segmentsStats(includeSegmentFileSizes, includeUnloadedSegments);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

calling this will fail on a NoopEngine?
Should we instead return an empty stats object?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm there is a test for this here that shows that it's not failing but returning an empty object.

@s1monw
Copy link
Contributor Author

s1monw commented Mar 9, 2019

Perhaps also add a REST test showing that these stats are properly working for closed replicated indices? We can also do that as a follow-up if you want to keep this an engine-level change only.

I will add one.

@s1monw
Copy link
Contributor Author

s1monw commented Mar 11, 2019

@ywelsch I pushed updates

@ywelsch ywelsch self-requested a review March 12, 2019 15:34
ywelsch
ywelsch approved these changes Mar 14, 2019
View reviewed changes
Copy link
Contributor

@ywelsch ywelsch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've left one more question. Looking good otherwise.

rest-api-spec/src/main/resources/rest-api-spec/api/indices.stats.json
"default" : "open",
"description" : "Whether to expand wildcard expression to concrete indices that are open, closed or both."
},
"forbid_closed_indices": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we introducing this parameter only for BWC of behavior in 7.x or do you think this could still be useful in 8.0? In 8.0 I would expect this parameter to be false by default, and I'm not sure why I would ever explicitly set it to true.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ywelsch I must have missed something, the tests failed if I'd not have that in there. Maybe I ran tests before the relevant change was introduced?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there is a misunderstanding. What I meant by "In 8.0 I would expect this parameter to be false by default" is that I think we should make it default to false in 8.0, so that if you include closed indices in your indices stats request, that the request does not fail by default.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ywelsch I would like to merge as is and then go and change the defaults for stats in a separate pr. ok with you?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, let's do it like that

@s1monw s1monw merged commit 159eb72 into elastic:master Mar 20, 2019
@s1monw s1monw deleted the stats_on_noop_engine branch March 20, 2019 10:24
s1monw added a commit that referenced this pull request Mar 20, 2019
@s1monw
Return cached segments stats if include_unloaded_segments is true (#…
235f579 
…39698)

Today we don't return segments stats for closed indices which makes it
hard to tell how much memory such an index would require. With this change
we return the statistics if requested by setting `include_unloaded_segments` to
true on the rest request.

Relates to #39512
s1monw added a commit that referenced this pull request Mar 20, 2019
@s1monw
Bump BWC version to 7.1 after backport
b8f8ed7 
Relates to #39698
pgomulka pushed a commit to pgomulka/elasticsearch that referenced this pull request Mar 25, 2019
@s1monw @pgomulka
Return cached segments stats if include_unloaded_segments is true (e…
8e714f1 
…lastic#39698)

Today we don't return segments stats for closed indices which makes it
hard to tell how much memory such an index would require. With this change
we return the statistics if requested by setting `include_unloaded_segments` to
true on the rest request.

Relates to elastic#39512
pgomulka pushed a commit to pgomulka/elasticsearch that referenced this pull request Mar 25, 2019
@s1monw @pgomulka
Bump BWC version to 7.1 after backport
0e7c581 
Relates to elastic#39698
@tlrx tlrx mentioned this pull request May 7, 2019
Closed
50 tasks
@codebrain codebrain mentioned this pull request Aug 5, 2019
Closed
37 tasks
@gwbrown gwbrown mentioned this pull request Oct 2, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ywelsch ywelsch ywelsch approved these changes

@tlrx tlrx tlrx approved these changes

Assignees
No one assigned
Labels
:Distributed/Engine Anything around managing Lucene and the Translog in an open shard. >enhancement v7.2.0 v8.0.0-alpha1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@s1monw @elasticmachine @tlrx @ywelsch @jakelandis