Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor _cluster/stats .nodes.fs deduplication #94780

Merged

Conversation

joegallo
Copy link
Contributor

@joegallo joegallo commented Mar 27, 2023

Related to #94744 and to #24472

Refactors the ClusterStatsNodes fs deduplication logic in order to make it easier to test, then adds four test scenarios (two of which currently pass, the two that fail are bugs that should be fixed -- I'll be fixing them in other PRs).

In the response from GET _cluster/stats, this section:

{
  [...]
  "nodes": {
    [...]
    "fs": {
      "total_in_bytes": 821412495360,
      "free_in_bytes": 820742066176,
      "available_in_bytes": 820742066176
    },
    [...]
  }
  [...]
}

is currently governed by some logic that de-duplicates by ip address, this PR twiddles the code of that logic around a little bit without actually changing the logic.

@joegallo joegallo added :Data Management/Stats Statistics tracking and retrieval APIs >refactoring Team:Data Management Meta label for data/management team v8.8.0 labels Mar 27, 2023
@joegallo joegallo requested a review from masseyke March 27, 2023 14:40
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@joegallo joegallo changed the title Refactor ClusterStats fs deduplication Refactor _cluster/stats fs deduplication Mar 27, 2023
@joegallo joegallo changed the title Refactor _cluster/stats fs deduplication Refactor _cluster/stats .nodes.fs deduplication Mar 27, 2023
deduplicator.add(address2, newFsInfo(path2));
FsInfo.Path total = deduplicator.getTotal();

// wait a second, this is the super-special case -- you can't actually have two nodes doing this unless something
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What very interesting thing causes this? It seems to me like it would be a bug that we even got here, right?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not that I'm arguing that we ought to be asserting the opposite of what you're saing -- I'm just wondering if behavior in this case ought to even be defined.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a pretty common case in practice, even if it's not at all common in theory. I'll catch up with you offline.

@joegallo joegallo requested a review from masseyke March 27, 2023 19:18
Copy link
Member

@masseyke masseyke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@joegallo joegallo merged commit 790a43a into elastic:main Mar 27, 2023
@joegallo joegallo deleted the refactor-cluster-stats-fs-deduplication branch March 27, 2023 19:20
joegallo added a commit to joegallo/elasticsearch that referenced this pull request Mar 28, 2023
joegallo added a commit to joegallo/elasticsearch that referenced this pull request Mar 28, 2023
elasticsearchmachine pushed a commit that referenced this pull request Mar 28, 2023
* Refactor _cluster/stats .nodes.fs deduplication (#94780)

* Fix FsInfo device deduplication (#94744)

* Fix _cluster/stats .nodes.fs deduplication (#94798)
elasticsearchmachine pushed a commit that referenced this pull request Mar 28, 2023
* Refactor _cluster/stats .nodes.fs deduplication (#94780)

* Fix FsInfo device deduplication (#94744)

* Fix _cluster/stats .nodes.fs deduplication (#94798)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Stats Statistics tracking and retrieval APIs >refactoring Team:Data Management Meta label for data/management team v7.17.10 v8.7.1 v8.8.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants