Add validators to check head node instance type and shared storage type w.r.t cluster size #6623
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hanwen-cluster
added
the
skip-changelog-update
hehe7318
reviewed
Dec 13, 2024
| f"Head node instance type {head_node_instance_type} has {head_node_memory} GB of memory. " | ||
| f"Please choose a head node instance type with at least {required_memory} GB of memory" | ||
| f" to manage {total_max_compute_nodes} compute nodes.", | ||
| FailureLevel.ERROR, |
Contributor
There was a problem hiding this comment.
Why decide to use FailureLevel.Error here but not a Warning? I think customers have the right to decide what instance type they pay for, but we have a responsibility to warn them.
Contributor
Author
There was a problem hiding this comment.
Because the requirement is very relaxed.
For example, the validator allows t3.micro to manage 10 nodes, t3.medium to manage 85 nodes, t3.xlarge to manage infinite nodes. If the requirement is violated, it is very likely the scaling up would fail.
Also, the customer could suppress the validation anyway
hanwen-cluster
force-pushed
the
release-3.12
branch
from
e8d4b13 to
ec5c7db
Compare
| if total_max_compute_nodes > 100: | ||
| self._add_failure( | ||
| "EBS shared storage is mounted on the head node and shared to the compute nodes. " | ||
| "This is a performance bottle neck if the compute nodes rely on this shared storage. " |
Contributor
There was a problem hiding this comment.
It's not clear why this is a performance bottleneck. Performance of what components?
hanwen-cluster
force-pushed
the
release-3.12
branch
from
ec5c7db to
e42cd15
Compare
hanwen-cluster
force-pushed
the
release-3.12
branch
2 times, most recently
from
ec5c7db to
3378898
Compare
|
|
||
| class HeadNodeMemorySizeValidator(Validator): | ||
| """ | ||
| Head Node Instance Type Validator. |
Contributor
There was a problem hiding this comment.
Probably need to change this to match
| if total_max_compute_nodes > 100: | ||
| self._add_failure( | ||
| "EBS shared storage is mounted on the head node and shared to the compute nodes. " | ||
| "Therefore, the head node network bandwidth is a performance bottle neck " |
Contributor
There was a problem hiding this comment.
Maybe call it network performance? It’s still ambiguous like this
…pe w.r.t cluster size The requirements set in these validators are minimum. Users should leave more safety margin considering their workloads. Signed-off-by: Hanwen <hanwenli@amazon.com>
hanwen-cluster
force-pushed
the
release-3.12
branch
from
3378898 to
d5fe1b8
Compare
hanwen-cluster
added a commit
to hanwen-cluster/aws-parallelcluster
that referenced
this pull request
Dec 20, 2024
These tests started to fail after aws#6623. The tests didn't encounter scaling issue because the tests were not launching many compute nodes to use the full capacity of the cluster. Signed-off-by: Hanwen <hanwenli@amazon.com>
hanwen-cluster
added a commit
that referenced
this pull request
Dec 20, 2024
These tests started to fail after #6623. The tests didn't encounter scaling issue because the tests were not launching many compute nodes to use the full capacity of the cluster. Signed-off-by: Hanwen <hanwenli@amazon.com>
hgreebe
pushed a commit
to hgreebe/aws-parallelcluster
that referenced
this pull request
Feb 26, 2025
These tests started to fail after aws#6623. The tests didn't encounter scaling issue because the tests were not launching many compute nodes to use the full capacity of the cluster. Signed-off-by: Hanwen <hanwenli@amazon.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The requirements set in these validators are minimum. Users should leave more safety margin considering their workloads.
Tests
Please review the guidelines for contributing and Pull Request Instructions.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.