[FLINK-33246][tests] Add AutoRescalingITCase #23510
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| builder.setParallelismForJobVertex(vertex.getID(), parallelism2, parallelism2); | ||
| } | ||
|
|
||
| restClusterClient.updateJobResourceRequirements(jobID, builder.build()).join(); |
There was a problem hiding this comment.
IIRC, this call doesn't wait for the requirements to be satisfied.
Should we wait after this call, that the actual DoP is actually what we want?
There was a problem hiding this comment.
I was looking into this and think it is not needed for this test because the test job will only start completing after it restarted from the rescale, so we can be sure that the rescaling happened first.
There was a problem hiding this comment.
Relying on that condition seems a bit fragile to me and makes it more difficult to understand.
- there were no restarts; they are disabled here, but I'm not sure if the restarts are disabled in every tests, and whether they will stay as such.
- the DoP might change if some TM is slow to start; the scheduler will first schedule with lower DoP, and once TM is up, will rescale to full DoP. That happened in some tests previously
flink-tests/src/test/java/org/apache/flink/test/checkpointing/AutoRescalingITCase.java
Outdated
Show resolved
Hide resolved
flink-runtime/src/test/java/org/apache/flink/runtime/testutils/CommonTestUtils.java
Outdated
Show resolved
Hide resolved
StefanRRichter
force-pushed
the
srichter-FLINK-33246
branch
2 times, most recently
from
882f41e
to
b0bb6d6
Compare
StefanRRichter
force-pushed
the
srichter-FLINK-33246
branch
from
b0bb6d6
to
5d6a42e
Compare
|
Closing in favor of #23591 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
Adds
AutoRescalingITCase, an adapted rescaling IT case that uses checkpointing and changing resource requirements to trigger rescaling.Does this pull request potentially affect one of the following parts:
@Public(Evolving): (no)Documentation