-
Notifications
You must be signed in to change notification settings - Fork 13.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FLINK-33246][tests] Add AutoRescalingITCase #23510
Conversation
builder.setParallelismForJobVertex(vertex.getID(), parallelism2, parallelism2); | ||
} | ||
|
||
restClusterClient.updateJobResourceRequirements(jobID, builder.build()).join(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IIRC, this call doesn't wait for the requirements to be satisfied.
Should we wait after this call, that the actual DoP is actually what we want?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was looking into this and think it is not needed for this test because the test job will only start completing after it restarted from the rescale, so we can be sure that the rescaling happened first.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Relying on that condition seems a bit fragile to me and makes it more difficult to understand.
- there were no restarts; they are disabled here, but I'm not sure if the restarts are disabled in every tests, and whether they will stay as such.
- the DoP might change if some TM is slow to start; the scheduler will first schedule with lower DoP, and once TM is up, will rescale to full DoP. That happened in some tests previously
flink-tests/src/test/java/org/apache/flink/test/checkpointing/AutoRescalingITCase.java
Outdated
Show resolved
Hide resolved
flink-runtime/src/test/java/org/apache/flink/runtime/testutils/CommonTestUtils.java
Outdated
Show resolved
Hide resolved
882f41e
to
b0bb6d6
Compare
b0bb6d6
to
5d6a42e
Compare
Closing in favor of #23591 |
What is the purpose of the change
Adds
AutoRescalingITCase
, an adapted rescaling IT case that uses checkpointing and changing resource requirements to trigger rescaling.Does this pull request potentially affect one of the following parts:
@Public(Evolving)
: (no)Documentation