Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-10099] [test] Rework YarnResourceManagerTest #6499

Closed
wants to merge 2 commits into from

Conversation

tisonkun
Copy link
Member

@tisonkun tisonkun commented Aug 6, 2018

What is the purpose of the change

to avoid OOM and gather mock stuff so that we can replace it once possible.

also some modifies to structure the test.

Brief change log

  • introduce mockContainer and mockContainerStatus
  • introduce runTest

Verifying this change

the test is self-verified

@tisonkun tisonkun force-pushed the hotfix-rework-test branch 2 times, most recently from 9a93c27 to 0e849c5 Compare August 6, 2018 08:39
@tisonkun
Copy link
Member Author

tisonkun commented Aug 6, 2018

cc @GJL @tillrohrmann

@tillrohrmann tillrohrmann self-requested a review August 6, 2018 10:07
Copy link
Contributor

@tillrohrmann tillrohrmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution @tisonkun. The changes look good to me. I had some minor comments which we should address before merging. It would also be great if you could create a JIRA issue for this fix, since it is not super trivial. Please also update the commit accordingly.

@@ -258,7 +272,7 @@ protected void runAsync(final Runnable runnable) {
rmServices.metricRegistry,
rmServices.jobLeaderIdService,
new ClusterInformation("localhost", 1234),
fatalErrorHandler,
testingFatalErrorHandler,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice one :-)


@Override
public void setId(ContainerId containerId) {

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we throw an UnsupportedOperationException in order to be on the safe side if someone should ever call this method?


@Override
public void setNodeId(NodeId nodeId) {

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here with UnsupportedOperationException


@Override
public Token getContainerToken() {
return null;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it safe to return null here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the ContainerPBImpl it looks safe to do so.


@Override
public void setContainerToken(Token token) {

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UnsupportedOperationException.


@Override
public void setNodeHttpAddress(String s) {

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UnsupportedOperationException


@Override
public void setState(ContainerState containerState) {

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UnsupportedOperationException


@Override
public void setExitStatus(int exitStatus) {

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UnsupportedOperationException?


@Override
public void setDiagnostics(String diagnostics) {

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UnsupportedOperationException?

@@ -415,13 +553,14 @@ public void testDeleteApplicationFiles() throws Exception {

resourceManager.deregisterApplication(ApplicationStatus.SUCCEEDED, null);
assertFalse("YARN application directory was not removed", Files.exists(applicationDir.toPath()));

stopResourceManager();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we could add a function

runTest(RunnableWithException testMethod) {
  startResourceManager();
  try {
    testMethod.run()
  } finally {
    stopResourceManager
  }
}

This method could then also be used by the testOnContainerCompleted test.

rpcService.stopService().get();
}
}

static class TestingContainer extends Container {
Copy link
Member

@GJL GJL Aug 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not convinced about this. The mock was introduced with FLINK-8177 because Hadoop 2.9 added new methods.
See aa1f833#diff-73ea3caf58623db1fe68ef42ceadaa88L239

Copy link
Member

@GJL GJL left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution, @tisonkun. I am afraid that we cannot replace the container mock because a previous container implementation was removed for a reason: FLINK-8177

@tisonkun tisonkun changed the title [hotfix] loose dependency to mockito [FLINK-10099] [test] Rework YarnResourceManagerTest Aug 8, 2018
@tisonkun
Copy link
Member Author

tisonkun commented Aug 8, 2018

@tillrohrmann @GJL thanks for your reviews!
I revert the change replacing mocks, but gather them so that we can replace it once possible. Besides, add a runTest as @tillrohrmann suggested.

@tillrohrmann
Copy link
Contributor

Thanks for addressing our comments @tisonkun. Merging this PR.

@asfgit asfgit closed this in 3f40783 Aug 10, 2018
@tisonkun tisonkun deleted the hotfix-rework-test branch August 10, 2018 10:00
Clarkkkkk pushed a commit to Clarkkkkk/flink that referenced this pull request Mar 7, 2019
Introduce methods to mock a Yarn Container and ContainerStatus.

Properly shutdown a started ResourceManager.

This closes apache#6499.

(cherry picked from commit 3f40783)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants