Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Nio/CPU issue and CI failures #1543

Merged
merged 3 commits into from
May 13, 2016

Conversation

rohityadavcloud
Copy link
Member

  • Reverts ea2286 that introduced a wakeup on each connection loop run.
  • In SSL handshake code removes delegated tasks to be run in separate threads.

/cc @kiwiflyer @swill @jburwell and others for review

@kiwiflyer please help me test this fix and share if it makes the NioConnection robust now, without having the selector consume a lot of CPU. Thanks.

- Reverts ea2286 that introduced a wakeup on each connection loop run.
- In SSL handshake code removes delegated tasks to be run in separate threads.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
if (s_logger.isTraceEnabled()) {
s_logger.trace("SSL: Running delegated task!");
}
task.run();
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jburwell thanks for pointing out, I've made the runnable task run in the current thread only. The delegated tasks in case of SSL handshake could be resolving domain names to address.

@kiwiflyer
Copy link
Contributor

@rhtyd This patch seems to have fixed the cpu issue. I've got this running on a hardware lab right now.
We'll also install this on a larger lab tomorrow and stress test the agent a bit.

Thought you'd like to know before I hit the sack for the night!

@swill
Copy link
Contributor

swill commented May 13, 2016

Thanks for the fix @rhtyd and for testing this @kiwiflyer. 👍

@rohityadavcloud
Copy link
Member Author

@kiwiflyer Thanks you for testing this so quickly and sharing.

- Fixes oobm integration test to skip if known ipmitool bug is hit
- Fixes ProcessTest unit test case to use sleep
- Removes redundant unit test that covers code in ProcessTest

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
@rohityadavcloud rohityadavcloud changed the title CLOUDSTACK-9348: Make NioConnectio loop less aggressive CLOUDSTACK-9348: Fix Nio/CPU issue and CI failures May 13, 2016
@rohityadavcloud rohityadavcloud changed the title CLOUDSTACK-9348: Fix Nio/CPU issue and CI failures Fix Nio/CPU issue and CI failures May 13, 2016
@abhinandanprateek
Copy link
Contributor

LGTM based on code review.

@DaanHoogland
Copy link
Contributor

code LGTM. trusting @kiwiflyer I will probably test later but am a bit stuck today.

Changes the cleanup sequence as the cleanup code causes exceptions in
several Travis runs such as:
https://travis-ci.org/apache/cloudstack/jobs/129925224

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
@rohityadavcloud
Copy link
Member Author

I've also fixed a integration/component test that has failed on Travis several time for various PRs due to cleanup order (tearDown issue):

==== Marvin Init Successful ====
=== TestName: test_listVolume_as_domainadmin | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_cross_domainid | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_cross_domainid_accountid | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_domainid | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_domainid_accountid | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_domainid_accountid_listall_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_domainid_accountid_listall_false_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_domainid_accountid_listall_false_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_domainid_accountid_listall_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_domainid_accountid_listall_true_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_domainid_accountid_listall_true_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_domainid_accountid_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_domainid_accountid_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_domainid_listall_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_domainid_listall_false_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_domainid_listall_false_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_domainid_listall_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_domainid_listall_true_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_domainid_listall_true_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_domainid_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_domainid_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_listall_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_listall_false_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_listall_false_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_listall_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_listall_true_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_listall_true_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_domainadmin_true_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_cross_domainid | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_cross_domainid_accountid | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_domainid | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_domainid__rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_domainid_accountid | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_domainid_accountid_listall_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_domainid_accountid_listall_false_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_domainid_accountid_listall_false_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_domainid_accountid_listall_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_domainid_accountid_listall_true_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_domainid_accountid_listall_true_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_domainid_accountid_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_domainid_accountid_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_domainid_listall_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_domainid_listall_false_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_domainid_listall_false_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_domainid_listall_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_domainid_listall_true_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_domainid_listall_true_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_domainid_true_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_listall_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_listall_false_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_listall_false_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_listall_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_listall_true_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_listall_true_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_regularuser_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_domainid | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_domainid_accountid | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_domainid_accountid_listall_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_domainid_accountid_listall_false_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_domainid_accountid_listall_false_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_domainid_accountid_listall_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_domainid_accountid_listall_true_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_domainid_accountid_listall_true_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_domainid_accountid_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_domainid_accountid_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_domainid_listall_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_domainid_listall_false_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_domainid_listall_false_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_domainid_listall_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_domainid_listall_true_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_domainid_listall_true_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_domainid_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_domainid_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_listall_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_listall_false_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_listall_false_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_listall_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_listall_true_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_listall_true_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_rec_false | Status : SUCCESS ===

=== TestName: test_listVolume_as_rootadmin_rec_true | Status : SUCCESS ===

=== TestName: test_listVolume_by_id_as_domainadmin_ownedbyusersindomain | Status : SUCCESS ===

=== TestName: test_listVolume_by_id_as_domainadmin_ownedbyusersinsubdomain | Status : SUCCESS ===

=== TestName: test_listVolume_by_id_as_domainadmin_ownedbyusersinsubdomain2 | Status : SUCCESS ===

=== TestName: test_listVolume_by_id_as_domainadmin_ownedbyusersnotindomain | Status : SUCCESS ===

=== TestName: test_listVolume_by_id_as_domainadmin_owns | Status : SUCCESS ===

=== TestName: test_listVolume_by_id_as_rootadmin_Volumesownedbyothers | Status : SUCCESS ===

=== TestName: test_listVolume_by_id_as_rootadmin_owns | Status : SUCCESS ===

=== TestName: test_listVolume_by_id_as_user_own | Status : SUCCESS ===

=== TestName: test_listVolume_by_id_as_user_volumefromotherdomain | Status : SUCCESS ===

=== TestName: test_listVolume_by_id_as_user_volumefromsamedomaindifferentaccount | Status : SUCCESS ===

@swill
Copy link
Contributor

swill commented May 13, 2016

Most of these things are fixes to Jenkins and travis issues, so those jobs being green is good validation of the fixes. The main functionality change has been validated by @kiwiflyer, so I will get this merged right away. Thanks for being on top of this @rhtyd, I really appreciate it. 👍

@asfgit asfgit merged commit acc781d into apache:master May 13, 2016
asfgit pushed a commit that referenced this pull request May 13, 2016
Fix Nio/CPU issue and CI failures- Reverts ea2286 that introduced a wakeup on each connection loop run.
- In SSL handshake code removes delegated tasks to be run in separate threads.

/cc @kiwiflyer @swill @jburwell and others for review

@kiwiflyer please help me test this fix and share if it makes the NioConnection robust now, without having the selector consume a lot of CPU. Thanks.

* pr/1543:
  test: fix cleanup sequence for test_acl_listvolume test
  CLOUDSTACK-9299: Fix test failures on CI
  CLOUDSTACK-9348: Make NioConnectio loop less aggressive

Signed-off-by: Will Stevens <williamstevens@gmail.com>
@rohityadavcloud
Copy link
Member Author

Thanks @swill

@swill
Copy link
Contributor

swill commented May 13, 2016

Thanks for the effort stabilizing Jenkins and Travis the last few days. This has been super helpful as we get close to the freeze. Very much appreciated. Most of the runs are coming back clean now. 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants