HDDS-5766. Speed up some OM tests by skipping SCM safemode check #2662
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Updated tests ran in: Saving 208 + 291 + 134 + 29 = 662 seconds saved (~ 11 minutes). |
adoroszlai
reviewed
Sep 20, 2021
| @@ -367,6 +376,13 @@ public void testAclOperations() throws IOException { | |||
|
|
|||
| @Test | |||
| public void testAclOperationsHA() throws Exception { | |||
| // This test needs a cluster with DNs and SCM to wait on safemode | |||
| cluster.shutdown(); | |||
There was a problem hiding this comment.
This looks a bit strange. Would it be better to get rid of @Before and explicitly create cluster with desired properties for each test? (Similar to what TestOzoneManagerListVolumes did before HDDS-5717.)
There was a problem hiding this comment.
Yea I think you are correct. I have pushed a new commit that refactors things to ensure we only get the cluster we need for each test.
adoroszlai
approved these changes
Sep 21, 2021
errose28
added a commit
to errose28/ozone
that referenced
this pull request
Sep 30, 2021
* HDDS-4944: (81 commits) HDDS-5750. [Multi-Tenant] GetS3Secret should retrieve secret from new tables as well (apache#2649) HDDS-5476. [Multi-Tenant] Support Ozone s3 authentication with arbitrary accessId that is not same as the kerberos ID (apache#2635) HDDS-5770. Silent failures of k3s install are difficult to debug (apache#2667) HDDS-5759. Bump aspectj version (apache#2658) HDDS-5773. Avoid code duplication for mini cluster without datanodes (apache#2669) HDDS-5691. Restrict Recon NSSummaryEndpoint and ContainerEndpoint to admins. (apache#2638) HDDS-5771. Speed up TestDatanodeHddsVolumeFailureToleration by reducing dead interval (apache#2668) HDDS-5767. Unit check may timeout (apache#2664) HDDS-5765. Test cluster provider possibly returns null (apache#2663) HDDS-5768. Skip safemode check in TestOzoneManagerRocksDBLogging (apache#2665) HDDS-5766. Speed up some OM tests by skipping SCM safemode check (apache#2662) HDDS-5761. should not shutdown om when setting a bigger bucket quota than volume quota (apache#2659) HDDS-5758. Speed up TestKeyInputStream and TestChunkInputStream by combining some tests (apache#2656) HDDS-5607. remove container manager v1 code (apache#2525) HDDS-5753. Split parts of misc suite (apache#2654) HDDS-5751. Use Mini Cluster Provider to speed up TestHDDSUpgrade (apache#2650) HDDS-5728. ContainerBalancer should use remaining space to calculate utilization. (apache#2625) HDDS-5402 Support list node based on NodeOperationalState and NodeState options in printTopology CLI (apache#2645) HDDS-5749. Reuse mini-clusters in TestOzoneFsHAURLs (apache#2647) HDDS-5717. Refactor TestOzoneManagerListVolumes to reuse mini-ozone clusters (apache#2615) ...
sadanand48
pushed a commit
that referenced
this pull request
Oct 2, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Many of the OzoneManager integration tests do not really use SCM or the datanodes. Therefore we can skip the safemode check in SCM and start zero datanodes to make the mini-cluster come up faster.
With just this change, on my laptop:
The runtime for these tests on Github is:
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5766
How was this patch tested?
Existing tests.