-
Notifications
You must be signed in to change notification settings - Fork 484
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HDDS-8553. Improve scanner integration tests. #4936
HDDS-8553. Improve scanner integration tests. #4936
Conversation
* Integration tests for the background container data scanner. This scanner | ||
* checks all data and metadata in the container. | ||
*/ | ||
@RunWith(Parameterized.class) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: since it is new code, it would be great if you use JUnit-5 and its parameterized tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@errose28 LGTM +1, dev config is introduced to have a control for test case, right?
Thanks @errose28 for the patch, @hemantk-12, @sumitagrawl for the review.
100% agree, we should make this a recommendation. In this case I think we can file a follow-up task: the tests are useful in their current form and there is no harm in having them as JUnit4 for some time. |
* master: (96 commits) HDDS-8586 Recon. - API for Count of deletePending keys and amount of data mapped to such keys. (apache#4923) HDDS-8908. Intermittent failure in TestBlockDeletion#testBlockDeletion (apache#4958) HDDS-8910. Replace LockManager with striped lock in ContainerStateManager (apache#4962) HDDS-8917. Move protobuf conversion out of the lock in PipelineStateManagerImpl (apache#4965) HDDS-8825. Use apache/hadoop 3.3.5 docker image (apache#4963) HDDS-8906. Avoid stream when getting in-service healthy nodes (apache#4960) HDDS-8907. Store volume count when storage report is updated (apache#4957) HDDS-8905. PipelineManager metrics should not be synchronized (apache#4959) HDDS-8553. Improve scanner integration tests. (apache#4936) HDDS-8854. Avoid unnecessary DatanodeDetails creation for NodeStateManager lookup (apache#4925) HDDS-8315. [Snapshot] Added unit tests for SnapshotDiffManager (apache#4716) HDDS-7968. [Snapshot] Improve KeyDeletingService to reclaim eligible key blocks in snapshot's deletedTable (apache#4935) HDDS-8838. Update default datanode check empty containter on disk to false (apache#4937) HDDS-8763. Support RocksDB iterator with ByteBuffer. (apache#4942) HDDS-8543. FSO directory should reflect bucket/cluster default replication (apache#4947) HDDS-8898. Replication limit should not be less than reconstruction weight (apache#4954) HDDS-8739. Snapdiff should return complete absolute path in Diff Entry (apache#4823) HDDS-8908. Mark TestBlockDeletion#testBlockDeletion as flaky HDDS-8534. Support asynchronous service logging (apache#4663) HDDS-8879. Cleanup SecurityConfig and related class initialization (apache#4921) ...
* tmp-dir-refactor: (99 commits) HDDS-8586 Recon. - API for Count of deletePending keys and amount of data mapped to such keys. (apache#4923) Fix SCM HA finalization compat test HDDS-8908. Intermittent failure in TestBlockDeletion#testBlockDeletion (apache#4958) HDDS-8910. Replace LockManager with striped lock in ContainerStateManager (apache#4962) HDDS-8917. Move protobuf conversion out of the lock in PipelineStateManagerImpl (apache#4965) HDDS-8825. Use apache/hadoop 3.3.5 docker image (apache#4963) HDDS-8906. Avoid stream when getting in-service healthy nodes (apache#4960) HDDS-8907. Store volume count when storage report is updated (apache#4957) HDDS-8905. PipelineManager metrics should not be synchronized (apache#4959) HDDS-8553. Improve scanner integration tests. (apache#4936) HDDS-8854. Avoid unnecessary DatanodeDetails creation for NodeStateManager lookup (apache#4925) HDDS-8315. [Snapshot] Added unit tests for SnapshotDiffManager (apache#4716) HDDS-7968. [Snapshot] Improve KeyDeletingService to reclaim eligible key blocks in snapshot's deletedTable (apache#4935) HDDS-8838. Update default datanode check empty containter on disk to false (apache#4937) HDDS-8763. Support RocksDB iterator with ByteBuffer. (apache#4942) HDDS-8543. FSO directory should reflect bucket/cluster default replication (apache#4947) HDDS-8898. Replication limit should not be less than reconstruction weight (apache#4954) HDDS-8739. Snapdiff should return complete absolute path in Diff Entry (apache#4823) HDDS-8908. Mark TestBlockDeletion#testBlockDeletion as flaky HDDS-8534. Support asynchronous service logging (apache#4663) ...
* master: (79 commits) HDDS-8914. Datanode may fail to start due to duplicate VolumeInfoMetrics (apache#4966) HDDS-8921. Add support for EC in Freon SCM block generator (apache#4982) HDDS-8927. Metadata scanner should not scan unhealthy containers. (apache#4976) HDDS-8929. Avoid list allocation for pipeline search (apache#4980) HDDS-8778. Support recursive volume delete using Ozone sh command. (apache#4842) HDDS-8885. Quota repair count enable quota feature for old bucket/volume. (apache#4941) HDDS-8771. Refactor volume level tmp directory for generic usage. (apache#4838) HDDS-8922. Random EC read pipeline ID causes XceiverClient cache churn (apache#4971) HDDS-8586 Recon. - API for Count of deletePending keys and amount of data mapped to such keys. (apache#4923) HDDS-8908. Intermittent failure in TestBlockDeletion#testBlockDeletion (apache#4958) HDDS-8910. Replace LockManager with striped lock in ContainerStateManager (apache#4962) HDDS-8917. Move protobuf conversion out of the lock in PipelineStateManagerImpl (apache#4965) HDDS-8825. Use apache/hadoop 3.3.5 docker image (apache#4963) HDDS-8906. Avoid stream when getting in-service healthy nodes (apache#4960) HDDS-8907. Store volume count when storage report is updated (apache#4957) HDDS-8905. PipelineManager metrics should not be synchronized (apache#4959) HDDS-8553. Improve scanner integration tests. (apache#4936) HDDS-8854. Avoid unnecessary DatanodeDetails creation for NodeStateManager lookup (apache#4925) HDDS-8315. [Snapshot] Added unit tests for SnapshotDiffManager (apache#4716) HDDS-7968. [Snapshot] Improve KeyDeletingService to reclaim eligible key blocks in snapshot's deletedTable (apache#4935) ...
What changes were proposed in this pull request?
Adds more robust integration testing of the background and on demand metadata and data container scanners. Improvements include:
A few bugs around loading corrupt container yaml files have been fixed here as well to allow the tests pass.
What is the link to the Apache JIRA
HDDS-8553
How was this patch tested?