Skip to content

HDDS-5386. Add a NSSummaryTask to write NSSummary info into RDB#2389

Merged
smengcl merged 10 commits intoapache:masterfrom
yuangu002:HDDS-5386-3
Jul 13, 2021
Merged

HDDS-5386. Add a NSSummaryTask to write NSSummary info into RDB#2389
smengcl merged 10 commits intoapache:masterfrom
yuangu002:HDDS-5386-3

Conversation

@yuangu002
Copy link
Contributor

@yuangu002 yuangu002 commented Jul 6, 2021

What changes were proposed in this pull request?

FileSizeCountTask counts the number of bins inside each buckets.
Try to build on FileSizeCountTask to count what we need for NSSummary as well.
[6.28] Added an independent Recon task to process namespace summary info per objectID in the attempt to decouple with other task modules.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5386

How was this patch tested?

Unit test: org.apache.hadoop.ozone.recon.tasks.TestNSSummaryTask

@yuangu002 yuangu002 mentioned this pull request Jul 6, 2021
Closed
@yuangu002
Copy link
Contributor Author

yuangu002 commented Jul 6, 2021

@smengcl @avijayanhwx @vivekratnavel Could you guys review?

smengcl
smengcl reviewed Jul 7, 2021
View reviewed changes
Copy link
Contributor

@smengcl smengcl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @yuangu002. Looks good overall

yuangu002 added 3 commits July 8, 2021 16:27
@yuangu002
Added a field in NSSummary to track all subdirs of bucket/directory. …
70d0df3 
…Redesigned NSSummaryTask to write DirectoryTable updates to the new data structure. Refactored the source and tests to track updates only if FSO is enabled. Added more unit tests.
@yuangu002
type fix
af7978c
@yuangu002
retrigger tests
8e24cfa
avijayanhwx
avijayanhwx reviewed Jul 9, 2021
View reviewed changes
avijayanhwx
avijayanhwx approved these changes Jul 13, 2021
View reviewed changes
Copy link
Contributor

@avijayanhwx avijayanhwx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM +1

@smengcl smengcl merged commit 154c0c2 into apache:master Jul 13, 2021
@smengcl
Copy link
Contributor

smengcl commented Jul 13, 2021

Merged. Thanks @yuangu002 for the contribution. Thanks @avijayanhwx for the review.

errose28 added a commit to errose28/ozone that referenced this pull request Jul 15, 2021
@errose28
Merge branch 'master' into HDDS-5337
00eb1a4 
* master: (90 commits)
  HDDS-5296. Replication Manager should not sent replicate command to DN which is not IN_SERVICE (apache#2377)
  HDDS-3432. Enable TestBlockDeletion test cases. (apache#2393)
  HDDS-5386. Add a NSSummaryTask to write NSSummary info into RDB (apache#2389)
  HDDS-5384 OM refreshPipeline should not invoke the expensive OmKeyLocationInfoGroup.getLocationList() (apache#2367)
  HDDS-5426. Remove unused method parameter in XceiverClientGrpc.java (apache#2409)
  HDDS-5433. Rename hadooprpc proto compilation in pom to refer to proto2. (apache#2407)
  HDDS-5422. Avoid eager string formatting in preconditions (apache#2396)
  HDDS-5427. Either re-interrupt this method or rethrow the "InterruptedException" that can be caught here (apache#2404)
  HDDS-5394. Fix skipped volume check due to disk.check.min.gap. (apache#2373)
  HDDS-5428. Fix potential BigDecimal precision in StorageUnit.java (apache#2405)
  HDDS-5421. SCM throws NPE during JMX call. (apache#2398)
  HDDS-5400. Remove inflight actions in Replication Manager if node switches to not IN_SERVICE (apache#2381)
  HDDS-5424. Pipeline list command's output should be in local timezone. (apache#2400)
  HDDS-5416. Intermittent test failure in TestSCMPipelineManager#testPipelineReload (apache#2394)
  HDDS-5417. Fix json parse stackoverflow for cmd ozone debug container list. (apache#2391)
  HDDS-5415. Fix TestSCMNodeManager after merge of master at 1d8f972 into upgrade branch (apache#2388)
  Fix checkstyle after merge.
  HDDS-5336. Fix datanode capacity related race condition. (apache#2327)
  HDDS-5333. Remove getRequestType method from OM request classes.
  HDDS-5109. Track OM prepare intermittent integration test failure. (apache#2288)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@avijayanhwx avijayanhwx avijayanhwx approved these changes

@smengcl smengcl smengcl approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@yuangu002 @smengcl @avijayanhwx