HDDS-4332: ListFileStatus - do lookup in directory and file tables

[rakeshadr]

What changes were proposed in this pull request?

https://issues.apache.org/jira/browse/HDDS-4332

What is the link to the Apache JIRA

This task is to perform look up of the user given key path in the directory, file and openFile tables.
OzoneFileSystem APIs:

• GetFileStatus
• listStatus

How was this patch tested?

Added UTs - TestHadoopDirTreeGeneratorV1, TestOzoneFileInterfacesV1, TestOzoneFileSystemV1...

[linyiqun]

@rakeshadr , thanks for working on this. Initial review comments from me.
Please have a look for this when you get a chance.

[rakeshadr]

@rakeshadr , thanks for working on this. Initial review comments from me.
Please have a look for this when you get a chance.

Thanks a lot for the review comments. I will update the PR.

[linyiqun]

Thanks for updating the PR, @rakeshadr . One further review comment below.

In additional, current test change not fully cover the test for listStatusV1. Example, current OzoneFS test change doesn't address the case for listStatus with other startKey specified.
I see the test class TestKeyManagerImpl does the good coverage for listStatus call, can we add a test unit like that or make a minor refactor based on that?

[rakeshadr]

Thanks for updating the PR, @rakeshadr . One further review comment below.

In additional, current test change not fully cover the test for listStatusV1. Example, current OzoneFS test change doesn't address the case for listStatus with other startKey specified.
I see the test class TestKeyManagerImpl does the good coverage for listStatus call, can we add a test unit like that or make a minor refactor based on that?

Thanks @linyiqun . Thats really a good point. I also looked at keyManagerImpl UT and it requires some refactoring effort to modify the createDir and other logic. I will work on and update you.

[bharatviswa504]

Thank You @rakeshadr for the contribution.
I have reviewed code changes, have few comments, pending test changes review.

[rakeshadr]

Thank You @rakeshadr for the contribution.
I have reviewed code changes, have few comments, pending test changes review.

Thanks a lot @bharatviswa504 for the review comments. I've updated the PR. Appreciate if you could provide clarifications on few of the comments.

[rakeshadr]

Thanks for updating the PR, @rakeshadr . One further review comment below.

In additional, current test change not fully cover the test for listStatusV1. Example, current OzoneFS test change doesn't address the case for listStatus with other startKey specified.
I see the test class TestKeyManagerImpl does the good coverage for listStatus call, can we add a test unit like that or make a minor refactor based on that?

I've created HDDS-4412 jira to refactor TestKeyManagerImpl unit testing. Today, it uses stale APIs and IMHO will refactor this in a separate task to make this patch comparatively smaller:-)

[rakeshadr]

Thanks @linyiqun for the detailed reviews and your quick responses really helped a lot. Appreciate if you could provide your overall vote/confirmation about the patch.

[linyiqun]

I've created HDDS-4412 jira to refactor TestKeyManagerImpl unit testing. Today, it uses stale APIs and IMHO will refactor this in a separate task to make this patch comparatively smaller:-)

Let's track that on HDDS-4412,

Thanks @linyiqun for the detailed reviews and your quick responses really helped a lot. Appreciate if you could provide your overall vote/confirmation about the patch.

+1. Give my approval, : ).

[bharatviswa504]

HDDS-4391 using Paths object has performance disadvantage, so can we see we can use Hadoop FS Path, as Paths use FileSystem and then initialize UnixPath.

We can look into this in further Jira.

One minor comment, take care of that in test opened Jira for refactoring.

[bharatviswa504]

Thank You @rakeshadr for the contribution and @linyiqun for the review.

[rakeshadr]

HDDS-4391 using Paths object has performance disadvantage, so can we see we can use Hadoop FS Path, as Paths use FileSystem and then initialize UnixPath.

We can look into this in further Jira.

One minor comment, take care of that in test opened Jira for refactoring.

Noted these points and will take care in the following jira tasks.

Thanks a lot @bharatviswa504, @linyiqun for the continuous support in reviews and committing the changes!