HDDS-7253. Fix exception when '/' in key name

[xichen01]

This is a resubmitted MR, the old one is #3774
This MR fixes an issue that may be making ITestOzoneContractMkdir flaky.

What changes were proposed in this pull request?

Fix exception when '/' in key name

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-7253

How was this patch tested?

How to reproduce the bug

• First put a key that containing '/'

[root@Linux /root/ozone]% bin/ozone sh key put s3v/testbucket/dir1/dir2/key1 ~/testfile
[root@Linux /root/ozone]% bin/ozone sh key ls s3v/testbucket/ | grep name
  "name" : "dir1/dir2/key1"

• Execute command

ls command for keys containing '/'
before this commit

[root@Linux /root/ozone]% bin/ozone fs -ls ofs://localhost/s3v/testbucket/
Found 1 items
drwxrwxrwx   - root root          0 2022-09-19 14:11 ofs://localhost/s3v/testbucket/dir1
[root@Linux /root/ozone]% bin/ozone fs -ls ofs://localhost/s3v/testbucket/dir1
ls: `ofs://localhost/s3v/testbucket/dir1': No such file or directory

after this commit

[root@Linux /root/ozone]% bin/ozone fs -ls ofs://localhost/s3v/testbucket/
Found 1 items
drwxrwxrwx   - root root          0 2022-09-19 14:37 ofs://localhost/s3v/testbucket/dir1
[root@Linux /root/ozone]% bin/ozone fs -ls ofs://localhost/s3v/testbucket/dir1
Found 1 items
drwxrwxrwx   - root root          0 2022-09-19 14:37 ofs://localhost/s3v/testbucket/dir1/dir2

for mv, mkdir, count command for keys containing '/'

before this commit

[root@Linux /root/ozone]% bin/ozone fs -count ofs://localhost/s3v/testbucket/
count: dir1: No such file or directory!
[root@Linux /root/ozone]% bin/ozone fs -mv ofs://localhost/s3v/testbucket/dir1 ofs://localhost/s3v/testbucket/dir2
mv: `ofs://localhost/s3v/testbucket/dir1': No such file or directory
[root@Linux /root/ozone]% bin/ozone fs -mkdir ofs://localhost/s3v/testbucket/dir1/dir2/key2
mkdir: `ofs://localhost/s3v/testbucket/dir1/dir2': No such file or directory

after this commit

[root@Linux /root/ozone]% bin/ozone fs -count ofs://localhost/s3v/testbucket/
           3            1            1048576 ofs://localhost/s3v/testbucket
[root@Linux /root/ozone]% bin/ozone fs -mv ofs://localhost/s3v/testbucket/dir1 ofs://localhost/s3v/testbucket/newdir1
[root@Linux /root/ozone]% bin/ozone fs -ls ofs://localhost/s3v/testbucket/
Found 1 items
drwxrwxrwx   - root root          0 2022-09-19 14:39 ofs://localhost/s3v/testbucket/newdir1
[root@Linux /root/ozone]% bin/ozone fs -mkdir ofs://localhost/s3v/testbucket/newdir1/dir2/dir3
[root@Linux /root/ozone]% bin/ozone fs -ls ofs://localhost/s3v/testbucket/newdir1/dir2/
Found 2 items
drwxrwxrwx   - root root          0 2022-09-19 14:43 ofs://localhost/s3v/testbucket/newdir1/dir2/dir3
-rw-rw-rw-   3 root root    1048576 2022-09-19 14:39 ofs://localhost/s3v/testbucket/newdir1/dir2/key1

[kaijchen]

100x ITestOzoneContractMkdir: https://github.com/kaijchen/ozone/actions/runs/3609358975

[kaijchen]

LGTM, the regression in #3774 has been fixed.
@sadanand48 Do you want to take another look?

[kaijchen]

Thanks @xichen01 for the update.

[kaijchen]

Looks like TestObjectStoreWithLegacyFS.testFlatKeyStructureWithOBS is flaky:
https://github.com/apache/ozone/actions/runs/3612387380/jobs/6087464989#step:5:3182

[kaijchen]

100x TestObjectStoreWithLegacyFS :
https://github.com/kaijchen/ozone/actions/runs/3613121247

@xichen01, sometimes after a rename, the old keyName is still in table:

2022-12-04 20:13:06,066 [Time-limited test] WARN  om.TestObjectStoreWithLegacyFS (TestObjectStoreWithLegacyFS.java:assertKeyCount(160)) - keyName: /dnhdqlmxyt/iskcvwgnho/dir1/dir2/dir3/key-1, expected prefix: /dnhdqlmxyt/iskcvwgnho/dir, suffix: dir1/dir2/dir3/key-1
// renaming "dir1/dir2/dir3/key-1" to "dir1/NewKey-1"
2022-12-04 20:13:06,082 [Time-limited test] WARN  om.TestObjectStoreWithLegacyFS (TestObjectStoreWithLegacyFS.java:assertKeyCount(160)) - keyName: /dnhdqlmxyt/iskcvwgnho/dir1/dir2/dir3/key-1, expected prefix: /dnhdqlmxyt/iskcvwgnho/dir, suffix: dir1/NewKey-1

[kaijchen]

Seems this issue exists before this PR, but with a smaller chance (1/300 vs 2/100).
https://github.com/kaijchen/ozone/actions/runs/3613290473/jobs/6089472749#step:5:3351

The problem is probably due to write cache then read db without cache.
I have created HDDS-7588 to track it.