NSFS | NC | Handle Concurrency When Reading Entries and They Deleted by Another Process

[shirady]

Explain the changes

1. Fix the concurrency issue on the CLI delete account (when a bucket was deleted during account deletion)
2. Fix the concurrency issue on the CLI list:
   • CLI list buckets (when a bucket was deleted during the list)
   • CLI list accounts (when an account was deleted during the list)
3. Fix the concurrency issue on S3 API list buckets (when a bucket was deleted during the list)
4. Add TODO in accountspace_fs.
5. Add test (not related to the issue, was added during debug)

Issues: Fixed #7920

1. GAP - accountspace_fs list (IAM ListUsers API) should also be fixed.

Testing Instructions:

Manual Tests:

1. To see the failure and the changes with a manual test we will simulate 2 processes: the first process executes the operation that reads the directory content and we add sleep after it in the code (for more information see comment below), and the second process deletes one of the entries.
2. First, create the FS_ROOT and a directory for a bucket: mkdir -p /tmp/nsfs_root1/my-bucket and give permissions chmod 777 /tmp/nsfs_root1/ chmod 777 /tmp/nsfs_root1/my-bucket.
   This will be the argument for:

• new_buckets_path flag /tmp/nsfs_root1 (that we will use in the account commands)
• path in the buckets commands /tmp/nsfs_root1/my-bucket (that we will use in bucket commands).

3. Create an account: sudo node src/cmd/manage_nsfs account add --name <account-name> --uid <uid> --gid <gid> --new_buckets_path <new_buckets_path>
4. Create 3 buckets: sudo node src/cmd/manage_nsfs bucket add --name <bucket-name> --owner <account-name> --path <path> (use the --owner from the previous account add command, for example: bucket-1, bucket-2, bucket-3).

For example - when deleting an account we check if the account has buckets (hence we list them):
5. (process 1 - open a tab in the terminal) Delete account: sudo node src/cmd/manage_nsfs account delete --name <account-name> --debug 5 (we use the debug flag to see printing related to the filesystem, we expect to see on the deleted config file an error message "No such file or directory")
6. (process 2 - open a tab in the terminal) Delete bucket (bucket-2 for example): sudo node src/cmd/manage_nsfs bucket delete --name <bucket-name>

• Doc added/updated
• Tests added

[shirady]

Additional Testing Notes:

The code part you will add after reading the directory and getting the entries is sleep, for example:
After this line: const entries = await nb_native().fs.readdir(fs_context, <path>);

        console.log('SDSD entries', entries);
        await set_sleep(30000);

async function set_sleep(ms) {
    console.log('SDSD sleep', ms);
    await sleep(ms);
}

  function sleep(ms) {
    return new Promise(resolve => {
      setTimeout(resolve, ms);
    });
}

Open a terminal tab for each process:

1. List buckets (CLI) and delete a bucket:
   Requirement: 1 account (shira-1001) and 3 buckets (bucket-1001-01, bucket-1001-02, bucket-1001-03), delete one of them (bucket-1001-02):
   process 1 - sudo node src/cmd/manage_nsfs bucket list --wide --debug 5 (with sleep)
   process 2 - sudo node src/cmd/manage_nsfs bucket delete --name bucket-1001-02

2. Delete account (CLI) and delete its bucket:
   Requirement: 1 account (shira-1001) that has 1 bucket (bucket-1001-01), delete this bucket:
   process 1 - sudo node src/cmd/manage_nsfs account delete --name shira-1001 --debug 5 (with sleep)
   process 2 - sudo node src/cmd/manage_nsfs bucket delete --name bucket-1001-01

3. List buckets (S3 API) and delete a bucket:
   Requirement:

• Running the nsfs server sudo node src/cmd/nsfs --debug 5
• 1 account (shira-1001) and 3 buckets (bucket-1001-01, bucket-1001-02, bucket-1001-03), delete one of them (bucket-1001-02)
• Create the alias for S3: alias s3-nc-user-1-s3='AWS_ACCESS_KEY_ID=<access-key> AWS_SECRET_ACCESS_KEY=<secret-key> aws --no-verify-ssl --endpoint-url https://localhost:6443
  process 1 - s3-nc-user-1-s3 s3 ls (with sleep)
  process 2 - sudo node src/cmd/manage_nsfs bucket delete --name bucket-1001-02 (bucket deleted)

4. List accounts (CLI) and delete an account:
   Requirement: 3 accounts (shira-1001, shira-1002, shira-1003), delete one of them (shira-1002):
   process 1 - sudo node src/cmd/manage_nsfs account list --wide --debug 5 (with sleep)
   process 2 - sudo node src/cmd/manage_nsfs account delete --name shira-1002