Account update with account name is not updating bucket_name.json file automatically #8080

anandhu-karattu · 2024-05-27T09:25:18Z

###Environment info

NooBaa Version: noobaa-core-5.15.3-20240514.el9.x86_64
Platform: Standalone Noobaa

Actual behavior

1.Account update with account name is not updating bucket_name.json file automatically.
I/O is failing due to this issue.

Expected behavior

If the account name is updated with a new name, the corresponding buckets should get auto updated with the new account name.
Otherwise it is a tedious task to manually update all the buckets with owner details

Steps to reproduce

Create an account with noobaa-cli

[root@3node-x-worker1 ~]# noobaa-cli account add --name account_test_old --uid 7000 --gid 7000 --new_buckets_path /mnt/fs1/account_test_old 2>/dev/null
{
  "response": {
    "code": "AccountCreated",
    "reply": {
      "_id": "66544e9f738d7528d7f9e70b",
      "name": "account_test_old",
      "email": "account_test_old",
      "creation_date": "2024-05-27T09:13:03.211Z",
      "access_keys": [
        {
          "access_key": "BjLLk92yXAEFxOKaOFyA",
          "secret_key": "xYPDVxbTaH6g8b42T2ecOk/j3I53spnA/JkDtEi2"
        }
      ],
      "nsfs_account_config": {
        "uid": 7000,
        "gid": 7000,
        "new_buckets_path": "/mnt/fs1/account_test_old",
        "fs_backend": "GPFS"
      },
      "allow_bucket_creation": true,
      "master_key_id": "664e107282dba0d07d51b455"
    }
  }
}

Create a bucket with noobaa-cli.

[root@3node-x-worker1 ~]# noobaa-cli bucket add --name bucket-old --owner account_test_old --path /mnt/fs1/account_test_old/bucket_old 2>/dev/null
{
  "response": {
    "code": "BucketCreated",
    "reply": {
      "_id": "66544f33b1260396610d32cc",
      "name": "bucket-old",
      "owner_account": "66544e9f738d7528d7f9e70b",
      "system_owner": "account_test_old",
      "bucket_owner": "account_test_old",
      "versioning": "DISABLED",
      "creation_date": "2024-05-27T09:15:31.368Z",
      "path": "/mnt/fs1/account_test_old/bucket_old",
      "should_create_underlying_storage": false,
      "fs_backend": "GPFS"
    }
  }
}

Update the account name to a new name : PASS

[root@3node-x-worker1 ~]# noobaa-cli account update --name account_test_old --new_name account_test_new
load_nsfs_nc_config.setting config.NSFS_NC_CONF_DIR /gpfs/fs-ces-shared/ces/s3-config
nsfs: config_dir_path=/gpfs/fs-ces-shared/ces/s3-config config.json= {
  ENDPOINT_FORKS: 2,
  ENDPOINT_PORT: 6001,
  ENDPOINT_SSL_PORT: 6443,
  ALLOW_HTTP: true,
  UV_THREADPOOL_SIZE: 16,
  GPFS_DL_PATH: '/usr/lpp/mmfs/lib/libgpfs.so',
  NOOBAA_LOG_LEVEL: 'default',
  NSFS_NC_STORAGE_BACKEND: 'GPFS',
  NSFS_NC_CONFIG_DIR_BACKEND: 'GPFS',
  NSFS_DIR_CACHE_MAX_DIR_SIZE: 536870912,
  NSFS_DIR_CACHE_MAX_TOTAL_SIZE: 1073741824,
  NC_MASTER_KEYS_GET_EXECUTABLE: '/usr/lpp/mmfs/bin/cess3_key_get',
  NC_MASTER_KEYS_PUT_EXECUTABLE: '/usr/lpp/mmfs/bin/cess3_key_put',
  NC_MASTER_KEYS_STORE_TYPE: 'executable'
}
2024-05-27 02:16:52.852822 [PID-1489615/TID-1489615] FS::GPFS GPFS_DL_PATH=/usr/lpp/mmfs/lib/libgpfs.so
2024-05-27 02:16:52.853560 [PID-1489615/TID-1489615] FS::GPFS found GPFS lib file GPFS_DL_PATH=/usr/lpp/mmfs/lib/libgpfs.so
May-27 2:16:53.876 [/1489615]   [LOG] CONSOLE:: detect_fips_mode: found /proc/sys/crypto/fips_enabled with value 0
(node:1489615) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.

Please migrate your code to use AWS SDK for JavaScript (v3).
For more information, check the migration guide at https://a.co/7PzMCcy
(Use `node --trace-warnings ...` to show where the warning was created)
May-27 2:16:53.976 [/1489615]   [LOG] CONSOLE:: read_rand_seed: reading 32 bytes from /dev/urandom ...
May-27 2:16:53.993 [/1489615]   [LOG] CONSOLE:: read_rand_seed: got 32 bytes from /dev/urandom, total 32 ...
May-27 2:16:53.994 [/1489615]   [LOG] CONSOLE:: read_rand_seed: closing fd ...
May-27 2:16:53.995 [/1489615]   [LOG] CONSOLE:: init_rand_seed: seeding with 32 bytes
May-27 2:16:53.996 [/1489615]   [LOG] CONSOLE:: generate_entropy: entropy_avail 256
May-27 2:16:54.041 [/1489615]   [LOG] CONSOLE:: generate_entropy: adding entropy: dd if=/dev/vda bs=1048576 count=32 skip=215443 | md5sum
May-27 2:16:54.199 [/1489615]   [LOG] CONSOLE:: init_rand_seed: done
May-27 2:16:54.217 [/1489615]    [L0] core.manage_nsfs.nc_master_key_manager:: init_from_exec: get master keys response status=OK, version=1
{
  "response": {
    "code": "AccountUpdated",
    "reply": {
      "_id": "66544e9f738d7528d7f9e70b",
      "name": "account_test_new",
      "email": "account_test_new",
      "creation_date": "2024-05-27T09:13:03.211Z",
      "access_keys": [
        {
          "access_key": "BjLLk92yXAEFxOKaOFyA",
          "secret_key": "xYPDVxbTaH6g8b42T2ecOk/j3I53spnA/JkDtEi2"
        }
      ],
      "nsfs_account_config": {
        "uid": 7000,
        "gid": 7000,
        "new_buckets_path": "/mnt/fs1/account_test_old",
        "fs_backend": "GPFS"
      },
      "allow_bucket_creation": true,
      "master_key_id": "664e107282dba0d07d51b455"
    }
  }
}

Now verify the ownership of the bucket. It is still showing old account name

[root@3node-x-worker1 ~]# jq "." /gpfs/fs-ces-shared/ces/s3-config/buckets/bucket-old.json
{
  "_id": "66544f33b1260396610d32cc",
  "name": "bucket-old",
  "owner_account": "66544e9f738d7528d7f9e70b",
  "system_owner": "**account_test_old**",
  "bucket_owner": "**account_test_old**",
  "versioning": "DISABLED",
  "creation_date": "2024-05-27T09:15:31.368Z",
  "path": "/mnt/fs1/account_test_old/bucket_old",
  "should_create_underlying_storage": false,
  "fs_backend": "GPFS"
}

More information - Screenshots / Logs / Other output

The text was updated successfully, but these errors were encountered:

romayalon · 2024-05-27T10:53:23Z

@anandhu-karattu
Can you share the I/O failure you see?

anandhu-karattu · 2024-05-27T16:23:24Z

I am trying to upload an object in to the bucket "bucket-old", it is failing

[root@anan-rhel921 ~]# AWS_ACCESS_KEY_ID=BjLLk92yXAEFxOKaOFyA AWS_SECRET_ACCESS_KEY=xYPDVxbTaH6g8b42T2ecOk/j3I53spnA/JkDtEi2 aws --endpoint https://10.11.71.87:6443 --no-verify-ssl s3 cp awscliv2.zip s3://bucket-old
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.11.71.87'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
upload failed: ./awscliv2.zip to s3://bucket-old/awscliv2.zip An error occurred (AccessDenied) when calling the CreateMultipartUpload operation: Access Denied

Issue is only with the existing buckets , since the owner name is not updated to new_name.

shirady · 2024-06-03T08:45:15Z

Hi @anandhu-karattu,
I added a fix for this issue, please pay attention that it is a short-term fix:

After this fix, the S3 request should not return an error.
But the buckets that the account owns still have the properties system_owner and bucket_owner with the previous name (in the bucket config) <- this would be handled in a long-term fix

I'm adding the label "Request Validation" so you can test it (and not close it for now).

cc: @romayalon

anandhu-karattu · 2024-06-04T06:51:17Z

After this fix, the S3 request should not return an error.

What does this mean? @shirady

shirady · 2024-06-04T06:54:48Z

@anandhu-karattu I meant for the error Access Denied.

romayalon · 2024-06-04T11:00:42Z

@anandhu-karattu @shirady Shira fixed the IO failure but not the actual update of the bucket config.json, a fix for the update will be taken care in #7734, Shira please keep me honest here.

shirady · 2024-06-04T11:29:35Z

@romayalon I agree with you.
just for it to be clear the long-term solution would be to change the config structure and map the account and bucket IDs to the config file, and use only these IDs as a reference.
I didn't close the issue because the fix is partial.

anandhu-karattu · 2024-06-07T07:11:33Z

Verified on build "noobaa-core-5.15.4-20240605.el9.x86_64"
Looks like the IO FAILURE is still there.

How to reproduce:

Create an account > Bucket > Upload object in to bucket ==> PASS
Change the account name to new name ==> PASS

[root@anan-21 ~]# noobaa-cli account update --name account-65675 --new_name account_test_new
{
  "response": {
    "code": "AccountUpdated",
    "reply": {
      "_id": "6662ad6dc999b40f7e95e9bf",
      "name": "account_test_new",
      "email": "account_test_new",
      "creation_date": "2024-06-07T06:49:17.510Z",
      "access_keys": [
        {
          "access_key": "5ZQ5YaCYDOGfZhDC07sM",
          "secret_key": "JpEojbabc57kMfNk/rGBPUW0bW/jTHWXwqhLfIJs"
        }
      ],
      "nsfs_account_config": {
        "uid": 3844,
        "gid": 4890,
        "new_buckets_path": "/mnt/gpfs0/account_65675/",
        "fs_backend": "GPFS"
      },
      "allow_bucket_creation": true,
      "master_key_id": "6662ad6dadbc37d3ecb52d68"
    }
  }
}

Upload the object again to the same bucket ==> FAILED with access denied error

[root@akarattuparambil-scale-host ~]# AWS_ACCESS_KEY_ID=5ZQ5YaCYDOGfZhDC07sM AWS_SECRET_ACCESS_KEY=JpEojbabc57kMfNk/rGBPUW0bW/jTHWXwqhLfIJs aws --endpoint https://10.0.100.21:6443 --no-verify-ssl s3 cp awscliv2.zip s3://bucket-65675
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
upload failed: ./awscliv2.zip to s3://bucket-65675/awscliv2.zip An error occurred (AccessDenied) when calling the UploadPart operation: Access Denied

anandhu-karattu · 2024-06-07T07:13:01Z

@shirady looks like the fix is failing for me. Can you please check the issue again?

anandhu-karattu · 2024-06-07T12:46:44Z

This is the corresponding error found in noobaa.log

Jun  7 14:45:36 anan-21 [230684]: [nsfs/230684]    [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket_config_path /mnt/cesSharedRoot/ces/s3-config/buckets/bucket-65675.json
Jun  7 14:45:36 anan-21 [230684]: [nsfs/230684]    [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket_config_path /mnt/cesSharedRoot/ces/s3-config/buckets/bucket-65675.json
Jun  7 14:45:36 anan-21 [230684]: [nsfs/230684] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><Message>Access Denied</Message><Resource>/bucket-65675/awscliv2.zip?uploads</Resource><RequestId>lx4oj92t-f3f5kt-oxw</RequestId></Error> POST /bucket-65675/awscliv2.zip?uploads {"host":"10.0.100.21:6443","accept-encoding":"identity","content-type":"application/zip","user-agent":"aws-cli/2.15.30 Python/3.11.8 Linux/4.18.0-477.10.1.el8_8.x86_64 exe/x86_64.rhel.8 prompt/off command/s3.cp","x-amz-date":"20240607T124536Z","x-amz-content-sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","authorization":"AWS4-HMAC-SHA256 Credential=5ZQ5YaCYDOGfZhDC07sM/20240607/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date, Signature=5476b626e3c290390c773896cf9c2276ba88c9eead4bccfcc293d4f8fc8b4b9b","content-length":"0"} Error: Access Denied    at authorize_request_policy (/usr/local/noobaa-core/src/endpoint/s3/s3_rest.js:243:15)    at async Promise.all (index 1)    at async authorize_request (/usr/local/noobaa-core/src/endpoint/s3/s3_rest.js:205:5)    at async handle_request (/usr/local/noobaa-core/src/endpoint/s3/s3_rest.js:115:5)    at async Object.s3_rest [as handler] (/usr/local/noobaa-core/src/endpoint/s3/s3_rest.js:65:9)

romayalon · 2024-06-09T04:35:59Z

@anandhu-karattu we didn't backport it to stage_5.15.4...
@madhuthorat please evaluate this priority and let us know if this should be backported..
cc: @nimrod-becker

anandhu-karattu · 2024-06-10T03:57:49Z

@romayalon we are not supporting account name update in scale release 5.2.1.
So this fix is not mandatory for us in 5.15.4 @madhuthorat
But we may support this in future releases, so we will expect a future fix (complete fix as per your plan)

madhuthorat · 2024-06-10T05:44:22Z

@romayalon we are not supporting account name update in scale release 5.2.1. So this fix is not mandatory for us in 5.15.4 @madhuthorat But we may support this in future releases, so we will expect a future fix (complete fix as per your plan)

Right, we won't support accountName update in CES S3 MVP GA, so please take a call if you want to add to 5.15.4 or not.

romayalon · 2024-06-16T07:53:50Z

@madhuthorat @anandhu-karattu We decided not to backport it to 5.15.4,
@anandhu-karattu could you validate it on master?

anandhu-karattu · 2024-06-18T09:49:42Z

Verified on build "noobaa-core-5.17.0-20240617.el9.x86_64"
There is no access denied error this time. (verification steps are mentioned above)
Marking as verified. @romayalon @shirady

shirady · 2024-06-25T09:41:32Z

Hi,
The short-term solution was verified (see comment above):

Removing the "request validation" label as it was validated.
I will remove myself from the assignee since another developer might be assigned for the fix of the long-term solution

cc: @romayalon

romayalon · 2024-08-22T12:16:07Z

@anandhu-karattu @madhuthorat we removed the bucket owner property on master, adding request validation lable, can you please verify?

anandhu-karattu added the NS-FS label May 27, 2024

shirady self-assigned this May 28, 2024

shirady mentioned this issue May 28, 2024

NSFS | NC | Add Condition in authorize_request_policy #8086

Merged

2 tasks

shirady added Request Validation Non Containerized Non containerized labels Jun 3, 2024

romayalon added this to the 5.15.5 milestone Jun 4, 2024

shirady removed the Request Validation label Jun 25, 2024

shirady removed their assignment Jun 25, 2024

romayalon assigned alphaprinz Jun 25, 2024

romayalon mentioned this issue Jul 7, 2024

Nc account by id #8167

Closed

2 tasks

romayalon mentioned this issue Aug 21, 2024

NC | Bucket Owner Removal #8289

Merged

2 tasks

romayalon closed this as completed Aug 22, 2024

romayalon added the Request Validation label Aug 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Account update with account name is not updating bucket_name.json file automatically #8080

Account update with account name is not updating bucket_name.json file automatically #8080

anandhu-karattu commented May 27, 2024

romayalon commented May 27, 2024

anandhu-karattu commented May 27, 2024

shirady commented Jun 3, 2024 •

edited

Loading

anandhu-karattu commented Jun 4, 2024 •

edited

Loading

shirady commented Jun 4, 2024

romayalon commented Jun 4, 2024 •

edited

Loading

shirady commented Jun 4, 2024

anandhu-karattu commented Jun 7, 2024

anandhu-karattu commented Jun 7, 2024

anandhu-karattu commented Jun 7, 2024

romayalon commented Jun 9, 2024

anandhu-karattu commented Jun 10, 2024

madhuthorat commented Jun 10, 2024

romayalon commented Jun 16, 2024

anandhu-karattu commented Jun 18, 2024

shirady commented Jun 25, 2024

romayalon commented Aug 22, 2024

Account update with account name is not updating bucket_name.json file automatically #8080

Account update with account name is not updating bucket_name.json file automatically #8080

Comments

anandhu-karattu commented May 27, 2024

Actual behavior

Expected behavior

Steps to reproduce

More information - Screenshots / Logs / Other output

romayalon commented May 27, 2024

anandhu-karattu commented May 27, 2024

shirady commented Jun 3, 2024 • edited Loading

anandhu-karattu commented Jun 4, 2024 • edited Loading

shirady commented Jun 4, 2024

romayalon commented Jun 4, 2024 • edited Loading

shirady commented Jun 4, 2024

anandhu-karattu commented Jun 7, 2024

anandhu-karattu commented Jun 7, 2024

anandhu-karattu commented Jun 7, 2024

romayalon commented Jun 9, 2024

anandhu-karattu commented Jun 10, 2024

madhuthorat commented Jun 10, 2024

romayalon commented Jun 16, 2024

anandhu-karattu commented Jun 18, 2024

shirady commented Jun 25, 2024

romayalon commented Aug 22, 2024

shirady commented Jun 3, 2024 •

edited

Loading

anandhu-karattu commented Jun 4, 2024 •

edited

Loading

romayalon commented Jun 4, 2024 •

edited

Loading