Applying retention to an ObjectBucketClaim for Loki #75

computate · 2023-01-13T17:49:22Z

I have learned from the RH Internal forum-noobaa that applying retention to an ObjectBucketClaim for Loki involves running a noobaa CLI tool as a cluster admin:
https://github.com/noobaa/noobaa.github.io/blob/master/noobaa-operator-cli.md
There are ways to filter and set expiration days on bucket entries. I could use your help here please, because I don't seem to have access to query and apply these rules.
noobaa Bucket API
noobaa common API

First try getting the status of noobaa in openshift-storage:

noobaa status -n openshift-storage

noobaa -n openshift-storage api bucket_api get_bucket_lifecycle_configuration_rules '{"name": "openshift-operators-redhat-376b03f8-7d3e-4b64-b98a-12c422944f5b"}'

Set bucket lifecycle

noobaa -n openshift-storage api bucket_api set_bucket_lifecycle_configuration_rules '{ 
  "name": "openshift-operators-redhat-376b03f8-7d3e-4b64-b98a-12c422944f5b"
  , "rules": [
    { 
      "id": "123", 
      "status": "Enabled", 
      "filter": {},
      "expiration": {"days": 90}
    }
  ] 
}'

jtriley says we are using NooBaa Operator Version: 5.10.6

The text was updated successfully, but these errors were encountered:

jtriley · 2023-01-19T16:27:42Z

I was able to set a bucket lifecycle configuration by adding a (bogus) prefix:

$ noobaa -n openshift-storage api bucket_api set_bucket_lifecycle_configuration_rules '{
  "name": "openshift-operators-redhat-376b03f8-7d3e-4b64-b98a-12c422944f5b",
  "rules": [
    {
      "id": "123",
      "status": "Enabled",
      "prefix": "tmp/autoexpire",
      "expiration": {"days": 90}
    }
  ]
}'

INFO[0000] ✅ Exists: NooBaa "noobaa"
INFO[0000] ✅ Exists: Service "noobaa-mgmt"
INFO[0000] ✅ Exists: Secret "noobaa-operator"
INFO[0000] ✅ Exists: Secret "noobaa-admin"
INFO[0001] ✈️  RPC: bucket.set_bucket_lifecycle_configuration_rules() Request: map[name:openshift-operators-redhat-376b03f8-7d3e-4b64-b98a-12c422944f5b rules:[map[expiration:map[days:120] id:123 prefix:tmp/autoexpire status:Enabled]]]
WARN[0001] RPC: GetConnection creating connection to wss://localhost:56115/rpc/ 0x14000d9f440
INFO[0001] RPC: Connecting websocket (0x14000d9f440) &{RPC:0x14000481c70 Address:wss://localhost:56115/rpc/ State:init WS:<nil> PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s cancelPings:<nil>}
INFO[0001] RPC: Connected websocket (0x14000d9f440) &{RPC:0x14000481c70 Address:wss://localhost:56115/rpc/ State:init WS:<nil> PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s cancelPings:<nil>}
INFO[0001] ✅ RPC: bucket.set_bucket_lifecycle_configuration_rules() Response OK: took 58.6ms
null

However, still stuck when trying to GET that same lifecycle configuration:

$ noobaa -n openshift-storage api bucket_api get_bucket_lifecycle_configuration_rules '{
  "name": "openshift-operators-redhat-376b03f8-7d3e-4b64-b98a-12c422944f5b"
}'
INFO[0000] ✅ Exists: NooBaa "noobaa"
INFO[0000] ✅ Exists: Service "noobaa-mgmt"
INFO[0000] ✅ Exists: Secret "noobaa-operator"
INFO[0000] ✅ Exists: Secret "noobaa-admin"
FATA[0001] ❌ invalid character '}' looking for beginning of object key string
jtriley in ~/repos/nerc
11:21:29 ❯ vim /tmp/nooba-test.sh
jtriley in ~/repos/nerc took 12s
11:21:44 ➜ bash /tmp/nooba-test.sh
INFO[0000] ✅ Exists: NooBaa "noobaa"
INFO[0000] ✅ Exists: Service "noobaa-mgmt"
INFO[0000] ✅ Exists: Secret "noobaa-operator"
INFO[0000] ✅ Exists: Secret "noobaa-admin"
INFO[0001] ✈️  RPC: bucket.get_bucket_lifecycle_configuration_rules() Request: map[name:openshift-operators-redhat-376b03f8-7d3e-4b64-b98a-12c422944f5b]
WARN[0001] RPC: GetConnection creating connection to wss://localhost:56132/rpc/ 0x14000c8d2c0
INFO[0001] RPC: Connecting websocket (0x14000c8d2c0) &{RPC:0x14000146370 Address:wss://localhost:56132/rpc/ State:init WS:<nil> PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s cancelPings:<nil>}
INFO[0001] RPC: Connected websocket (0x14000c8d2c0) &{RPC:0x14000146370 Address:wss://localhost:56132/rpc/ State:init WS:<nil> PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s cancelPings:<nil>}
ERRO[0001] ⚠️  RPC: bucket.get_bucket_lifecycle_configuration_rules() Call failed: json: cannot unmarshal array into Go struct field .reply of type map[string]interface {}
FATA[0001] ❌ json: cannot unmarshal array into Go struct field .reply of type map[string]interface {}

This looks like a bug in the reply-handling portion of the code somewhere...

jtriley · 2023-01-19T16:48:51Z

We were able to get further using the AWS CLI instead via a debug pod running in the openshift-operators-redhat namespace:

$ oc ns openshift-operators-redhat
$ oc --as=system:admin debug --as-root=true
$ dnf clean all
$ dnf install python3-pip python3-virtualenv
$ virtualenv awscli
$ source awscli/bin/activate
(awscli) $ pip install awscli
(awscli) $ aws configure
# enter aws access key and secret access key from object bucket claim secret
(awscli) $ aws --no-verify-ssl --endpoint-url=https://s3.openshift-storage.svc s3api get-bucket-lifecycle-configuration --bucket openshift-operators-redhat-376b03f8-7d3e-4b64-b98a-12c422944f5b
/root/awscli/lib/python3.6/site-packages/urllib3/connectionpool.py:1052: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3.openshift-storage.svc'.
 Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
{
    "Rules": [
        {
            "Expiration": {
                "Days": 120,
                "ExpiredObjectDeleteMarker": false
            },
            "ID": "123",
            "Prefix": "tmp/autoexpire",
            "Status": "Enabled"
        }
    ]
}

Use these commands inside the openshift-operators-redhat namespace to get the bucket host, bucket name, access key id, and secret access key for the aws configure step and --endpoint-url value above:

$ oc get configmap -o yaml openshift-operators-redhat-objectbucketclaim -o 'jsonpath={.data.BUCKET_HOST}'
$ oc get configmap -o yaml openshift-operators-redhat-objectbucketclaim -o 'jsonpath={.data.BUCKET_NAME}'
$ oc get secrets openshift-operators-redhat-objectbucketclaim -o jsonpath='{.data.AWS_ACCESS_KEY_ID}'  | base64 -d
$ oc get secrets openshift-operators-redhat-objectbucketclaim -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}'  | base64 -d

jtriley · 2023-01-19T17:00:59Z

Just confirming (and documenting the commands used) that we can both put and get noobaa bucket lifecycle configs via awscli. Here we're changing the expiration days from 123 to 180:

+ aws --no-verify-ssl --endpoint-url=https://s3.openshift-storage.svc s3api get-bucket-lifecycle-configuration --bucket openshift-operators-redhat-376b03f8-7d3e-4b64-b98a-12c422944f5b
/awscli/lib/python3.6/site-packages/urllib3/connectionpool.py:1052: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3.openshift-storage.svc'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
{
    "Rules": [
        {
            "Expiration": {
                "Days": 123,
                "ExpiredObjectDeleteMarker": false
            },
            "ID": "123",
            "Prefix": "tmp/autoexpire",
            "Status": "Enabled"
        }
    ]
}
+ aws --no-verify-ssl --endpoint-url=https://s3.openshift-storage.svc s3api put-bucket-lifecycle-configuration --bucket openshift-operators-redhat-376b03f8-7d3e-4b64-b98a-12c422944f5b --lifecycle-configuration '{
  "Rules": [
    {
      "ID": "123",
      "Status": "Enabled",
      "Prefix": "tmp/autoexpire",
      "Expiration": {"Days": 180}
    }
  ]
}'
/awscli/lib/python3.6/site-packages/urllib3/connectionpool.py:1052: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3.openshift-storage.svc'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
+ aws --no-verify-ssl --endpoint-url=https://s3.openshift-storage.svc s3api get-bucket-lifecycle-configuration --bucket openshift-operators-redhat-376b03f8-7d3e-4b64-b98a-12c422944f5b
/awscli/lib/python3.6/site-packages/urllib3/connectionpool.py:1052: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3.openshift-storage.svc'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
{
    "Rules": [
        {
            "Expiration": {
                "Days": 180,
                "ExpiredObjectDeleteMarker": false
            },
            "ID": "123",
            "Prefix": "tmp/autoexpire",
            "Status": "Enabled"
        }
    ]
}

computate · 2023-01-26T20:15:10Z

For the aws v2 api see: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
dnf install -y unzip
unzip awscliv2.zip
aws/install
pip uninstall awscli

computate · 2023-01-26T20:15:35Z

To add certificate verification:

export AWS_CA_BUNDLE=/run/secrets/kubernetes.io/serviceaccount/service-ca.crt

computate · 2023-01-27T19:27:15Z

Use these commands inside the openshift-operators-redhat namespace to get the bucket host, bucket name, access key id, and secret access key for the aws configure step and --endpoint-url value above:

$ oc get configmap -o yaml openshift-operators-redhat-objectbucketclaim -o 'jsonpath={.data.BUCKET_HOST}'
$ oc get configmap -o yaml openshift-operators-redhat-objectbucketclaim -o 'jsonpath={.data.BUCKET_NAME}'
$ oc get secrets openshift-operators-redhat-objectbucketclaim -o jsonpath='{.data.AWS_ACCESS_KEY_ID}'  | base64 -d
$ oc get secrets openshift-operators-redhat-objectbucketclaim -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}'  | base64 -d

Create a Loki debug pod and configure the aws CLI to point to the S3 bucket:

$ oc project openshift-operators-redhat
$ oc --as=system:admin debug --as-root=true
$ dnf clean all
$ dnf install python3-pip python3-virtualenv
$ virtualenv awscli
$ source awscli/bin/activate
(awscli) $ pip install awscli
(awscli) $ aws configure
# enter aws access key and secret access key from object bucket claim secret

Now you can create the bucket retention:

aws --endpoint-url=https://s3.openshift-storage.svc s3api put-bucket-lifecycle-configuration --bucket openshift-operators-redhat-41733583-e42b-4b42-8181-550b09249988 --lifecycle-configuration '{
  "Rules": [
    {
      "ID": "loki-retention",
      "Status": "Enabled",
      "Prefix": "",
      "Expiration": {"Days": 90}
    }
  ]
}'

Check that the bucket retention has been applied:

(awscli) sh-4.4# aws --endpoint-url=https://s3.openshift-storage.svc s3api get-bucket-lifecycle-configuration --bucket openshift-operators-redhat-41733583-e42b-4b42-8181-550b09249988                             
{
    "Rules": [
        {
            "Expiration": {
                "Days": 90,
                "ExpiredObjectDeleteMarker": false
            },
            "ID": "loki-retention",
            "Prefix": "",
            "Status": "Enabled"
        }
    ]
}

joachimweyl · 2023-02-16T14:24:38Z

@computate what are the next steps for this issue?

computate · 2023-02-16T15:07:19Z

@joachimweyl This bucket retention was successfully applied earlier when the logs were in the openshift-operators-redhat namespace. I am in the process of moving logs to the recommended openshift-logging namespace from the openshift-operators-redhat to get the Grafana integration with the logs working again after working with the Red Hat engineers on the upgrade of the LokiOperator. The problem is the NooBaa Object Bucket Storage stopped working. I will reapply the bucket retention in the new namespace when I am able to create an ObjectBucketClaim again. I'm working with @larsks on the error creating NooBaa ObjectBucketClaims.

joachimweyl · 2023-02-16T15:20:08Z

@larsks do we have a GH Issue for the NooBaa ObjectBucketClaim issue? I found this old one, should we just reopen that?

joachimweyl · 2023-02-28T14:02:34Z

@computate in yesterday's meeting it sounded like the noobaa issue was resolved. Can this issue be closed?

joachimweyl · 2023-03-07T13:52:32Z

Closed by OCP-on-NERC/nerc-ocp-config#201

computate self-assigned this Jan 23, 2023

computate mentioned this issue Feb 1, 2023

Size NESE allocation correctly for nerc-ocp-infra cluster #38

Closed

computate mentioned this issue Mar 6, 2023

Re-enable cluster logging from prod to infra cluster Loki OCP-on-NERC/nerc-ocp-config#201

Merged

joachimweyl closed this as completed Mar 7, 2023

joachimweyl added the openshift This issue pertains to NERC OpenShift label Aug 16, 2023

computate mentioned this issue Nov 2, 2023

Determine the right backup strategy for logs #63

Open

schwesig mentioned this issue Mar 25, 2024

Explore & Optimize: Ensuring Effective Log Storage Management for the Observability Cluster #505

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Applying retention to an ObjectBucketClaim for Loki #75

Applying retention to an ObjectBucketClaim for Loki #75

computate commented Jan 13, 2023 •

edited

Loading

jtriley commented Jan 19, 2023

jtriley commented Jan 19, 2023 •

edited

Loading

jtriley commented Jan 19, 2023

computate commented Jan 26, 2023

computate commented Jan 26, 2023

computate commented Jan 27, 2023 •

edited

Loading

joachimweyl commented Feb 16, 2023

computate commented Feb 16, 2023

joachimweyl commented Feb 16, 2023 •

edited

Loading

joachimweyl commented Feb 28, 2023

joachimweyl commented Mar 7, 2023

Applying retention to an ObjectBucketClaim for Loki #75

Applying retention to an ObjectBucketClaim for Loki #75

Comments

computate commented Jan 13, 2023 • edited Loading

jtriley commented Jan 19, 2023

jtriley commented Jan 19, 2023 • edited Loading

jtriley commented Jan 19, 2023

computate commented Jan 26, 2023

computate commented Jan 26, 2023

computate commented Jan 27, 2023 • edited Loading

joachimweyl commented Feb 16, 2023

computate commented Feb 16, 2023

joachimweyl commented Feb 16, 2023 • edited Loading

joachimweyl commented Feb 28, 2023

joachimweyl commented Mar 7, 2023

computate commented Jan 13, 2023 •

edited

Loading

jtriley commented Jan 19, 2023 •

edited

Loading

computate commented Jan 27, 2023 •

edited

Loading

joachimweyl commented Feb 16, 2023 •

edited

Loading