noobaa log is not performing logrotate #8153

rkomandu · 2024-06-18T11:34:07Z

Environment info

NooBaa Version: VERSION
Platform: Kubernetes 1.14.1 | minikube 1.1.1 | OpenShift 4.1 | other: specify

Noobaa staging build of 4.15.4 (noobaa-core-5.15.4-20240611.el9.x86_64)

Actual behavior

When the noobaa.log reached the 105M, it should automatically rotate as per the logrotate setting, it doesn't seem to be the case on the BM (physical machine). @naveenpaul1 and I looked at the system in detail and after running manually (restarting rsyslog) it worked.

All the discussion is in this thread https://ibmandredhatguests.slack.com/archives/C015Z7SDWQ0/p1718358615847589 (Guy and Naveen)

Jun 13 05:00:33 c83f1-app4 rsyslogd[3316031]: file size limit cmd for file '/var/log/noobaa.log' did no resolve situation  [v8.2102.0-117.el9]

 systemctl status rsyslog
● rsyslog.service - System Logging Service
     Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; preset: enabled)
     Active: active (running) since Thu 2024-06-13 04:30:07 EDT; 5 days ago
       Docs: man:rsyslogd(8)
             https://www.rsyslog.com/doc/
   Main PID: 1714498 (rsyslogd)
      Tasks: 3 (limit: 3297529)
     Memory: 90.3M
        CPU: 2min 58.056s
     CGroup: /system.slice/rsyslog.service
             └─1714498 /usr/sbin/rsyslogd -n

Jun 15 19:20:25 c83f1-app3 rsyslogd[1714498]: imjournal: journal files changed, reloading...  [v8.2102.0-117.el9 try https://www.rsyslog.com/e/0 ]
Jun 16 00:00:01 c83f1-app3 systemd[1]: rsyslog.service: Sent signal SIGHUP to main process 1714498 (rsyslogd) on client request.
Jun 16 00:00:11 c83f1-app3 rsyslogd[1714498]: [origin software="rsyslogd" swVersion="8.2102.0-117.el9" x-pid="1714498" x-info="https://www.rsyslog.com"] rsys>
Jun 16 00:00:11 c83f1-app3 rsyslogd[1714498]: file size limit cmd for file '/var/log/noobaa.log' did no resolve situation  [v8.2102.0-117.el9]
Jun 16 04:05:20 c83f1-app3 rsyslogd[1714498]: imjournal: journal files changed, reloading...  [v8.2102.0-117.el9 try https://www.rsyslog.com/e/0 ]
Jun 16 17:17:13 c83f1-app3 rsyslogd[1714498]: imjournal: journal files changed, reloading...  [v8.2102.0-117.el9 try https://www.rsyslog.com/e/0 ]
Jun 16 17:17:14 c83f1-app3 rsyslogd[1714498]: imjournal: journal files changed, reloading...  [v8.2102.0-117.el9 try https://www.rsyslog.com/e/0 ]
Jun 17 06:30:04 c83f1-app3 rsyslogd[1714498]: imjournal: journal files changed, reloading...  [v8.2102.0-117.el9 try https://www.rsyslog.com/e/0 ]
Jun 17 06:30:04 c83f1-app3 rsyslogd[1714498]: imjournal: journal files changed, reloading...  [v8.2102.0-117.el9 try https://www.rsyslog.com/e/0 ]
Jun 17 19:42:11 c83f1-app3 rsyslogd[1714498]: imjournal: journal files changed, reloading...  [v8.2102.0-117.el9 try https://www.rsyslog.com/e/0 ]

after performing this

/usr/sbin/logrotate -v /etc/logrotate.d/noobaa/logrotate_noobaa.conf || true 

# ls -lh /var/log/noobaa*
-rw-r--r--. 1 root root  18M Jun 18 03:27 /var/log/noobaa_events.log
-rw-r--r--. 1 root root  28M Jun 18 05:40 /var/log/noobaa.log
-rw-r--r--. 1 root root 6.1M Jun 14 06:44 /var/log/noobaa.log.1.gz

Where we haven't performed the rsyslog restart, here it is

c83f1-app3 ~]# ls -lh /var/log/noobaa*
-rw-r--r--. 1 root root  17M Jun 17 07:46 /var/log/noobaa_events.log
-rw-r--r--. 1 root root 106M Jun 16 00:00 /var/log/noobaa.log   --> greater than 105M.

Expected behavior

It should automatically logrotate when the file size is reached.

Steps to reproduce

More information - Screenshots / Logs / Other output

Posted above the log file

The text was updated successfully, but these errors were encountered:

naveenpaul1 · 2024-06-18T16:22:50Z

@rkomandu please check the logrotation when it hits the size limit in your node, Last time when you did the upgrade log size was already larger than configured value and I believe logrotate failed because of that. If logrotate successful when noobaa.log size reaches 105M then this issue should be invalid otherwise we need to fix this issue.

rkomandu · 2024-06-19T04:14:57Z

Ran IO workload onto the app2 node serving the IP, logrotate didn't happen

[root@c83f1-app2 ~]# ls -lh /var/log/noobaa*
-rw-r--r--. 1 root root  18M Jun 18 10:14 /var/log/noobaa_events.log
-rw-r--r--. 1 root root 100M Jun 18 23:46 /var/log/noobaa.log
-rw-r--r--. 1 root root 6.1M Jun 14 06:44 /var/log/noobaa.log.1.gz
[root@c83f1-app2 ~]# date
Tue Jun 18 11:46:38 PM EDT 2024


Every 30.0s: ls -lh /var/log/noobaa*                                                                                      c83f1-app2: Wed Jun 19 00:04:49 2024

-rw-r--r--. 1 root root  18M Jun 18 23:46 /var/log/noobaa_events.log
-rw-r--r--. 1 root root 105M Jun 19 00:04 /var/log/noobaa.log
-rw-r--r--. 1 root root 6.1M Jun 14 06:44 /var/log/noobaa.log.1.gz

Every 30.0s: ls -lh /var/log/noobaa*                                                                                      c83f1-app2: Wed Jun 19 00:14:19 2024

-rw-r--r--. 1 root root  18M Jun 18 23:46 /var/log/noobaa_events.log
-rw-r--r--. 1 root root 106M Jun 19 00:13 /var/log/noobaa.log
-rw-r--r--. 1 root root 6.1M Jun 14 06:44 /var/log/noobaa.log.1.gz

rkomandu · 2024-06-19T04:18:46Z

rsyslog on the node


systemctl status rsyslog
● rsyslog.service - System Logging Service
     Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; preset: enabled)
     Active: active (running) since Fri 2024-06-14 06:44:09 EDT; 4 days ago
       Docs: man:rsyslogd(8)
             https://www.rsyslog.com/doc/
   Main PID: 1517556 (rsyslogd)
      Tasks: 3 (limit: 3297520)
     Memory: 267.3M
        CPU: 5min 57.981s
     CGroup: /system.slice/rsyslog.service
             └─1517556 /usr/sbin/rsyslogd -n

Jun 18 23:52:47 c83f1-app2 rsyslogd[1517556]: imjournal: 29754 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Jun 18 23:54:15 c83f1-app2 rsyslogd[1517556]: imjournal: journal files changed, reloading...  [v8.2102.0-117.el9 try https://www.rsyslog.com/e/0 ]
Jun 18 23:55:16 c83f1-app2 rsyslogd[1517556]: imjournal from <c83f1-app2:node>: begin to drop messages due to rate-limiting
Jun 19 00:02:48 c83f1-app2 rsyslogd[1517556]: imjournal: 60938 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Jun 19 00:05:15 c83f1-app2 rsyslogd[1517556]: imjournal: journal files changed, reloading...  [v8.2102.0-117.el9 try https://www.rsyslog.com/e/0 ]
Jun 19 00:05:17 c83f1-app2 rsyslogd[1517556]: imjournal from <c83f1-app2:->: begin to drop messages due to rate-limiting
Jun 19 00:12:49 c83f1-app2 rsyslogd[1517556]: imjournal: 60688 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Jun 19 00:13:00 c83f1-app2 rsyslogd[1517556]: file size limit cmd for file '/var/log/noobaa.log' did no resolve situation  [v8.2102.0-117.el9]
Jun 19 00:15:16 c83f1-app2 rsyslogd[1517556]: imjournal from <c83f1-app2:node>: begin to drop messages due to rate-limiting
Jun 19 00:16:12 c83f1-app2 rsyslogd[1517556]: imjournal: journal files changed, reloading...  [v8.2102.0-117.el9 try https://www.rsyslog.com/e/0 ]

naveenpaul1 · 2024-06-19T06:32:40Z

@rkomandu Could you please share the output for command
cat /etc/logrotate.d/noobaa/logrotate_noobaa.conf

rkomandu · 2024-06-19T06:47:30Z

/var/log/noobaa.log
{
    daily
    maxsize 100M
    minsize 50M
    start 1
    missingok
    rotate 100
    compress
    create 644 root root
    sharedscripts
    postrotate
        killall -HUP rsyslogd || true
        killall -HUP syslogd || true
    endscript
}

/var/log/noobaa_events.log
{
    daily
    maxsize 100M
    minsize 10M
    start 1
    missingok
    rotate 100
    compress
    create 644 root root
    sharedscripts
    postrotate
        killall -HUP rsyslogd || true
        killall -HUP syslogd || true
    endscript
}

/var/log/client_noobaa.log
{
    size 100M
    start 1
    missingok
    rotate 10
    compress
    create 644 root root
    sharedscripts
    postrotate
        killall -HUP rsyslogd || true
        killall -HUP syslogd || true
    endscript
}

ls -l /etc/logrotate.d/noobaa/logrotate_noobaa.conf
lrwxrwxrwx. 1 root root 66 Jun 11 19:07 /etc/logrotate.d/noobaa/logrotate_noobaa.conf -> /usr/local/noobaa-core/src/deploy/standalone/logrotate_noobaa.conf

rkomandu · 2024-06-20T04:13:14Z

@naveenpaul1 , as we discussed in our rsyslog call regarding the log rotate not functioning as expected.

Below is on a fresh BM system, the log hasn't been rotated automatically

gpfs-p-gui0.:  -rw-r--r--. 1 root root 106M Jun 19 12:48 /var/log/noobaa.log
gpfs-p-gui1.:  -rw-r--r--. 1 root root 64M Jun 20 00:09 /var/log/noobaa.log

rkomandu · 2024-06-21T07:19:46Z

@naveenpaul1
One more thing is once it hits 106M, the noobaa.log doesn't get updated, unless we did the "rsyslog" restart or delete the contents (for testing purpose) and run workload , the noobaa.log gets updated. This is going to be a problem in the field.

Hence there are two things here (which can be interrelated w/r/t logrotate not happening, hence the noobaa.log can't add more data to the current size of 106M).

Request you to look through this as a priority to get workaround for our GA (i know we can run manually logrotate command), but still we might loose some data once log reaches that size limit.

@guymguym this has to have solution for us and fix this as part of the overall logging implementation later on.

rkomandu · 2024-06-26T06:54:40Z

@naveenpaul1 , as per our slack discussion (https://ibm-systems-storage.slack.com/archives/D06A78LUJUQ/p1719223293820039?thread_ts=1719219424.304859&cid=D06A78LUJUQ) noobaa-5.17 rpm is installed and still we see issue w/r/t some minutes are kind of lost while executing the IO benchmark (rpm: noobaa-core-5.17.0-20240624.el9.x86_64 --> noobaa-core-5.17.0-rsyslog-fail-bm-20240624.el9.x86_64.rpm )

It is stopping after every few mins (in noobaa.log --> we see this as follows

first time after 02:17:14 --> 02:26:00 no update to noobaa.log

Jun 26 02:17:14 c83f1-app3 [3579724]: [nsfs/3579724] ESC[36m   [L0]ESC[39m core.endpoint.s3.ops.s3_put_object_uploadId:: PUT OBJECT PART bucket-warp-put-navee
n-logfix SLoL)hK(/159.iX8AZNr9Bsd7aPR2.rnd ESC[33m5ESC[39m
Jun 26 02:17:14 c83f1-app3 [3579724]: [nsfs/3579724] ESC[36m   [L0]ESC[39m core.endpoint.s3.ops.s3_put_object_uploadId:: PUT OBJECT PART bucket-warp-put-navee
n-logfix owPRxk5n/159.bmgkOc9vBfVVny(C.rnd ESC[33m1ESC[39m
Jun 26 02:17:14 c83f1-app3 [3579724]: [nsfs/3579724] ESC[36m   [L0]ESC[39m core.endpoint.s3.ops.s3_put_object_uploadId:: PUT OBJECT PART bucket-warp-put-navee
n-logfix VX3tWBPi/160.f1pw3AG3HP0ssl8o.rnd ESC[33m1ESC[39m
Jun 26 02:26:00 c83f1-app3 [3579724]: [nsfs/3579724] ESC[36m   [L0]ESC[39m core.endpoint.s3.ops.s3_put_object_uploadId:: PUT OBJECT PART bucket-warp-put-navee
n-logfix UIGJfSZF/669.2yVIDYnYd4p7QnGp.rnd ESC[33m4ESC[39m
Jun 26 02:26:01 c83f1-app3 [3579725]: [nsfs/3579725] ESC[36m   [L0]ESC[39m core.endpoint.s3.ops.s3_put_object_uploadId:: PUT OBJECT PART bucket-warp-put-navee
n-logfix ks1rkTAt/676.IhjJbVbgl0(gRwcc.rnd ESC[33m2ESC[39m
Jun 26 02:26:01 c83f1-app3 [3579724]: [nsfs/3579724] ESC[36m   [L0]ESC[39m core.endpoint.s3.ops.s3_put_object_uploadId:: PUT OBJECT PART bucket-warp-put-navee
n-logfix LyJvcuXQ/679.YX6tZmCc7CsrW8lQ.rnd ESC[33m5ESC[39m
Jun 26 02:26:01 c83f1-app3 [3579724]: [nsfs/3579724] ESC[36m   [L0]ESC[39m core.endpoint.s3.ops.s3_put_object_uploadId:: PUT OBJECT PART bucket-warp-put-navee
n-logfix qr5L6PsE/671.yuFesdgrNImeWtvq.rnd ESC[33m1ESC[39m

Second time no update to the noobaa.log (it doesn't update, same is showed above for about 8m 41sec i.e 02:27:20 --> 02:36:01)

Jun 26 02:27:20 c83f1-app3 [3579725]: [nsfs/3579725] ESC[36m   [L0]ESC[39m core.endpoint.s3.ops.s3_put_object_uploadId:: PUT OBJECT PART bucket-warp-put-naveen-logfix ytlGVf8J/753.ykZfE)UHa8J90V6B.rnd ESC[33m1ESC[39m
Jun 26 02:27:20 c83f1-app3 [3579725]: [nsfs/3579725] ESC[36m   [L0]ESC[39m core.endpoint.s3.ops.s3_put_object_uploadId:: PUT OBJECT PART bucket-warp-put-naveen-logfix kD8483FL/751.yQP3w6MByQKcWXZt.rnd ESC[33m5ESC[39m
Jun 26 02:27:20 c83f1-app3 [3579732]: [nsfs/3579732] ESC[36m   [L0]ESC[39m core.endpoint.s3.ops.s3_put_object_uploadId:: PUT OBJECT PART bucket-warp-put-naveen-logfix SLoL)hK(/749.)BxyIzEVzZ9Dzzvr.rnd ESC[33m4ESC[39m
Jun 26 02:36:01 c83f1-app3 [3579732]: [nsfs/3579732] ESC[36m   [L0]ESC[39m core.endpoint.s3.ops.s3_put_object_uploadId:: PUT OBJECT PART bucket-warp-put-naveen-logfix ks1rkTAt/1269.TcPGfaDC)xSblgSo.rnd ESC[33m6ESC[39m
Jun 26 02:36:01 c83f1-app3 [3579732]: [nsfs/3579732] ESC[36m   [L0]ESC[39m core.endpoint.s3.ops.s3_put_object_uploadId:: PUT OBJECT PART bucket-warp-put-naveen-logfix dy402Q9k/1271.hjMRvkX97(Rnu5LS.rnd ESC[33m5ESC[39m
Jun 26 02:36:01 c83f1-app3 [3579725]: [nsfs/3579725] ESC[36m   [L0]ESC[39m core.endpoint.s3.ops.s3_put_object_uploadId:: PUT OBJECT PART bucket-warp-put-naveen-logfix 2tx9o8wV/1264.YCTlp1QKDWDhwK34.rnd ESC[33m1ESC[39m
Jun 26 02:36:01 c83f1-app3 [3579725]: [nsfs/3579725] ESC[36m   [L0]ESC[39m core.endpoint.s3.ops.s3_put_object_uploadId:: PUT OBJECT PART bucket-warp-put-naveen-logfix 9i9yITHW/1262.iETckKbWoW4PNmJI.rnd ESC[33m5ESC[39m
Jun 26 02:36:01 c83f1-app3 [3579725]: [nsfs/3579725] ESC[36m   [L0]ESC[39m core.endpoint.s3.ops.s3_put_object_uploadId:: PUT OBJECT PART bucket-warp-put-naveen-logfix kD8483FL/1268.YqELtyblRo4tWj2N.rnd ESC[33m3ESC[39m

timestamps as well (02:31:22 EDT and 02:34:17 EDT)

"ls -lrt /var/log/noobaa.log" ; date
c83f1-app4-hs200:  -rw-r--r--. 1 root root 47453 Jun 26 02:31 /var/log/noobaa.log
c83f1-app2-hs200:  -rw-r--r--. 1 root root 49303 Jun 26 02:31 /var/log/noobaa.log
c83f1-app3-hs200:  -rw-r--r--. 1 root root 4238272 Jun 26 02:27 /var/log/noobaa.log
Wed Jun 26 02:31:22 AM EDT 2024

 "ls -lrt /var/log/noobaa.log" ; date
c83f1-app3-hs200:  -rw-r--r--. 1 root root 4238272 Jun 26 02:27 /var/log/noobaa.log
c83f1-app4-hs200:  -rw-r--r--. 1 root root 56106 Jun 26 02:34 /var/log/noobaa.log
c83f1-app2-hs200:  -rw-r--r--. 1 root root 58465 Jun 26 02:34 /var/log/noobaa.log
Wed Jun 26 02:34:17 AM EDT 2024

As you can observe from above noobaa.log, there is no log entry after 02:27:20 --> 02:36:01 (all of them are lost) , it is happening repeatedly .

Fix has to be checked further why does it halt in between for updating the logs while IO is still running, as the logging should be continuous

 on the system wrt noobaa.sh and noobaa.conf file 

 ls -lrt /etc/logrotate.d/noobaa/logrotate_noobaa.*
-rwxr-xr-x. 1 root root  88 Jun 24 05:25 /etc/logrotate.d/noobaa/logrotate_noobaa.sh
-rw-r--r--. 1 root root 791 Jun 24 05:25 /etc/logrotate.d/noobaa/logrotate_noobaa.conf

less /etc/logrotate.d/noobaa/logrotate_noobaa.sh
#!/bin/bash
/usr/sbin/logrotate -v /etc/logrotate.d/noobaa/logrotate_noobaa.conf || true

noobaa.conf file (snippet)
/var/log/noobaa.log
{
    daily
    maxsize 100M
    minsize 50M
    start 1
    missingok
    rotate 100
    compress
    create 644 root root
    sharedscripts
    postrotate
        killall -HUP rsyslogd || true
        killall -HUP syslogd || true
    endscript
}

naveenpaul1 · 2024-06-26T10:23:35Z

@rkomandu @guymguym We need to remove imjournal rate limit from rsyslog, because of this only max 20k messages can be pushed to imjournal every 10mnt
We can disable rate limit using below configuration
$imjournalRatelimitInterval 0
$imjournalRatelimitBurst 0

rkomandu · 2024-07-01T03:42:50Z

@naveenpaul1 and I had been on the BM and saw the system didn't get through with the rotation, so Naveen will analyze further and come back on the fix.

this logging is an important fix required from our customer / in-house issues for tracking, loosing them in between as posted in my last comment is not a good sign as well.

naveenpaul1 · 2024-07-01T04:25:45Z

I was checking the issue in BM shared by @rkomandu.
Testing the issue with a few changes

moving script(logrotate_noobaa.sh) and logrotate file(logrotate_noobaa.conf) file to /etc/logrotate.d/noobaa, Before we were using symlink for that.
update script location in noobaa_syslog.conf to /etc/logrotate.d/noobaa/logrotate_noobaa.sh
Changes : https://github.com/noobaa/noobaa-core/compare/master...naveenpaul1:noobaa-core:rsyslog-fail-bm?expand=1

and looks like script(/etc/logrotate.d/noobaa/logrotate_noobaa.sh) mentioned in outchannel is not getting executed, I can execute the script manually from the terminal but not from rsyslog.
@rkomandu @romayalon

romayalon · 2024-07-04T14:03:05Z

@rkomandu @naveenpaul1 Fixed by Naveen on #8182, closing and adding request validation label
Ravi please verify so we can backport it to 5.15.5

rkomandu · 2024-07-05T06:01:05Z

@romayalon @guymguym
disabling the ratelimit in the entire rsyslog is acceptable? as it might turn on for other services running on the system ?

rkomandu · 2024-07-05T06:31:06Z

@naveenpaul1 @romayalon

we should delete the /etc/logrotate.d/noobaa directory
secondly why there are 2 files

 diff logrotate_noobaa.conf noobaa-logrotate
14a15
>         systemctl reload syslog-ng > /dev/null 2>&1 || true
31a33
>         systemctl reload syslog-ng > /dev/null 2>&1 || true
46a49
>         systemctl reload syslog-ng > /dev/null 2>&1 || true

 ls -lrt logrotate_noobaa.conf noobaa-logrotate
-rw-r--r--. 1 root root 791 Jun 24 05:25 logrotate_noobaa.conf
-rw-r--r--. 1 root root 971 Jul  4 19:05 noobaa-logrotate

Aren't we having a single file ? it is confusing for the customer as well
This is on the 5.17 master branch noobaa-core-5.17.0-20240704.el9.x86_64

naveenpaul1 · 2024-07-05T07:07:30Z

@rkomandu
Q1 : We have made changes to debug module if customer set debug level to all then only all the message will pushed to log and ratelimit disable will have any effect in that case only. Default log level will not send all logs so this should be fine most of the time.
Q2: logrotate_noobaa.conf is leftover from previous RPM installation, Please try in clean system.

rkomandu · 2024-07-05T07:24:38Z

@naveenpaul1
it is a clean system (that is the reason we should be cautious when we clean-up the configuration, if these files are created other than rpm based ). it looks like the clean-up is not done from your side on the c83f1-app2 node

I don't see the same on the app3 and app4. please ensure to clean it up

Till now once the noobaa rpm is removed/erased, the respective files are cleared all the time.

naveenpaul1 · 2024-07-05T07:28:17Z

@rkomandu That file was created manually for testing that's why it didnt got removed by rpm uninstall, You can ignore it for now.

rkomandu · 2024-07-05T09:04:00Z

@naveenpaul1 , so there is no limit of 105M now looks like with the new rpm

Every 20.0s: mmdsh -N all "ls -lrt /var/log/noobaa.log*; ls -lh /var/log/noobaa.log*"                                     c83f1-app2: Fri Jul  5 05:01:26 2024

c83f1-app2-hs200:  -rw-r--r--. 1 root root 111265068 Jul  5 05:01 /var/log/noobaa.log
c83f1-app2-hs200:  -rw-r--r--. 1 root root 107M Jul  5 05:01 /var/log/noobaa.log
...another instance 
c83f1-app2-hs200:  -rw-r--r--. 1 root root 112906803 Jul  5 05:03 /var/log/noobaa.log
c83f1-app2-hs200:  -rw-r--r--. 1 root root 108M Jul  5 05:03 /var/log/noobaa.log

naveenpaul1 · 2024-07-05T09:30:26Z

@rkomandu When logrotate rotate the noobaa.log at midnight it will use noobaa-logroate file to check for file size limit.

rkomandu · 2024-07-06T15:11:30Z

@naveenpaul1

logrotate did take place at 23:59

"ls -lrt /var/log/noobaa.log*" --> on all nodes 
c83f1-app3-hs200:  -rw-r--r--. 1 root root 9578573 Jul  5 23:59 /var/log/noobaa.log-20240706.gz
c83f1-app3-hs200:  -rw-r--r--. 1 root root 2402675 Jul  6 11:07 /var/log/noobaa.log
c83f1-app2-hs200:  -rw-r--r--. 1 root root 9665399 Jul  5 23:59 /var/log/noobaa.log-20240706.gz
c83f1-app2-hs200:  -rw-r--r--. 1 root root 2400803 Jul  6 11:07 /var/log/noobaa.log
c83f1-app4-hs200:  -rw-r--r--. 1 root root 16642430 Jul  5 23:59 /var/log/noobaa.log-20240706.gz
c83f1-app4-hs200:  -rw-r--r--. 1 root root  2376338 Jul  6 11:07 /var/log/noobaa.log

As discussed with you at length over slack, the app3, app4 nodes didn't have that extra file (logrotate_noobaa.conf) which was manually created by you on app2.

rpm -qf /etc/logrotate.d/logrotate_noobaa.conf
file /etc/logrotate.d/logrotate_noobaa.conf is not owned by any package

rpm: noobaa-core-5.17.0-20240704.el9.x86_64

naveenpaul1 · 2024-07-07T05:37:56Z

Thank you @rkomandu for your confirmation.

rkomandu added the NS-FS label Jun 18, 2024

romayalon assigned naveenpaul1 Jun 30, 2024

naveenpaul1 mentioned this issue Jul 3, 2024

NC | Noobaa log is not performing logrotate #8182

Merged

2 tasks

romayalon closed this as completed Jul 4, 2024

rkomandu mentioned this issue Jul 6, 2024

Enhance logrotate function to support syslog-ng if it used #8098

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

noobaa log is not performing logrotate #8153

noobaa log is not performing logrotate #8153

rkomandu commented Jun 18, 2024 •

edited

Loading

naveenpaul1 commented Jun 18, 2024

rkomandu commented Jun 19, 2024

rkomandu commented Jun 19, 2024

naveenpaul1 commented Jun 19, 2024

rkomandu commented Jun 19, 2024

rkomandu commented Jun 20, 2024

rkomandu commented Jun 21, 2024

rkomandu commented Jun 26, 2024

naveenpaul1 commented Jun 26, 2024

rkomandu commented Jul 1, 2024

naveenpaul1 commented Jul 1, 2024 •

edited

Loading

romayalon commented Jul 4, 2024

rkomandu commented Jul 5, 2024

rkomandu commented Jul 5, 2024 •

edited

Loading

naveenpaul1 commented Jul 5, 2024 •

edited

Loading

rkomandu commented Jul 5, 2024 •

edited

Loading

naveenpaul1 commented Jul 5, 2024

rkomandu commented Jul 5, 2024

naveenpaul1 commented Jul 5, 2024

rkomandu commented Jul 6, 2024

naveenpaul1 commented Jul 7, 2024

noobaa log is not performing logrotate #8153

noobaa log is not performing logrotate #8153

Comments

rkomandu commented Jun 18, 2024 • edited Loading

Environment info

Actual behavior

Expected behavior

Steps to reproduce

More information - Screenshots / Logs / Other output

naveenpaul1 commented Jun 18, 2024

rkomandu commented Jun 19, 2024

rkomandu commented Jun 19, 2024

naveenpaul1 commented Jun 19, 2024

rkomandu commented Jun 19, 2024

rkomandu commented Jun 20, 2024

rkomandu commented Jun 21, 2024

rkomandu commented Jun 26, 2024

naveenpaul1 commented Jun 26, 2024

rkomandu commented Jul 1, 2024

naveenpaul1 commented Jul 1, 2024 • edited Loading

romayalon commented Jul 4, 2024

rkomandu commented Jul 5, 2024

rkomandu commented Jul 5, 2024 • edited Loading

naveenpaul1 commented Jul 5, 2024 • edited Loading

rkomandu commented Jul 5, 2024 • edited Loading

naveenpaul1 commented Jul 5, 2024

rkomandu commented Jul 5, 2024

naveenpaul1 commented Jul 5, 2024

rkomandu commented Jul 6, 2024

naveenpaul1 commented Jul 7, 2024

rkomandu commented Jun 18, 2024 •

edited

Loading

naveenpaul1 commented Jul 1, 2024 •

edited

Loading

rkomandu commented Jul 5, 2024 •

edited

Loading

naveenpaul1 commented Jul 5, 2024 •

edited

Loading

rkomandu commented Jul 5, 2024 •

edited

Loading