Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: The database keeps growing after upgrade to 29.0.0 #45046

Closed
5 of 8 tasks
janusn opened this issue Apr 26, 2024 · 4 comments
Closed
5 of 8 tasks

[Bug]: The database keeps growing after upgrade to 29.0.0 #45046

janusn opened this issue Apr 26, 2024 · 4 comments
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap 29-feedback bug

Comments

@janusn
Copy link

janusn commented Apr 26, 2024

⚠️ This issue respects the following points: ⚠️

Bug description

The database keeps growing since nextcloud server was upgraded to 29.0.0 yesterday morning. The growth seems to be accelerating! 😱

On my set up, there is a dvr rotating video file on an external storage but the database did not grow before nc29. The number of video files are more or less before and after the nc update.

I run occ files:scan <user> --path="<path>" and $ occ preview:generate-all every 4 hours in order to generate preview for external storages. However, I have noticed the database grows continuously, independent of the files:scan and preview:generate-all.

I have checked the size of the database directory based on snapshots. Here is a summary of the size of the database directory as reported by $ du -sm:

  • 20240425T0030 3893MB
  • 20240425T0130 3893MB
  • 20240425T0230 3893MB
  • 20240425T0330 3893MB
  • 20240425T0430 3893MB
    nc updated to 29.0.0 at 05:16
  • 20240425T0530 3893MB
  • 20240425T0630 3902MB
  • 20240425T0730 3903MB
  • 20240425T0830 3906MB
  • 20240425T0930 3932MB
  • 20240425T1030 3961MB
  • 20240425T1130 3986MB
  • 20240425T1230 4019MB
  • 20240425T1330 4041MB
  • 20240425T1430 4066MB
  • 20240425T1530 4095MB
  • 20240425T1630 4124MB
  • 20240425T1730 4197MB
  • 20240425T1830 4301MB
  • 20240425T1930 4409MB
  • 20240425T2030 4529MB
  • 20240425T2130 4625MB
  • 20240425T2230 4725MB
  • 20240425T2330 4805MB
  • 20240426T0030 4925MB
  • 20240426T0130 4941MB

Steps to reproduce

  1. log the size of the database
  2. upgrade to nc 29.0.0
  3. log the size of the database every hour.

Expected behavior

The size of the database remain stable.

Installation method

Community Docker image

Nextcloud Server version

29

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.3

Web server

Nginx

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Upgraded to a MAJOR version (ex. 22 to 23)

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

  • Default user-backend (database)
  • LDAP/ Active Directory
  • SSO - SAML
  • Other

Configuration report

{
    "system": {
        "memcache.local": "\\OC\\Memcache\\APCu",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "version": "29.0.0.19",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "overwritehost": "***REMOVED SENSITIVE VALUE***",
        "overwriteprotocol": "https",
        "installed": true,
        "default_phone_region": "GB",
        "maintenance": false,
        "loglevel": 2,
        "theme": "",
        "filelocking.enabled": "true",
        "upgrade.disable-web": true,
        "filesystem_check_changes": 0,
        "maintenance_window_start": 1,
        "dbtype": "mysql",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "tls",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "jpeg_quality": "80",
        "preview_max_scale_factor": 1,
        "enabledPreviewProviders": [
            "OC\\Preview\\BMP",
            "OC\\Preview\\GIF",
            "OC\\Preview\\JPEG",
            "OC\\Preview\\Krita",
            "OC\\Preview\\MarkDown",
            "OC\\Preview\\OpenDocument",
            "OC\\Preview\\PNG",
            "OC\\Preview\\TXT",
            "OC\\Preview\\XBitmap"
        ],
        "preview_imaginary_url": "***REMOVED SENSITIVE VALUE***",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379,
            "timeout": 1.5
        }
    }
}

List of activated Apps

Enabled:
  - activity: 2.21.1
  - circles: 29.0.0-dev
  - cloud_federation_api: 1.12.0
  - comments: 1.19.0
  - contactsinteraction: 1.10.0
  - dashboard: 7.9.0
  - dav: 1.30.1
  - federatedfilesharing: 1.19.0
  - federation: 1.19.0
  - files: 2.1.0
  - files_downloadlimit: 2.0.0
  - files_external: 1.21.0
  - files_pdfviewer: 2.10.0
  - files_reminders: 1.2.0
  - files_sharing: 1.21.0
  - files_trashbin: 1.19.0
  - files_versions: 1.22.0
  - firstrunwizard: 2.18.0
  - logreader: 2.14.0
  - lookup_server_connector: 1.17.0
  - nextcloud_announcements: 1.18.0
  - notifications: 2.17.0
  - oauth2: 1.17.0
  - password_policy: 1.19.0
  - previewgenerator: 5.5.0
  - privacy: 1.13.0
  - provisioning_api: 1.19.0
  - recommendations: 2.1.0
  - related_resources: 1.4.0
  - serverinfo: 1.19.0
  - settings: 1.12.0
  - sharebymail: 1.19.0
  - support: 1.12.0
  - survey_client: 1.17.0
  - suspicious_login: 7.0.0
  - systemtags: 1.19.0
  - text: 3.10.0
  - theming: 2.4.0
  - twofactor_backupcodes: 1.18.0
  - twofactor_totp: 11.0.0-dev
  - updatenotification: 1.19.1
  - user_status: 1.9.0
  - viewer: 2.3.0
  - weather_status: 1.9.0
  - workflowengine: 2.11.0
Disabled:
  - admin_audit: 1.19.0
  - bruteforcesettings: 2.9.0
  - encryption: 2.17.0
  - files_rightclick: 1.6.0 (installed 1.6.0)
  - photos: 2.5.0 (installed 2.3.0)
  - user_ldap: 1.20.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

Due to the fact the log is way too long, I have put it at pastebin.

https://pastebin.com/JTTA0tki

Additional info

No response

@janusn janusn added 0. Needs triage Pending check for reproducibility or if it fits our roadmap bug labels Apr 26, 2024
@Dennis1993
Copy link
Contributor

Thanks for your report!
Can you look into the database which table is growing so fast?

@janusn
Copy link
Author

janusn commented Apr 26, 2024

Thanks for the prompt reply.

I have check the database. The table seems to be growing was oc_filecache. It has slowed down and becomes stable again. I will keep an eye on how it goes for next couple days.

The following are the output of SELECT COUNT(fileid) FROM oc_filecache ; with databases from snapshots:

20240425T0030 4113378
20240425T0430 4109820
 nc upgraded to 29.0.0
20240425T0530 4112696
20240425T0630 4141935
20240425T0730 4172441
20240425T0830 4203854
20240425T0930 4236038
20240425T1030 4267886
20240425T1130 4299755
20240425T1230 4330780
20240425T1330 4361222
20240425T1430 4392668
20240425T1530 ***can't start mariadb. skipped ***
20240425T1630 4450938
20240425T1730 4547876
20240425T1830 4688197
20240425T1930 4829444
20240425T2030 4972022
20240425T2130 5107206
20240425T2230 5248395
20240425T2330 5368348
20240426T0030 5505339
20240426T0130 5519785
20240426T0230 5519785
20240426T0330 5519785
20240426T0430 5519163
20240426T0530 5515686
20240426T1730 5510022

@janusn
Copy link
Author

janusn commented Apr 28, 2024

It seems stable so far. It might be caused by something else.

> SELECT COUNT(fileid) FROM oc_filecache ;
5448814

@janusn janusn closed this as completed Apr 28, 2024
@muchachagrande
Copy link

muchachagrande commented May 25, 2024

This is happening to me right now. It seems to be caused by a full file scan that is also generating previews of all images.
Imaginary is fully working.
This started to happen 2 days ago after upgrading from 18.0.6 to 19.0.1.

EDIT:
Confirmed... I followed the tail of audit.log and there is a process scanning all files.
The messages say: "app":"admin_audit","method":"","url":"--","message":"Preview accessed:...." (width: "256", height: "256" crop: "", mode: "fill")"

EDIT 2:
After some time... two or three days, the full scan completed. May be some operations will be faster now given that every file is indexed in the database.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap 29-feedback bug
Projects
None yet
Development

No branches or pull requests

4 participants