Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nextcloud displays wrong used storage #25283

Closed
olizimmermann opened this issue Jan 22, 2021 · 119 comments
Closed

Nextcloud displays wrong used storage #25283

olizimmermann opened this issue Jan 22, 2021 · 119 comments
Assignees
Labels
1. to develop Accepted and waiting to be taken care of 25-feedback bug

Comments

@olizimmermann
Copy link

olizimmermann commented Jan 22, 2021

How to use GitHub

  • Please use the 👍 reaction to show that you are affected by the same issue.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.

Steps to reproduce

  1. Install Nextcloud
  2. Create some users
  3. Upload some data
  4. check at the users overview the used storage
  5. compare it with the real used storage within the users folder via shell

Expected behaviour

Tell us what should happen
Should be 1:1 like all my other users.

Actual behaviour

Tell us what happens instead
Real used storage: 9.3 Gb
Nextcloud shows me: 18.2

Server configuration

Operating system:
FreeBsd
Web server:
Nginx
Database:

PHP version:
7.04
Nextcloud version: (see Nextcloud admin page)
20.0.5
Updated from an older Nextcloud/ownCloud or fresh install:

Where did you install Nextcloud from:
TrueNas - Plugin - latest Update via Updater
Signing status:

Signing status
Login as admin user into your Nextcloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results here.

No errors have been found.

List of activated apps:

App list Notes, Decks, Tasks
If you have access to your command line run e.g.:
sudo -u www-data php occ app:list
from within your Nextcloud installation folder

Nextcloud configuration:

Config report
If you have access to your command line run e.g.:
sudo -u www-data php occ config:list system
from within your Nextcloud installation folder

or 

Insert your config.php content here. 
Make sure to remove all sensitive content such as passwords. (e.g. database password, passwordsalt, secret, smtp password, …)

CONFIG = array (
'apps_paths' =>
array (
0 =>
array (
'path' => '/usr/local/www/nextcloud/apps',
'url' => '/apps',
'writable' => true,
),
1 =>
array (
'path' => '/usr/local/www/nextcloud/apps-pkg',
'url' => '/apps-pkg',
'writable' => true,
),
),
'logfile' => '/var/log/nextcloud/nextcloud.log',
'passwordsalt' => 'XXXXX',
'secret' => 'XXXXX',
'trusted_domains' =>
array (
0 => 'localhost',
),
'datadirectory' => '/usr/local/www/nextcloud/data',
'dbtype' => 'mysql',
'version' => '20.0.5.2',
'overwrite.cli.url' => 'http://localhost',
'overwriteprotocol' => 'https',
'dbname' => 'nextcloud',
'dbhost' => 'localhost',
'dbport' => '',
'dbtableprefix' => 'oc_',
'mysql.utf8mb4' => true,
'dbuser' => 'XXX',
'dbpassword' => 'XXX',
'installed' => true,
'instanceid' => 'XXX',
'twofactor_enforced' => 'false',
'twofactor_enforced_groups' =>
array (
0 => 'admin',
),
'twofactor_enforced_excluded_groups' =>
array (
),
'mail_smtpmode' => 'smtp',
'mail_sendmailmode' => 'smtp',
'mail_domain' => 'gmail.com',
'mail_smtphost' => 'smtp.gmail.com',
'mail_smtpport' => '587',
'mail_smtpsecure' => 'tls',
'mail_smtpauthtype' => 'LOGIN'..

Are you using external storage, if yes which one: local/smb/sftp/...
No
Are you using encryption: yes/no
No
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
No

LDAP configuration (delete this part if not used)

LDAP config
With access to your command line run e.g.:
sudo -u www-data php occ ldap:show-config
from within your Nextcloud installation folder

Without access to your command line download the data/owncloud.db to your local
computer or access your SQL server remotely and run the select query:
SELECT * FROM `oc_appconfig` WHERE `appid` = 'user_ldap';


Eventually replace sensitive data as the name/IP-address of your LDAP server or groups.

Client configuration

Browser:
all / also in app
Operating system:

Logs

Web server error log

Web server error log
Insert your webserver log here

Nextcloud log (data/nextcloud.log)

Nextcloud log
Insert your Nextcloud log here
Shell:

image

Online:
image

@olizimmermann olizimmermann added 0. Needs triage Pending check for reproducibility or if it fits our roadmap bug labels Jan 22, 2021
@szaimen

This comment was marked as resolved.

@olizimmermann

This comment was marked as resolved.

@szaimen
Copy link
Contributor

szaimen commented Jun 22, 2021

Does it work after you run occ files:scan sz and occ files:cleanup?

@olizimmermann
Copy link
Author

Does it work after you run occ files:scan sz and occ files:cleanup?

Nope. It doesn't work. In Nextcloud it still says: 18.2GB used, but the disk only uses 9.3Gb...

@szaimen szaimen added 1. to develop Accepted and waiting to be taken care of and removed 0. Needs triage Pending check for reproducibility or if it fits our roadmap needs info labels Jun 22, 2021
@pierr0t
Copy link

pierr0t commented Sep 13, 2021

Hello,

It's a problem I discussed on the forums ... I use "Quota Warning" app in Nextcloud, but my problem is the opposite, for example in a NC 21.0.3 instance with a single user (he uses it to sync between 3 machines) the NC interface shows 94gb used and the "Quota warning" app reports 94% usage because I did set a 100GB quota for that user, so far so good.

Problem, if I check on the server itself (debian 10 up to date) the real usage is 152GB. I was explained that the quota do not account for the deleted files, the versions, etc ... so this makes user's quota completely useless, in this case I put a 100GB quota so the client is notified when reaching 90GB but on the server the client already reached 150GB ... This already lead to his NC beeing stuck because the server quota at 200GB was reached (and Debian blocked additional files) while the client thought everything was fine (he was seeing 15OGB usage).

I see the problem on all my NC instances (14), most of the time the storage as seen on the server largely exceeding the quota in NC ... I do not recall seeing the opposite like you do and generally I see the problem for all users not only one, so my problem might be different from you but I am continuously searching for new info on that problem so I thought I would follow this thread.

@alars
Copy link

alars commented Jul 9, 2022

On an NC 24.0.2 instance (with only local storage) the NC 'Users' page shows disk usage that is way off for half of the users and (almost) correct for the other half.
Example: User 1: actual disk usage 151GB, displayed usage only 648.8MB; user 2: actual disk usage 376GB, displayed usage only 2.3GB
Neither occ files:scan --all nor occ files:cleanup fixes the issue.

This is on Linux with Apache 2.4.41 and php 7.4.3

@Pinkbyte
Copy link

Pinkbyte commented Jul 27, 2022

+1 for this issue.

Nextcloud deployed from docker container (Image - nextcloud:24.0.3)

du -hs for ENTIRE Nextcloud data folder shows me 28,3 Gb, while in web interface for one user(which has most of this data) it is shown as 37,2 Gb used.

No external storage. Versions and trashbin plugins are enabled though

@htc1977
Copy link

htc1977 commented Aug 11, 2022

On an NC 24.0.2 instance (with only local storage) the NC 'Users' page shows disk usage that is way off for half of the users and (almost) correct for the other half. Example: User 1: actual disk usage 151GB, displayed usage only 648.8MB; user 2: actual disk usage 376GB, displayed usage only 2.3GB Neither occ files:scan --all nor occ files:cleanup fixes the issue.

This is on Linux with Apache 2.4.41 and php 7.4.3

same here
user 1 real usage: 19.8 GB -> nextcloud shows 2,9 GB
user 2 real usage: 41.2 GB -> nextcloud shows 62 KB

Ubuntu 20.04.4 LTS, Apache 2.4.41, php 7.4.3, NC 24.0.3

@jshatch
Copy link

jshatch commented Aug 15, 2022

I have run into something similar to the issue reported here, it may be the same thing. My user has about 250GB of data but the UI reports only 78MB used. I had run out of my quota at 250GB and it was reporting I could not upload any more, but it was saying 78MB/78MB used. When I upped the quota to 500GB it now reports 78MB/250GB used. I deleted a large file from my instance that would have very easily sent that 78MB number into the negative by a few GB, and it didn't move. The total briefly showed 500GB after that, but upon refreshing the page it returned to the previous value. Something is wrong, none of the DB repair or file scan occ commands do anything. I updated to 24.0.4 today and there is no change. Access to files is not affected, just reporting of space used.

@eloo
Copy link

eloo commented Aug 16, 2022

i see the same issue with 24.0.4

one user has aroung 12gb data but nextcloud is reporting 164kb :D

Please fix that. This makes the quota feature really unusable

@dominictrier
Copy link

We are seeing something similar to this i believe. (24.0.3)

It seems the updating of Folder sizes after deleing is severely delayed or not happening at all. Like is "sticky"
After deleting a file from a folder the folder size is not updated and the old size keeps counting against the quota.
Looking at the same folder via Filesystem or Webdav shows them as 0kb.
This is mostly affecting our main user, used for sharing out folders to the other users.

In our case it's a shared Hetzner instance. They have no solution and cant reproduce it on their test system.
They basically gave up and suggested to try to post on github and hope for the best.

@simonspa
Copy link
Contributor

simonspa commented Sep 8, 2022

I also see this issue, and it persists with NC 24.0.5:

du -hs ncdata/username
80G	ncdata/username

image

@eidermar
Copy link

We are seeing something similar to this i believe. (24.0.3)

It seems the updating of Folder sizes after deleing is severely delayed or not happening at all. Like is "sticky" After deleting a file from a folder the folder size is not updated and the old size keeps counting against the quota. Looking at the same folder via Filesystem or Webdav shows them as 0kb. This is mostly affecting our main user, used for sharing out folders to the other users.

In our case it's a shared Hetzner instance. They have no solution and cant reproduce it on their test system. They basically gave up and suggested to try to post on github and hope for the best.

We are also with Hetzner and the reported "GB used" value from within the web-interface is almost half from the actual space used according to the Hetzner console. As there is no access via terminal or a web-console I'm having a hard time to troubleshoot this. I suspect that there are leftovers from failed uploads somewhere in the file system as reported in other threads with similar topics. We tried deleting all individual as well as admin trash bins and looked through TBs of files to see if any excessive versioning was going on somewhere...without any success. Is there really no way to show the allocation of data in the web-interface? Or does anybody have any other idea what else we could try to get rid of this junk (almost 500GBs) that is obviously comfortably lurking around somewhere and blocking valuable space...without having access to the file structure or a command line?

TIA

@simonspa
Copy link
Contributor

We are seeing something similar to this i believe. (24.0.3)
It seems the updating of Folder sizes after deleing is severely delayed or not happening at all. Like is "sticky" After deleting a file from a folder the folder size is not updated and the old size keeps counting against the quota. Looking at the same folder via Filesystem or Webdav shows them as 0kb. This is mostly affecting our main user, used for sharing out folders to the other users.
In our case it's a shared Hetzner instance. They have no solution and cant reproduce it on their test system. They basically gave up and suggested to try to post on github and hope for the best.

We are also with Hetzner and the reported "GB used" value from within the web-interface is almost half from the actual space used according to the Hetzner console. As there is no access via terminal or a web-console I'm having a hard time to troubleshoot this. I suspect that there are leftovers from failed uploads somewhere in the file system as reported in other threads with similar topics. We tried deleting all individual as well as admin trash bins and looked through TBs of files to see if any excessive versioning was going on somewhere...without any success. Is there really no way to show the allocation of data in the web-interface? Or does anybody have any other idea what else we could try to get rid of this junk (almost 500GBs) that is obviously comfortably lurking around somewhere and blocking valuable space...without having access to the file structure or a command line?

TIA

So in my case it definitely is not leftovers from deleted files or versions. This is read data. I am hosting my own server, so I have full access to cli and database, but I'm not sure what info could help.

I am positive that this is just a problem in calculating the storage on the Nextcloud side, because both the user folder on the server as well as my locally synced data folder with the NC desktop client report > 70 GB of data being present.

@MichaelSp
Copy link

MichaelSp commented Sep 18, 2022

I had the same issue. I ended up creating a backup of oc_filecache, TRUNCATE TABLE oc_filecache and running occ files:scan --all. Now it looks better.

edit: Don't try this at home, kids. People - reported problems with this.

@Pinkbyte
Copy link

I had the same issue. I ended up creating a backup of oc_filecache, TRUNCATE TABLE oc_filecache and running occ files:scan --all. Now it looks better.

Yeah, that definitely helps, just wonder why running 'occ files:scan --all' did not help WITHOUT truncating cache tables in database

@tobyheywood
Copy link

As suggested by @MichaelSp and confirmed by @Pinkbyte the only way for me to fix this was to truncate the nc_filecache table and then run files:scan --all
Thanks for sharing the workaround

@alars
Copy link

alars commented Oct 3, 2022

Warning: Only truncate *c_filecache if you are not using encryption, otherwise all files are treated as plain unencrypted (and therefore unusable) afterwards!

@htc1977
Copy link

htc1977 commented Oct 14, 2022

When I truncate table oc_filecache and run occ files:scan --all, all my shares and favorites are gone. :(

@jshatch
Copy link

jshatch commented Oct 15, 2022

Rather than truncating tables or other hacks I would really like to see a dev weigh in on this issue and fix it in the code.

@anguschang007
Copy link

using NC25 with the same issue!

@seieric
Copy link

seieric commented Nov 3, 2022

Me, too. Even with NC25, the same issue occurs.

@PVince81
Copy link
Member

PVince81 commented Nov 3, 2022

I had this too, the problem was that some of the size accounting was not up to date.

Background: to optimize size propagation, Nextcloud uses an algorithm that adds or substracts sizes across all parent folders of a modified path. If for some reason this didn't happen, for example if the PHP process got killed shortly after an upload, the oc_filecache will contain wrong values.

Now, the "occ files:scan" command doesn't automatically fix sizes.
I'd say the "occ files:scan" command should be extended to also fix folder sizes since it's anyway going to go through all folders.

Thoughts ? @icewind1991

@doc75
Copy link
Contributor

doc75 commented Jan 22, 2023

I run the following command for my user to ensure that unencrypted_size is set to 0 on non encrypted file on the main storage:

update oc_filecache set unencrypted_size=0 where encrypted=0 and unencrypted_size!=0 and storage=(select numeric_id from oc_storages where id=1);

I then checked that this is correct by running the following request:

select storage, size, unencrypted_size, etag from oc_filecache where storage=1 and path='files';

whic returns:

 storage |    size    | unencrypted_size |     etag      
---------+------------+------------------+---------------
       1 | 6620645422 |                0 | 63cd2f3210742

After a couple of minutes, I rerun the same command and now the unencrypted_size is again not 0:

 storage |    size    | unencrypted_size |     etag      
---------+------------+------------------+---------------
       1 | 6620645422 |       6620645502 | 63cd3d4950c52

The files with unencrypted_size != 0 are the following:

               path               |    size    | unencrypted_size 
----------------------------------+------------+------------------
 files/Nextcloud/Sync/Joplin      |    5608590 |          5608599
 files/Nextcloud/Sync/Joplin/temp |         45 |               54
 files/Nextcloud                  | 1426060541 |       1426060550
 files                            | 6620645422 |       6620645502
                                  | 7018155246 |       7018155246
 files/Nextcloud/Sync             |    5608590 |          5608599

I am using Joplin (2.9.17).
All the directory mentioned are used by Joplin for Sync.

I will continue to check this a on a regular basis to see if I can find anything else interesting.

@doc75
Copy link
Contributor

doc75 commented Jan 22, 2023

Tools creating the issue:

  • Joplin WebDAV sync
  • iOS Nextcloud client
  • MacOS Desktop client

Tools not creating the issue:

  • Linux Desktop client
  • Web interface of Files App

Other finding:

  • I can only see directories having unencrypted_size != 0 (which might be normal due to bubble up of the information)

@ppfeufer
Copy link

ppfeufer commented Jan 29, 2023

image

The difference is quite huge.
Server-side encryption is enabled.
NC 25.0.3

@simonspa
Copy link
Contributor

The total size below the file list also includes shared-in folders that do not count towards your own user's quota. Might this explain the difference for you, @ppfeufer ?

@ppfeufer
Copy link

No shared folders.

@nunesgh
Copy link

nunesgh commented Mar 13, 2023

I have the same issue: quota usage reported by Nextcloud is greater than actual disk usage, and only increases.

  • Nextcloud Hub 3: 25.0.4
  • mysql: 10.5.18
  • PHP: 7.4.33

sudo -u www-data php occ config:list system

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "cipher": "AES-256-CFB",
        "login_form_autocomplete": false,
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "25.0.4.1",
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "htaccess.RewriteBase": "\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "enable_previews": true,
        "preview_max_x": 1024,
        "preview_max_y": 1024,
        "preview_max_scale_factor": 10,
        "enabledPreviewProviders": [
            "OC\\Preview\\PNG",
            "OC\\Preview\\JPEG"
        ],
        "filelocking.enabled": true,
        "memcache.local": "\\OC\\Memcache\\Redis",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": "***REMOVED SENSITIVE VALUE***"
        },
        "default_language": "en",
        "default_locale": "***REMOVED SENSITIVE VALUE***",
        "default_phone_region": "***REMOVED SENSITIVE VALUE***",
        "defaultapp": "dashboard,files",
        "knowledgebaseenabled": true,
        "skeletondirectory": "",
        "maintenance": false,
        "twofactor_enforced": true,
        "twofactor_enforced_groups": [
            "***REMOVED SENSITIVE VALUE***"
        ],
        "twofactor_enforced_excluded_groups": [
            "***REMOVED SENSITIVE VALUE***"
        ],
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_sendmailmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpsecure": "ssl",
        "mail_smtpport": "465",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "theme": "",
        "loglevel": 2,
        "logtimezone": "***REMOVED SENSITIVE VALUE***",
        "trashbin_retention_obligation": "auto, 180",
        "versions_retention_obligation": "auto",
        "updater.release.channel": "stable",
        "data-fingerprint": "***REMOVED SENSITIVE VALUE***",
        "connectivity_check_domains": [
            "www.eff.org",
            "www.nextcloud.com"
        ],
        "quota_include_external_storage": false
    }
}

sudo -u www-data php occ app:list

Enabled:
  - activity: 2.17.0
  - admin_audit: 1.15.0
  - announcementcenter: 6.5.1
  - bookmarks: 12.1.0
  - bruteforcesettings: 2.5.0
  - calendar: 4.2.4
  - checksum: 1.2.0
  - circles: 25.0.0
  - cloud_federation_api: 1.8.0
  - comments: 1.15.0
  - contacts: 5.0.3
  - contactsinteraction: 1.6.0
  - dashboard: 7.5.0
  - dav: 1.24.0
  - drop_account: 2.1.0
  - encryption: 2.13.0
  - external: 5.0.2
  - federatedfilesharing: 1.15.0
  - federation: 1.15.0
  - files: 1.20.1
  - files_accesscontrol: 1.15.1
  - files_antivirus: 4.0.2
  - files_automatedtagging: 1.15.2
  - files_external: 1.17.0
  - files_pdfviewer: 2.6.0
  - files_rightclick: 1.4.0
  - files_sharing: 1.17.0
  - files_trashbin: 1.15.0
  - files_versions: 1.18.0
  - fileslibreofficeedit: 1.1.0
  - firstrunwizard: 2.14.0
  - forms: 3.2.0
  - gpoddersync: 3.7.3
  - guests: 2.3.0
  - impersonate: 1.12.0
  - integration_github: 1.0.15
  - integration_gitlab: 1.0.12
  - integration_mastodon: 1.0.6
  - integration_reddit: 1.0.7
  - logreader: 2.10.0
  - lookup_server_connector: 1.13.0
  - news: 21.0.0
  - nextcloud_announcements: 1.14.0
  - notifications: 2.13.1
  - oauth2: 1.13.0
  - password_policy: 1.15.0
  - photos: 2.0.1
  - polls: 4.1.8
  - privacy: 1.9.0
  - provisioning_api: 1.15.0
  - quicknotes: 0.8.5
  - quota_warning: 1.15.0
  - ransomware_protection: 1.14.0
  - recommendations: 1.4.0
  - related_resources: 1.0.4
  - richdocuments: 7.1.1
  - richdocumentscode: 22.5.802
  - serverinfo: 1.15.0
  - settings: 1.7.0
  - sharebymail: 1.15.0
  - side_menu: 3.7.0
  - support: 1.8.0
  - survey_client: 1.13.0
  - systemtags: 1.15.0
  - tasks: 0.14.5
  - text: 3.6.0
  - theming: 2.0.1
  - twofactor_admin: 4.1.9
  - twofactor_backupcodes: 1.14.0
  - twofactor_totp: 7.0.0
  - updatenotification: 1.15.0
  - user_status: 1.5.0
  - viewer: 1.9.0
  - weather_status: 1.5.0
  - workflowengine: 2.7.0
Disabled:
  - suspicious_login: 4.1.0
  - user_ldap

I'm not sure if the following error is related, but it is flooding my Logging app.

{"reqId":"***REMOVED SENSITIVE VALUE***","level":3,"time":"***REMOVED SENSITIVE VALUE***","remoteAddr":"***REMOVED SENSITIVE VALUE***","user":"--","app":"no app in context","method":"GET","url":"/.env","message":"App encryption threw an error during app.php load: No mount for path /files_encryption/OC_DEFAULT_MODULE/pubShare_8c08540b.publicKey existing mounts: ","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36","version":"25.0.4.1","exception":{"Exception":"OCP\\Files\\NotFoundException","Message":"No mount for path /files_encryption/OC_DEFAULT_MODULE/pubShare_8c08540b.publicKey existing mounts: ","Code":0,"Trace":[{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/Files/Filesystem.php","line":320,"function":"find","class":"OC\\Files\\Mount\\Manager","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/Files/View.php","line":1166,"function":"resolvePath","class":"OC\\Files\\Filesystem","type":"::"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/Files/View.php","line":546,"function":"basicOperation","class":"OC\\Files\\View","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/Encryption/Keys/Storage.php","line":269,"function":"file_exists","class":"OC\\Files\\View","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/Encryption/Keys/Storage.php","line":229,"function":"getKey","class":"OC\\Encryption\\Keys\\Storage","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/Encryption/Keys/Storage.php","line":121,"function":"getKeyWithUid","class":"OC\\Encryption\\Keys\\Storage","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/apps/encryption/lib/KeyManager.php","line":614,"function":"getSystemUserKey","class":"OC\\Encryption\\Keys\\Storage","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/apps/encryption/lib/KeyManager.php","line":170,"function":"getPublicShareKey","class":"OCA\\Encryption\\KeyManager","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/apps/encryption/lib/Users/Setup.php","line":62,"function":"validateShareKey","class":"OCA\\Encryption\\KeyManager","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/apps/encryption/lib/AppInfo/Application.php","line":55,"function":"setupSystem","class":"OCA\\Encryption\\Users\\Setup","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/apps/encryption/appinfo/app.php","line":37,"function":"setUp","class":"OCA\\Encryption\\AppInfo\\Application","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/legacy/OC_App.php","line":306,"args":["***REMOVED SENSITIVE VALUE***/nextcloud/apps/encryption/appinfo/app.php"],"function":"require_once"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/legacy/OC_App.php","line":187,"function":"requireAppFile","class":"OC_App","type":"::"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/legacy/OC_App.php","line":141,"function":"loadApp","class":"OC_App","type":"::"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/Files/SetupManager.php","line":132,"function":"loadApps","class":"OC_App","type":"::"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/Files/SetupManager.php","line":340,"function":"setupBuiltinWrappers","class":"OC\\Files\\SetupManager","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/Files/SetupManager.php","line":380,"function":"setupRoot","class":"OC\\Files\\SetupManager","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/Files/Mount/Manager.php","line":95,"function":"setupForPath","class":"OC\\Files\\SetupManager","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/Files/View.php","line":1390,"function":"find","class":"OC\\Files\\Mount\\Manager","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/Files/Node/Root.php","line":205,"function":"getFileInfo","class":"OC\\Files\\View","type":"->"},{"function":"get","class":"OC\\Files\\Node\\Root","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/Files/Node/LazyFolder.php","line":72,"function":"call_user_func_array"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/Files/Node/LazyFolder.php","line":149,"function":"__call","class":"OC\\Files\\Node\\LazyFolder","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/Files/AppData/AppData.php","line":132,"function":"get","class":"OC\\Files\\Node\\LazyFolder","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/Template/JSCombiner.php","line":88,"function":"getFolder","class":"OC\\Files\\AppData\\AppData","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/Template/JSResourceLocator.php","line":125,"function":"process","class":"OC\\Template\\JSCombiner","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/Template/JSResourceLocator.php","line":77,"function":"cacheAndAppendCombineJsonIfExist","class":"OC\\Template\\JSResourceLocator","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/Template/ResourceLocator.php","line":78,"function":"doFind","class":"OC\\Template\\JSResourceLocator","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/TemplateLayout.php","line":379,"function":"find","class":"OC\\Template\\ResourceLocator","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/TemplateLayout.php","line":211,"function":"findJavascriptFiles","class":"OC\\TemplateLayout","type":"::"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/legacy/OC_Template.php","line":184,"function":"__construct","class":"OC\\TemplateLayout","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/Template/Base.php","line":132,"function":"fetchPage","class":"OC_Template","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/base.php","line":817,"function":"printPage","class":"OC\\Template\\Base","type":"->"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/base.php","line":1144,"function":"init","class":"OC","type":"::"},{"file":"***REMOVED SENSITIVE VALUE***/nextcloud/index.php","line":34,"args":["***REMOVED SENSITIVE VALUE***/nextcloud/lib/base.php"],"function":"require_once"}],"File":"***REMOVED SENSITIVE VALUE***/nextcloud/lib/private/Files/Mount/Manager.php","Line":118,"message":"App encryption threw an error during app.php load: No mount for path /files_encryption/OC_DEFAULT_MODULE/pubShare_8c08540b.publicKey existing mounts: ","CustomMessage":"App encryption threw an error during app.php load: No mount for path /files_encryption/OC_DEFAULT_MODULE/pubShare_8c08540b.publicKey existing mounts: "},"id":"***REMOVED SENSITIVE VALUE***"}

Thank you!

@icewind1991
Copy link
Member

For cases where encryption is enabled (or was previously enabled) , applying #36857 and running occ files:scan should resolve the issue

@Jayvratsinh
Copy link

For anyone still facing the same issue even after applying the fix in #36857, you can try this steps:
*Try at your own risk, I am not a developer or expert in this matter.

  1. open mysql
  2. run USE nextcloud;
  3. SELECT path, name, size, encrypted, unencrypted_size FROM oc_filecache WHERE encrypted = 0 AND unencrypted_size != 0; This will show you the files that are read twice as the column unencryptrd_size has some value causing it to be added twice to the occupied space calculation (quota calculation).
  4. UPDATE oc_filecache SET unencrypted_size = 0 WHERE encrypted = 0;
    WARNING! only use this commend if you have already disabled server side encryption using the steps mentioned in the offical administrator's guide provided by nextcloud.
    This should start showing you the correct quota used. If it doesn't, just clear the cache of the browser or give it some time. Altough this is not a permanent fix but works for the time being as you only have to use the command in step 4 to fix this next time.

For anyone working on this bug I have found a few things which migth help. Even after disabling the server side encryption,
'unencrypted_size' is written during the upload of the file. To verify this you could upload a file (around 500 MB for testing) and run the command mentioned in 3 step while the file is being uploaded. This happens on upload from any device meaning even when your phone or laptop auto upload things. And also if you restore any file after applying the fix mentioned above the system again writes the size of that file in both columns 'size' and encrypted_size'.

This is what I have dicovered today with the help of ChatGPT. Please let me know if this was helpful.

@simonspa
Copy link
Contributor

Congratulations to ChatGPT, that was able to find a post in the same ticket just a few pages up... :)

@Justinzobel
Copy link

Is this fixed in a known release?

@max-nextcloud
Copy link
Contributor

Is this fixed in a known release?

The fix related to unencrypted_size is in 27. Backport to 26 did not make it into the latest 26 release: #38555

@mejo-
Copy link
Member

mejo- commented Jul 25, 2023

Given that this was mainly about instances with encryption is enabled, let's close this issue as fixed in #36857.

If anyone still encounters issues with wrong quota calculation (on Nextcloud 27 or newer), please open a new issue about it.

@mejo- mejo- closed this as completed Jul 25, 2023
@doc75
Copy link
Contributor

doc75 commented Aug 6, 2023

@mejo- I confirm that the issue is not reproducible with NC 27.

@akhil1508
Copy link
Contributor

akhil1508 commented Aug 14, 2023

Any plans to backport this to Nextcloud 25?

@icewind1991 Is the fix in #36857 safe to apply to an instance running 25.0.9? Thanks 🫠

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
1. to develop Accepted and waiting to be taken care of 25-feedback bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.