Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 storage has folder size "pending" #7420

Closed
In-eX opened this issue Dec 7, 2017 · 30 comments
Closed

S3 storage has folder size "pending" #7420

In-eX opened this issue Dec 7, 2017 · 30 comments
Labels
1. to develop Accepted and waiting to be taken care of bug feature: external storage

Comments

@In-eX
Copy link

In-eX commented Dec 7, 2017

Steps to reproduce

  1. Mount an S3 bucket
  2. Upload Data via Windows client

Expected behaviour

The correct folder size should be displayed.

Actual behaviour

Folder size says "pending" on all levels in the file structure.

image

image

Server configuration

Operating system:
Ubuntu 16.04.3 64bit

Web server:
Apache/2.4.18
nginx/1.11.10

Database:
mysql Ver 14.14 Distrib 5.7.20

PHP version:
PHP-FPM 5.6.32

Nextcloud version: (see Nextcloud admin page)
12.0.4

Updated from an older Nextcloud/ownCloud or fresh install:
Updated to the newest release since ownCloud 9

Where did you install Nextcloud from:
php installation script

Signing status:

Signing status
No errors have been found.

List of activated apps:

App list
Enabled:
  - activity: 2.5.2
  - bruteforcesettings: 1.0.2
  - calendar: 1.5.6
  - comments: 1.2.0
  - contacts: 2.0.1
  - dav: 1.3.0
  - federatedfilesharing: 1.2.0
  - federation: 1.2.0
  - files: 1.7.2
  - files_external: 1.3.0
  - files_pdfviewer: 1.1.1
  - files_sharing: 1.4.0
  - files_texteditor: 2.4.1
  - files_trashbin: 1.2.0
  - files_versions: 1.5.0
  - files_videoplayer: 1.1.0
  - firstrunwizard: 2.1
  - gallery: 17.0.0
  - logreader: 2.0.0
  - lookup_server_connector: 1.0.0
  - mail: 0.7.5
  - nextcloud_announcements: 1.1
  - notifications: 2.0.0
  - oauth2: 1.0.5
  - password_policy: 1.2.2
  - provisioning_api: 1.2.0
  - serverinfo: 1.2.0
  - sharebymail: 1.2.0
  - survey_client: 1.0.0
  - systemtags: 1.2.0
  - theming: 1.3.0
  - twofactor_backupcodes: 1.1.1
  - updatenotification: 1.2.0
  - workflowengine: 1.2.0
Disabled:
  - admin_audit
  - encryption
  - user_external
  - user_ldap

Nextcloud configuration:

Config report
{
    "system": {
        "instanceid": "oc2fyztrch2c",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "domain.com"
        ],
        "datadirectory": "\/home\/andy\/oc-data",
        "overwrite.cli.url": "https:\/\/domain.com",
        "dbtype": "mysql",
        "version": "12.0.4.3",
        "dbname": "cloud_domain_com_1",
        "dbhost": "localhost",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "Europe\/Berlin",
        "installed": true,
        "knowledgebaseenabled": false,
        "enable_avatars": true,
        "mail_from_address": "cloud",
        "mail_smtpmode": "php",
        "mail_domain": "domain.com",
        "allow_user_to_change_display_name": true,
        "loglevel": 3,
        "maintenance": false,
        "theme": "",
        "singleuser": false,
        "appstore.experimental.enabled": true,
        "htaccess.RewriteBase": "\/",
        "updater.secret": "***REMOVED SENSITIVE VALUE***",
        "updater.release.channel": "stable",
        "filelocking.enabled": false,
        "redis": {
            "host": "localhost",
            "port": 6379
        },
        "memcache.local": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "auth.bruteforce.protection.enabled": true
    },
    "apps": {
        "activity": {
            "enabled": "yes",
            "installed_version": "2.5.2",
            "types": "filesystem"
        },
        "announcementcenter": {
            "enabled": "no",
            "installed_version": "1.1.1",
            "ocsid": "173921",
            "signed": "true",
            "types": ""
        },
        "backgroundjob": {
            "lastjob": "39"
        },
        "bruteforcesettings": {
            "enabled": "yes",
            "installed_version": "1.0.2",
            "types": ""
        },
        "calendar": {
            "enabled": "yes",
            "installed_version": "1.5.6",
            "signed": "true",
            "types": ""
        },
        "comments": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "logging"
        },
        "contacts": {
            "enabled": "yes",
            "installed_version": "2.0.1",
            "ocsid": "168708",
            "types": ""
        },
        "core": {
            "OC_Channel": "production",
            "backgroundjobs_mode": "cron",
            "incoming_server2server_share_enabled": "no",
            "installed.bundles": "[\"CoreBundle\"]",
            "installedat": "1459469183.8086",
            "lastcron": "1512672301",
            "lastupdateResult": "[]",
            "lastupdatedat": "1512672131",
            "moveavatarsdone": "yes",
            "oc.integritycheck.checker": "[]",
            "outgoing_server2server_share_enabled": "no",
            "previewsCleanedUp": "1",
            "public_files": "files_sharing\/public.php",
            "public_webdav": "dav\/appinfo\/v1\/publicwebdav.php",
            "remote_caldav": "dav\/appinfo\/v1\/caldav.php",
            "remote_calendar": "dav\/appinfo\/v1\/caldav.php",
            "remote_carddav": "dav\/appinfo\/v1\/carddav.php",
            "remote_contacts": "dav\/appinfo\/v1\/carddav.php",
            "remote_dav": "dav\/appinfo\/v2\/remote.php",
            "remote_files": "dav\/appinfo\/v1\/webdav.php",
            "remote_webdav": "dav\/appinfo\/v1\/webdav.php",
            "repairlegacystoragesdone": "yes",
            "scss.variables": "28b56eb75bff149ec8df6d6bdbcaacfa",
            "shareapi_allow_mail_notification": "yes",
            "shareapi_allow_public_notification": "yes",
            "shareapi_default_expire_date": "yes",
            "shareapi_expire_after_n_days": "30",
            "umgmt_send_email": "false",
            "umgmt_show_backend": "false",
            "umgmt_show_email": "false",
            "umgmt_show_last_login": "true",
            "umgmt_show_storage_location": "true",
            "updater.secret.created": "1506461277",
            "vendor": "nextcloud"
        },
        "dav": {
            "OCA\\DAV\\Migration\\ValueFixInsert_ran": "true",
            "buildCalendarSearchIndex": "yes",
            "enabled": "yes",
            "installed_version": "1.3.0",
            "types": "filesystem"
        },
        "federatedfilesharing": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": ""
        },
        "federation": {
            "autoAddServers": "0",
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "authentication"
        },
        "files": {
            "cronjob_scan_files": "500",
            "enabled": "yes",
            "installed_version": "1.7.2",
            "types": "filesystem"
        },
        "files_antivirus": {
            "av_chunk_size": "1024",
            "av_cmd_options": "",
            "av_host": "",
            "av_infected_action": "only_log",
            "av_mode": "socket",
            "av_path": "\/usr\/bin\/clamscan",
            "av_port": "0",
            "av_socket": "\/var\/run\/clamav\/clamd.ctl",
            "enabled": "no",
            "installed_version": "0.8.0.2",
            "ocsid": "157439",
            "signed": "true",
            "types": "filesystem"
        },
        "files_external": {
            "enabled": "yes",
            "installed_version": "1.3.0",
            "ocsid": "166048",
            "types": "filesystem"
        },
        "files_pdfviewer": {
            "enabled": "yes",
            "installed_version": "1.1.1",
            "ocsid": "166049",
            "types": ""
        },
        "files_sharing": {
            "enabled": "yes",
            "incoming_server2server_share_enabled": "no",
            "installed_version": "1.4.0",
            "outgoing_server2server_share_enabled": "no",
            "types": "filesystem"
        },
        "files_texteditor": {
            "enabled": "yes",
            "installed_version": "2.4.1",
            "ocsid": "166051",
            "types": ""
        },
        "files_trashbin": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "filesystem"
        },
        "files_versions": {
            "enabled": "yes",
            "installed_version": "1.5.0",
            "types": "filesystem"
        },
        "files_videoplayer": {
            "enabled": "yes",
            "installed_version": "1.1.0",
            "types": ""
        },
        "firstrunwizard": {
            "enabled": "yes",
            "installed_version": "2.1",
            "types": "logging"
        },
        "gallery": {
            "enabled": "yes",
            "installed_version": "17.0.0",
            "types": ""
        },
        "galleryplus": {
            "enabled": "no",
            "installed_version": "15.1.1",
            "ocsid": "169116",
            "signed": "true",
            "types": ""
        },
        "logreader": {
            "enabled": "yes",
            "installed_version": "2.0.0",
            "ocsid": "170871",
            "types": ""
        },
        "lookup_server_connector": {
            "enabled": "yes",
            "installed_version": "1.0.0",
            "types": "authentication"
        },
        "mail": {
            "enabled": "yes",
            "installed_version": "0.7.5",
            "signed": "true",
            "types": ""
        },
        "nextcloud_announcements": {
            "enabled": "yes",
            "installed_version": "1.1",
            "types": "logging"
        },
        "notifications": {
            "enabled": "yes",
            "installed_version": "2.0.0",
            "types": "logging"
        },
        "oauth2": {
            "enabled": "yes",
            "installed_version": "1.0.5",
            "types": "authentication"
        },
        "password_policy": {
            "enabled": "yes",
            "installed_version": "1.2.2",
            "types": ""
        },
        "provisioning_api": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "prevent_group_restriction"
        },
        "serverinfo": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": ""
        },
        "sharebymail": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "filesystem"
        },
        "spreedme": {
            "OWNCLOUD_TEMPORARY_PASSWORD_LOGIN_ENABLED": "true",
            "OWNCLOUD_TEMPORARY_PASSWORD_SIGNING_KEY": "",
            "SPREED_WEBRTC_BASEPATH": "\/",
            "SPREED_WEBRTC_IS_SHARED_INSTANCE": "false",
            "SPREED_WEBRTC_ORIGIN": "https:\/\/lm.domain.com:8082\/",
            "SPREED_WEBRTC_SHAREDSECRET": "",
            "enabled": "no",
            "installed_version": "0.3.5",
            "is_set_up": "true",
            "ocsid": "174436",
            "types": ""
        },
        "survey_client": {
            "apps": "no",
            "database": "no",
            "enabled": "yes",
            "encryption": "no",
            "files_sharing": "no",
            "installed_version": "1.0.0",
            "php": "no",
            "server": "no",
            "stats": "no",
            "types": ""
        },
        "systemtags": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "logging"
        },
        "templateeditor": {
            "enabled": "no",
            "installed_version": "0.2",
            "types": ""
        },
        "theming": {
            "cachebuster": "4",
            "color": "#1D2D44",
            "enabled": "yes",
            "installed_version": "1.3.0",
            "name": "1n3xCloud",
            "slogan": "nur wir sind cool!",
            "types": "logging",
            "url": "https:\/\/domain.com"
        },
        "twofactor_backupcodes": {
            "enabled": "yes",
            "installed_version": "1.1.1",
            "types": ""
        },
        "updatenotification": {
            "calendar": "1.4.1",
            "contacts": "1.5.1",
            "core": "9.1.2.2",
            "enabled": "[\"admin\"]",
            "installed_version": "1.2.0",
            "spreedme": "0.3.5",
            "types": ""
        },
        "workflowengine": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "filesystem"
        }
    }
}

Are you using external storage, if yes which one: local/smb/sftp/...
S3

Are you using encryption: yes/no
no

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
no

Client configuration

Browser:
any

Operating system:
any

Logs

Web server error log

Web server error log
No errors

Nextcloud log (data/nextcloud.log)

Nextcloud log
{"reqId":"XA4tA1FzLYXAm8STtEoS","level":3,"time":"2017-12-07T19:53:05+01:00","remoteAddr":"88.130.52.26","user":"a.martin","app":"PHP","method":"PROPFIND","url":"\/remote.php\/webdav\/Dokumente%20-%20S3","message":"Invalid argument supplied for foreach() at \/var\/www\/vhosts\/domain.com\/httpdocs\/apps\/files_external\/lib\/Lib\/Storage\/AmazonS3.php#291","userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/62.0.3202.94 Safari\/537.36","version":"12.0.4.3"}
{"reqId":"XA4tA1FzLYXAm8STtEoS","level":3,"time":"2017-12-07T19:53:05+01:00","remoteAddr":"88.130.52.26","user":"a.martin","app":"PHP","method":"PROPFIND","url":"\/remote.php\/webdav\/Dokumente%20-%20S3","message":"Invalid argument supplied for foreach() at \/var\/www\/vhosts\/domain.com\/httpdocs\/apps\/files_external\/lib\/Lib\/Storage\/AmazonS3.php#291","userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/62.0.3202.94 Safari\/537.36","version":"12.0.4.3"}
{"reqId":"XA4tA1FzLYXAm8STtEoS","level":3,"time":"2017-12-07T19:53:05+01:00","remoteAddr":"88.130.52.26","user":"a.martin","app":"PHP","method":"PROPFIND","url":"\/remote.php\/webdav\/Dokumente%20-%20S3","message":"Invalid argument supplied for foreach() at \/var\/www\/vhosts\/domain.com\/httpdocs\/apps\/files_external\/lib\/Lib\/Storage\/AmazonS3.php#291","userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/62.0.3202.94 Safari\/537.36","version":"12.0.4.3"}
{"reqId":"ivLnfMFaRFsTZlIbQMSJ","level":3,"time":"2017-12-07T19:53:06+01:00","remoteAddr":"88.130.52.26","user":"a.martin","app":"PHP","method":"GET","url":"\/index.php\/apps\/files\/ajax\/getstoragestats.php?dir=%2FDokumente%20-%20S3","message":"Invalid argument supplied for foreach() at \/var\/www\/vhosts\/domain.com\/httpdocs\/apps\/files_external\/lib\/Lib\/Storage\/AmazonS3.php#291","userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/62.0.3202.94 Safari\/537.36","version":"12.0.4.3"}

@MorrisJobke
Copy link
Member

cc @icewind1991

@MorrisJobke
Copy link
Member

S3 should be a lot smoother with 13. Could you test it again?

@In-eX
Copy link
Author

In-eX commented Dec 20, 2017

After upgrading from 12.0.4 to 13beta3 the problem persisted. A installation with the web installer (12.0.3) fixes the problem, but when I upgrade to 12.0.4 I'm back to my old problem. Note, that I'm using the same database for all versions (not at the same time ofc). I don't have any issues with 13beta3 and a new database.

Is there something wrong with the updater? Or did something get lost after all those small updates I did in the past?

@lesaff
Copy link

lesaff commented Jan 10, 2018

I am running into the same problem, running 13 b4

Error

Invalid argument supplied for foreach() at /###REDACTED###/apps/files_external/lib/Lib/Storage/AmazonS3.php#290

@MickVanDuijn
Copy link

After trying some stuff out, I noticed that when the root of the S3 bucket only contains folders, then the "Contents" attribute in the result paginator is not present. It should therefore be checked whether the "Contents" attribute is an array before iterating it with foreach similar to the CommonPrefixes.

@TecJon @lesaff For a quick workaround, you can add a dummy file .dummy with conents placeholder to the root of the S3 bucket.

@MickVanDuijn
Copy link

MickVanDuijn commented Feb 17, 2018

Sorry, I just noticed it still does not always show the correct size (it did at least once), but at least the logs don't overflow with Invalid argument supplied for foreach()

Edit: it sometimes shows the correct size on the shared folder, but for folders inside it is still pending.

@MorrisJobke
Copy link
Member

cc @icewind1991

@TecJon
Copy link

TecJon commented Feb 17, 2018

@MickVanDuijn thank you, unfortunately it didn't change for me.

This issue might only be a side issue to a bigger problem with s3 that I am facing. #8299

@k0ste
Copy link

k0ste commented Mar 31, 2018

Issue is still persist with nextCloud 13.0.1.

@jaakj
Copy link

jaakj commented Apr 29, 2018

Same problem here: NC 13.0.2, external storage is DigitalOcean's Spaces. All external folders showing "Pending" and log is flooded with error:

Invalid argument supplied for foreach() at /......./apps/files_external/lib/Lib/Storage/AmazonS3.php#295

@MorrisJobke
Copy link
Member

Same problem here: NC 13.0.2, external storage is DigitalOcean's Spaces. All external folders showing "Pending" and log is flooded with error:

Invalid argument supplied for foreach() at /......./apps/files_external/lib/Lib/Storage/AmazonS3.php#295

See #8842 and #9403

@k0ste
Copy link

k0ste commented Jun 15, 2018

I tested 13.0.4. Still in pending state.

selection_001

@nextcloud-bot nextcloud-bot added the stale Ticket or PR with no recent activity label Jul 16, 2018
@patricksebastien
Copy link

Same issue on 13.0.4 using snap installation on Ubuntu server.

@nextcloud-bot nextcloud-bot removed the stale Ticket or PR with no recent activity label Jul 30, 2018
@TecJon
Copy link

TecJon commented Jul 30, 2018

Same for 14.0.0.13 for an empty folder

@y0hnn
Copy link

y0hnn commented Jan 6, 2019

It seems with a GCP bucket with around 14000 objects, Nextcloud is stuck and can't access the files. Maybe a UTF8 problem?

@michael-fritzsch
Copy link

I applied the following patch to fix a bug, where root-level folders have not been identified in case they have been directly uploaded to S3 storage instead of upload via Nextcloud. This also fixed the "pending" size issue in Nextcloud 15.0.5.

https://github.com/nextcloud/server/compare/bugfix/6954/scan-external-s3

@MorrisJobke
Copy link
Member

cc @kesselb

@kesselb
Copy link
Contributor

kesselb commented Mar 19, 2019

Thanks for your feedback @mfridge

There are a few positive reports for this patch. I'll try to do some more testing and get it into 16.

@MorrisJobke
Copy link
Member

There are a few positive reports for this patch. I'll try to do some more testing and get it into 16.

Is there some PR for that already, because the beta is planned for tomorrow ;)

@cerebellum92
Copy link

Hi, I just want to report the same issue of pending size for S3 external storage.
This bug also makes tones of PUT (400,000 requests per day) requests for even only one user.
The patch @mfridge provided solved this issue for me.

@crypdick
Copy link

crypdick commented May 1, 2019

How can I apply this patch?

@cerebellum92
Copy link

How can I apply this patch?

@mfridge provided the link.
You just edit that file, delete (or command) the line with "-" sign in front of it, and add the line with "+" in front of it.

@skjnldsv skjnldsv added the 0. Needs triage Pending check for reproducibility or if it fits our roadmap label Jun 12, 2019
@cinghaman
Copy link

I am using Nextcloud on Hostio.cloud and added S3 as external storage and still see the Pending on the folders

@skjnldsv skjnldsv added 1. to develop Accepted and waiting to be taken care of and removed 0. Needs triage Pending check for reproducibility or if it fits our roadmap labels Aug 20, 2020
@CSchmidtDD
Copy link

When adding S3 via external mapping, the pending state is constant with our version of NC 20 as well.

@grk-
Copy link

grk- commented Apr 14, 2021

I applied the following patch to fix a bug, where root-level folders have not been identified in case they have been directly uploaded to S3 storage instead of upload via Nextcloud. This also fixed the "pending" size issue in Nextcloud 15.0.5.

https://github.com/nextcloud/server/compare/bugfix/6954/scan-external-s3

I'm also affected by this annoying bug.
Sadly this fix proposed by @mfridge is no longer available on github. Anyone kept it somewhere so we can fix this too?

@michael-fritzsch
Copy link

Hi @grk-, I think it's this commit: e387189

I don't use S3 storage anymore, as it turned out to be not reliable with Nextcloud in my case.

@santosh-goundar
Copy link

along with folder size, even it doesn't show correct modified time also ,please suggest

@sladg
Copy link

sladg commented Dec 7, 2021

along with folder size, even it doesn't show correct modified time also ,please suggest

Facing same issue.
image
ownCloud 10.8.0 (stable)

@PVince81
Copy link
Member

there were some issues with background scanning with S3 external storage recently.

please try again with recent versions
if the problem persists, feel free to reopen and specify your exact steps and version

@Desh-Deepak-Dhobi
Copy link

The same issue occurred at my end. And this was solved by setting up the Cron option from the Basic Settings section. The following are the steps:

  1. First you need to enable the Cron option from the Basic Settings section. Let’s open NextCloud as Admin user, go to SettingsBasic settings and select Cron.

  2. You must set up the following cron job to call the cron.php script. For Nextcloud to run smoothly, you will want to set up a cronjob. This is a task that is executed automatically in the background. Modify your Apache/Nginx cronjob:

sudo crontab -u www-data -e

If asked, press “1” to use the nano editor (which is super easy to use) and add the following line to your crontab file:

*/5  *  *  *  * php -f /var/www/nextcloud/cron.php

You have to replace the path /var/www/nextcloud/cron.php with the path to your current Nextcloud installation.

You can verify if the cron job has been added and scheduled by executing:

crontab -u www-data -l

Which returns:

*/5  *  *  *  * php -f /var/www/nextcloud/cron.php

This Cron method enables the execution of scheduled jobs without the inherent limitations the Web server might have.

[https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/background_jobs_configuration.html]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
1. to develop Accepted and waiting to be taken care of bug feature: external storage
Projects
None yet
Development

No branches or pull requests