[Bug]: occ files:scan stops indexing after encountering a folder without permissions, leaving the rest of the directory unindexed

[wonx]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

This is a follow-up of Bug #8576

When running occ files:scan (either manually or from a nightly cron job) on a directory stored in an external SMB storage, if the scan encounters a subfolder where the Nextcloud storage user has no read permissions, the scan process stops entirely at that point.

In the logs it appears as this line:
User <username> still has unscanned files after running background scan, background scan might be stopped prematurely

(possibly related bug: #43557)

When manually running a occ files:scan, the moment it reaches the folder without proper permissions it gets stuck at this message and doesn't continue:
Error during scan: Couldn't open SMB directory smb://192.168.4.145/Personal/Fotos/M%C3%B2bil%20%28sincronitzades%29/hola2: Path does not exist

(that specific folder was created manually by another user and couldn't be read due to lack of permissions)

As a result, all subsequent subfolders are not indexed, meaning that from the Nextcloud server’s perspective, they "do not exist" in the user’s files.

In practice, this leads to the desktop client believing that a large number of files have been deleted from the server, which then causes it to re-download or re-upload gigabytes of data repeatedly, despite the files being unchanged on the storage backend.

This behavior happened repeatedly every day after a nightly occ files:scan, and only stopped once permissions were corrected on the problematic folder.

Steps to reproduce

1. Configure an external SMB storage for a Nextcloud user.
2. In that storage, create a folder with no read permissions for the user that Nextcloud uses to access SMB.
3. Run occ files:scan --path="user/files/path/to/storage"
4. Observe that the scan stops at the problematic folder and never processes subsequent folders.
5. Check the web interface or WebDAV listing, all folders after the problematic one are missing from the server index.

Expected behavior

occ files:scan should log an error for the folder it cannot access, skip it, and continue scanning the remaining folders.
It should never leave the rest of the directory unindexed because of a single permissions issue.

Nextcloud Server version

31

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.4

Web server

Nginx

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "memcache.local": "\\OC\\Memcache\\APCu",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "dbtype": "mysql",
        "version": "31.0.8.1",
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "share_folder": "\/Shared",
        "maintenance": false,
        "filelocking.enabled": true,
        "theme": "",
        "loglevel": 1,
        "preview_max_x": 2048,
        "preview_max_y": 2048,
        "jpeg_quality": 60,
        "default_phone_region": "ES",
        "enabledPreviewProviders": [
            "OC\\Preview\\Illustrator",
            "OC\\Preview\\HEIC",
            "OC\\Preview\\Movie",
            "OC\\Preview\\MSOffice2003",
            "OC\\Preview\\MSOffice2007",
            "OC\\Preview\\MSOfficeDoc",
            "OC\\Preview\\PDF",
            "OC\\Preview\\PNG",
            "OC\\Preview\\JPEG",
            "OC\\Preview\\GIF",
            "OC\\Preview\\BMP",
            "OC\\Preview\\XBitmap",
            "OC\\Preview\\MP3",
            "OC\\Preview\\TXT",
            "OC\\Preview\\MarkDown",
            "OC\\Preview\\OpenDocument",
            "OC\\Preview\\Krita"
        ],
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_sendmailmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauth": 1,
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpsecure": "tls",
        "app_install_overwrite": [
            "files_excludedirs",
            "memories",
            "metadata",
            "contacts",
            "files_rightclick"
        ],
        "memories.exiftool": "\/config\/www\/nextcloud\/apps\/memories\/bin-ext\/exiftool-amd64-musl",
        "memories.vod.ffmpeg": "\/usr\/bin\/ffmpeg",
        "memories.vod.ffprobe": "\/usr\/bin\/ffprobe",
        "memories.vod.path": "\/config\/www\/nextcloud\/apps\/memories\/bin-ext\/go-vod-amd64",
        "memories.index.mode": "0",
        "twofactor_enforced": "false",
        "twofactor_enforced_groups": [],
        "twofactor_enforced_excluded_groups": [],
        "versions_retention_obligation": "auto, 30",
        "memories.gis_type": 1,
        "memories.db.triggers.fcu": true,
        "memcache.locking": "\\OC\\Memcache\\APCu",
        "upgrade.disable-web": true,
        "maintenance_window_start": 6,
        "bulkupload.enabled": false
    }
}

List of activated Apps

Enabled:
  - activity: 4.0.0
  - admin_audit: 1.21.0
  - app_api: 5.0.2
  - bookmarks: 15.1.3
  - bruteforcesettings: 4.0.0
  - circles: 31.0.0
  - cloud_federation_api: 1.14.0
  - comments: 1.21.0
  - contactsinteraction: 1.12.0
  - dashboard: 7.11.0
  - dav: 1.33.0
  - federatedfilesharing: 1.21.0
  - federation: 1.21.0
  - files: 2.3.1
  - files_downloadlimit: 4.0.0
  - files_excludedirs: 1.0.0-beta
  - files_external: 1.23.0
  - files_pdfviewer: 4.0.0
  - files_reminders: 1.4.0
  - files_sharing: 1.23.1
  - files_trashbin: 1.21.0
  - files_versions: 1.24.0
  - firstrunwizard: 4.0.0
  - logreader: 4.0.0
  - lookup_server_connector: 1.19.0
  - memories: 7.6.1
  - metadata: 0.22.0
  - nextcloud_announcements: 3.0.0
  - notifications: 4.0.0
  - oauth2: 1.19.1
  - password_policy: 3.0.0
  - previewgenerator: 5.9.0
  - privacy: 3.0.0
  - profile: 1.0.0
  - provisioning_api: 1.21.0
  - recognize: 9.0.3
  - related_resources: 2.0.0
  - serverinfo: 3.0.0
  - settings: 1.14.0
  - sharebymail: 1.21.0
  - support: 3.0.0
  - survey_client: 3.0.0
  - systemtags: 1.21.1
  - text: 5.0.0
  - theming: 2.6.1
  - twofactor_backupcodes: 1.20.0
  - twofactor_totp: 13.0.0-dev.0
  - updatenotification: 1.21.0
  - user_status: 1.11.0
  - viewer: 4.0.0
  - weather_status: 1.11.0
  - webhook_listeners: 1.2.0
  - workflowengine: 2.13.0
Disabled:
  - calendar: 4.4.4 (installed 4.4.4)
  - contacts: 5.3.2 (installed 5.3.2)
  - encryption: 2.19.0
  - files_inotify: 0.2.3
  - files_rightclick: 0.15.1 (installed 0.15.1)
  - mail: 3.3.1 (installed 3.3.1)
  - maps: 1.1.0 (installed 1.1.0)
  - photos: 4.0.0 (installed 2.0.1)
  - recommendations: 4.0.0 (installed 3.0.0)
  - spreed: 17.0.3 (installed 17.0.3)
  - suspicious_login: 9.0.1
  - twofactor_nextcloud_notification: 5.0.0
  - user_ldap: 1.22.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

Error during scan: Couldn't open SMB directory smb://<path-to-folder>: Path does not exist

Additional info

This behavior caused over 500 GB of unnecessary data transfers over several days, because the desktop client would repeatedly try to “resync” the missing files.
It would be beneficial if occ files:scan had an option to skip unreadable folders and proceed with the rest of the scan instead of stopping completely.

[undefinedid]

same behavior here.
my docker stack is fully automated and since the latest update with the tag nextcloud:fpm, the WebUI doesn't list anymore some files and folders that are present on the filesystem.

version is actually 31.0.8.1, and no rollback possible on the previous image tag 31.0.7.1.

my clients keeps trying to synchronize with unnecessary data transfers too.

occ files:scan doesn't fix anything.
it scans and lists objects scanned, but files and folders are still missing in the WebUI or for clients.

edits: typo & version

[undefinedid]

I can confirm the problem only concerns the latest version aka 31.0.8.1.

I managed to rollback my DB (pgsql) and the web server volume /var/www/html to nextcloud:31.0.7-fpm and everything is back to normal.

I just had to run a simple www-data@nextcloud-fpm:~/html$ php occ files:scan --all to update the few files I added since.