[Bug]: `occ dav:sync-system-addressbook` stops after 30000 entries

[Justinzobel]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

When running the above command, it stops at 30,000 entries.

Discussed here and suggested I open a report on GH. https://help.nextcloud.com/t/occ-dav-sync-system-addressbook-stops-at-30000/206828

Steps to reproduce

1. Have over 30,000 users
2. Run the command

Expected behavior

Process continues over all users.

Nextcloud Server version

29

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.1

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Upgraded to a MAJOR version (ex. 28 to 29)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
"system": {
"instanceid": "REMOVED SENSITIVE VALUE",
"passwordsalt": "REMOVED SENSITIVE VALUE",
"secret": "REMOVED SENSITIVE VALUE",
"trusted_domains": [
"REMOVED SENSITIVE VALUE",
],
"datadirectory": "REMOVED SENSITIVE VALUE",
"dbtype": "mysql",
"overwriteprotocol": "https",
"version": "29.0.8.1",
"overwrite.cli.url": "REMOVED SENSITIVE VALUE",
"htaccess.RewriteBase": "/",
"dbname": "REMOVED SENSITIVE VALUE",
"skeletondirectory": "",
"dbhost": "REMOVED SENSITIVE VALUE",
"dbport": "3306",
"dbtableprefix": "oc_",
"mysql.utf8mb4": true,
"dbuser": "REMOVED SENSITIVE VALUE",
"dbpassword": "REMOVED SENSITIVE VALUE",
"installed": true,
"memcache.local": "\OC\Memcache\APCu",
"memcache.locking": "\OC\Memcache\Redis",
"memcache.distributed": "\OC\Memcache\Redis",
"redis": {
"host": "REMOVED SENSITIVE VALUE",
"port": 6379,
"dbindex": 0,
"timeout": 1.5,
"read_timeout": 1.5
},
"check_for_working_wellknown_setup": true,
"check_for_working_htaccess": true,
"trashbin_retention_obligation": "auto, 1",
"ldapIgnoreNamingRules": false,
"ldapProviderFactory": "OCA\User_LDAP\LDAPProviderFactory",
"maintenance": false,
"maintenance_window_start": 5,
"integrity.check.disabled": false,
"theme": "",
"loglevel": 0,
"app_install_overwrite": [
"occweb",
"calendar"
],
"updater.release.channel": "stable",
"mail_smtpmode": "smtp",
"mail_smtpsecure": "tls",
"mail_sendmailmode": "smtp",
"mail_from_address": "REMOVED SENSITIVE VALUE",
"mail_domain": "REMOVED SENSITIVE VALUE",
"mail_smtpauthtype": "LOGIN",
"mail_smtpauth": 1,
"mail_smtphost": "REMOVED SENSITIVE VALUE",
"mail_smtpport": "587",
"mail_smtpname": "REMOVED SENSITIVE VALUE",
"mail_smtppassword": "REMOVED SENSITIVE VALUE"
}
}

List of activated Apps

Enabled:

• activity: 2.21.1
• admin_audit: 1.19.0
• calendar: 4.7.16
• cloud_federation_api: 1.12.0
• comments: 1.19.0
• contacts: 6.0.0
• contactsinteraction: 1.10.0
• dashboard: 7.9.0
• dav: 1.30.1
• federatedfilesharing: 1.19.0
• federation: 1.19.0
• files: 2.1.1
• files_downloadlimit: 2.0.0
• files_pdfviewer: 2.10.0
• files_reminders: 1.2.0
• files_sharing: 1.21.0
• files_trashbin: 1.19.0
• files_versions: 1.22.0
• firstrunwizard: 2.18.0
• gpoddersync: 3.10.0
• groupquota: 0.2.1
• logreader: 2.14.0
• lookup_server_connector: 1.17.0
• mail: 3.7.8
• maps: 1.4.0
• music: 2.0.1
• nextcloud_announcements: 1.18.0
• notes: 4.11.0
• notifications: 2.17.0
• oauth2: 1.17.1
• password_policy: 1.19.0
• photos: 2.5.0
• piwik: 0.13.0
• provisioning_api: 1.19.0
• recommendations: 2.1.0
• related_resources: 1.4.0
• serverinfo: 1.19.0
• settings: 1.12.0
• sharebymail: 1.19.0
• sociallogin: 5.6.6
• support: 1.12.0
• survey_client: 1.17.0
• systemtags: 1.19.0
• tasks: 0.16.1
• text: 3.10.1
• theming: 2.4.0
• twofactor_backupcodes: 1.18.0
• twofactor_totp: 11.0.0-dev
• updatenotification: 1.19.1
• user_ldap: 1.20.0
• user_status: 1.9.0
• viewer: 2.3.0
• weather_status: 1.9.0
• workflowengine: 2.11.0
  Disabled:
• bruteforcesettings: 2.9.0
• circles: 29.0.0-dev (installed 0.20.10)
• encryption: 2.17.0
• files_external: 1.21.0
• files_rightclick: 0.15.1 (installed 1.6.0)
• news: 24.0.0 (installed 24.0.0)
• privacy: 1.13.0 (installed 1.0.0)
• suspicious_login: 7.0.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

No response

Additional info

No response

[Justinzobel]

If I may suggest a new command occ config:bugreport that does all the above for the user, checks which webservers are installed on the system, which php versions, database servers, checks the signing status, etc and can be done with one command and one paste.

[szaimen]

If I may suggest a new command occ config:bugreport that does all the above for the user, checks which webservers are installed on the system, which php versions, database servers, checks the signing status, etc and can be done with one command and one paste.

You could try occ support:report

[Justinzobel]

If I may suggest a new command occ config:bugreport that does all the above for the user, checks which webservers are installed on the system, which php versions, database servers, checks the signing status, etc and can be done with one command and one paste.

You could try occ support:report

If this supplies all the required information for bug reports, that would be good to use in the GitHub issue template.

[kesselb]

cc @susnux @miaulalala @ChristophWurst because we were talking about it the other day.

[Justinzobel]

So I decided to leave this running overnight again after I did an update. Eventually it spits out this:

In Redis.php line 94:

  read error on connection to 127.0.0.1:6379


dav:sync-system-addressbook

So I guess it has to do with the connection to Redis timing out or hitting some sort of limit?

[ChristophWurst]

Looks like that is what's happening, yes

[Justinzobel]

I'm going to just try continuously run it and see if it gets through all the users with: while true;do sudo -u www-data php /var/www/nextcloud/occ dav:sync-system-addressbook;done