Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: APCu support gone in 27.0.0 #38796

Closed
5 of 8 tasks
heeplr opened this issue Jun 13, 2023 · 15 comments
Closed
5 of 8 tasks

[Bug]: APCu support gone in 27.0.0 #38796

heeplr opened this issue Jun 13, 2023 · 15 comments
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap 27-feedback bug feature: caching Related to our caching system: scssCacher, jsCombiner... feature: locking

Comments

@heeplr
Copy link

heeplr commented Jun 13, 2023

⚠️ This issue respects the following points: ⚠️

Bug description

After upgrading to 27.0.0, administration panel gives a warning, that the database is used for locking transaction files and that I should setup memcache/redis.

I already got APCu configured in config.php:

'filelocking.enabled' => true,
'memcache.local' => '\\OC\\Memcache\\APCu',

and it seems to work for other instances running 26.0.2 on the same host/php-fpm.

Steps to reproduce

  1. Run 26.0.2 with 'memcache.local' => '\OC\Memcache\APCu' working
  2. Upgrade to 27.0.0
  3. See warning in Administration settings overview

Expected behavior

No warning / working APCu memcache

Installation method

Community Web installer on a VPS or web space

Nextcloud Server version

27

Operating system

Other

PHP engine version

PHP 8.1

Web server

Nginx

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

Upgraded to a MAJOR version (ex. 22 to 23)

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

  • Default user-backend (database)
  • LDAP/ Active Directory
  • SSO - SAML
  • Other

Configuration report

"system": {                                                                                                                                                                                                
        "instanceid": "***REMOVED SENSITIVE VALUE***",                                                                                                                                                         
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",                                                                                                                                                       
        "secret": "***REMOVED SENSITIVE VALUE***",                                                                                                                                                             
        "trusted_domains": [                                                                                                                                                                                   
            "***REMOVED SENSITIVE VALUE***"                                                                                                                                                                                 
        ],                                                                                                                                                                                                     
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",                                                                                                                                                    
        "overwritecondaddr": "^192\\.168\\.101\\.1$",                                                                                                                                                          
        "datadirectory": "***REMOVED SENSITIVE VALUE***",                                                                                                                                                      
        "dbtype": "pgsql",                                                                                                                                                                                     
        "version": "27.0.0.8",
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "default_phone_region": "DE",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "sendmail",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "25",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "default_language": "de_DE",
        "filelocking.enabled": true,
        "maintenance": false,
        "theme": "",
        "loglevel": 0,
        "updater.release.channel": "stable",
        "appstoreenabled": true,
        "appstoreurl": "https:\/\/apps.nextcloud.com\/api\/v1",
        "app_install_overwrite": [
            "calendar",
            "news",
            "deck"
        ],
        "mail_sendmailmode": "pipe",
        "connectivity_check_domains": [
            "https:\/\/www.nextcloud.com",
            "https:\/\/www.startpage.com",
            "https:\/\/www.eff.org",
            "https:\/\/www.edri.org"
        ],
        "updater.secret": "***REMOVED SENSITIVE VALUE***"
    }
}

List of activated Apps

Enabled:                                                                                                                                                                                                       
  - activity: 2.19.0                                                                                                                                                                                           
  - bookmarks: 13.0.1                                                                                                                                                                                          
  - calendar: 4.4.2                                                                                                                                                                                            
  - cloud_federation_api: 1.10.0                                                                                                                                                                               
  - contacts: 5.3.0                                                                                                                                                                                            
  - contactsinteraction: 1.8.0                                                                                                                                                                                 
  - dav: 1.27.0                                                                                                                                                                                                
  - deck: 1.10.0                                                                                                                                                                                               
  - federatedfilesharing: 1.17.0                                                                                                                                                                               
  - files: 1.22.0                                                                                                                                                                                              
  - files_external: 1.19.0                                                                                                                                                                                     
  - files_pdfviewer: 2.8.0                                                                                                                                                                                     
  - files_rightclick: 1.6.0                                                                                                                                                                                    
  - files_sharing: 1.19.0                                                                                                                                                                                      
  - files_trashbin: 1.17.0                                                                                                                                                                                     
  - files_versions: 1.20.0                                                                                                                                                                                     
  - logreader: 2.12.0                                                                                                                                                                                          
  - lookup_server_connector: 1.15.0                                                                                                                                                                            
  - news: 21.2.0                                                                                                                                                                                               
  - nextcloud_announcements: 1.16.0
  - notes: 4.8.0
  - notifications: 2.15.0
  - oauth2: 1.15.0
  - photos: 2.3.0
  - privacy: 1.11.0
  - provisioning_api: 1.17.0
  - recommendations: 1.6.0
  - related_resources: 1.2.0
  - settings: 1.9.0
  - sharebymail: 1.17.0                                                                                                                                                                                [4/1906]
  - text: 3.8.0
  - theming: 2.2.0
  - twofactor_backupcodes: 1.16.0
  - twofactor_email: 2.7.3
  - twofactor_totp: 9.0.0
  - twofactor_webauthn: 1.2.0
  - updatenotification: 1.17.0
  - user_status: 1.7.0
  - viewer: 2.1.0
  - weather_status: 1.7.0
  - workflowengine: 2.9.0
Disabled:
  - admin_audit: 1.17.0
  - audioplayer: 3.3.1 (installed 3.3.1)
  - audioplayer_editor: 0.3.0 (installed 0.3.0)
  - bruteforcesettings: 2.7.0
  - circles: 27.0.0 (installed 25.0.0)
  - comments: 1.17.0 (installed 1.4.0)
  - dashboard: 7.7.0 (installed 7.0.0)
  - encryption: 2.15.0
  - federation: 1.17.0 (installed 1.6.0)
  - files_downloadactivity: 1.16.0 (installed 1.16.0)
  - files_mindmap: 0.0.28 (installed 0.0.28)
  - firstrunwizard: 2.16.0 (installed 2.3.0)
  - forms: 3.3.0 (installed 3.3.0)
  - password_policy: 1.17.0 (installed 1.6.0)
  - polls: 5.0.5 (installed 5.0.5)
  - serverinfo: 1.17.0 (installed 1.4.0)
  - support: 1.10.0 (installed 1.0.0)
  - survey_client: 1.15.0 (installed 1.2.0)
  - suspicious_login: 5.0.0
  - systemtags: 1.17.0 (installed 1.4.0)
  - user_ldap: 1.17.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

{"reqId":"TEPKEE0BvBCynO9vudLZ","level":0,"time":"2023-06-13T16:12:30+00:00","remoteAddr":"192.168.0.107","user":"username","app":"news","method":"GET","url":"/ocs/v2.php/search/providers?from=%2Fsettings%2Fadmin%2Foverview","message":"new parser added : FeedIo\\Standard\\Json","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","version":"27.0.0.8","data":{"app":"news"}}
{"reqId":"TEPKEE0BvBCynO9vudLZ","level":0,"time":"2023-06-13T16:12:30+00:00","remoteAddr":"192.168.0.107","user":"username","app":"news","method":"GET","url":"/ocs/v2.php/search/providers?from=%2Fsettings%2Fadmin%2Foverview","message":"new parser added : FeedIo\\Standard\\Atom","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","version":"27.0.0.8","data":{"app":"news"}}
{"reqId":"TEPKEE0BvBCynO9vudLZ","level":0,"time":"2023-06-13T16:12:30+00:00","remoteAddr":"192.168.0.107","user":"username","app":"news","method":"GET","url":"/ocs/v2.php/search/providers?from=%2Fsettings%2Fadmin%2Foverview","message":"new parser added : FeedIo\\Standard\\Rss","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","version":"27.0.0.8","data":{"app":"news"}}
{"reqId":"TEPKEE0BvBCynO9vudLZ","level":0,"time":"2023-06-13T16:12:30+00:00","remoteAddr":"192.168.0.107","user":"username","app":"news","method":"GET","url":"/ocs/v2.php/search/providers?from=%2Fsettings%2Fadmin%2Foverview","message":"new parser added : FeedIo\\Standard\\Rdf","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","version":"27.0.0.8","data":{"app":"news"}}
{"reqId":"rLX3bkmETYmc24FAULN9","level":0,"time":"2023-06-13T16:12:30+00:00","remoteAddr":"192.168.0.107","user":"username","app":"no app in context","method":"GET","url":"/index.php/.well-known/nodeinfo","message":"2 well known handlers registered","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","version":"27.0.0.8","data":[]}
{"reqId":"ckRSdieLsqMQijVmC7Pj","level":0,"time":"2023-06-13T16:12:30+00:00","remoteAddr":"192.168.0.107","user":"username","app":"no app in context","method":"GET","url":"/index.php/.well-known/webfinger","message":"2 well known handlers registered","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","version":"27.0.0.8","data":[]}
{"reqId":"74jVMwVN3Y3BYcfnv4f7","level":0,"time":"2023-06-13T16:12:31+00:00","remoteAddr":"192.168.0.107","user":"username","app":"no app in context","method":"GET","url":"/settings/ajax/checksetup","message":"Deprecated event type for OCP\\IDBConnection::CHECK_MISSING_PRIMARY_KEYS: Symfony\\Component\\EventDispatcher\\GenericEvent is used","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","version":"27.0.0.8","data":[]}
{"reqId":"74jVMwVN3Y3BYcfnv4f7","level":0,"time":"2023-06-13T16:12:32+00:00","remoteAddr":"192.168.0.107","user":"username","app":"no app in context","method":"GET","url":"/settings/ajax/checksetup","message":"Deprecated event type for OCP\\IDBConnection::CHECK_MISSING_INDEXES: Symfony\\Component\\EventDispatcher\\GenericEvent is used","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","version":"27.0.0.8","data":[]}
{"reqId":"74jVMwVN3Y3BYcfnv4f7","level":0,"time":"2023-06-13T16:12:33+00:00","remoteAddr":"192.168.0.107","user":"username","app":"no app in context","method":"GET","url":"/settings/ajax/checksetup","message":"Deprecated event type for OCP\\IDBConnection::CHECK_MISSING_COLUMNS: Symfony\\Component\\EventDispatcher\\GenericEvent is used","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0","version":"27.0.0.8","data":[]}

Additional info

No response

@heeplr heeplr added 0. Needs triage Pending check for reproducibility or if it fits our roadmap bug labels Jun 13, 2023
@kesselb
Copy link
Contributor

kesselb commented Jun 13, 2023

that the database is used for locking transaction files

The ACPu support is not gone (unless you uninstalled the php extension).

The warning is new in Nextcloud 27 to let you know that without configuration, the database locking provider is active.

memcache.local or memcache.distributed is not the default for memcache.locking.
I assume we could merge distributed and locking nowdays.

/**
* Memory caching backend for file locking
*
* Because most memcache backends can clean values without warning using redis
* is highly recommended to *avoid data loss*.
*
* Defaults to ``none``
*/
'memcache.locking' => '\\OC\\Memcache\\Redis',

@heeplr
Copy link
Author

heeplr commented Jun 13, 2023

ah, thank you.

I wanted to avoid Redis (it's a very small instance) and get rid of the warning.

I tried 'memcache.locking' => false, and the warning is gone. Will that impose hazards? If not, the issue is solved for me and this could be closed.

It'd be awesome, if one could use the filesystem (with tmpfs in RAM) for that, so no redis install is needed - but I guess that'd be a feature request.

@kesselb
Copy link
Contributor

kesselb commented Jun 13, 2023

Will that impose hazards?

false => NullCache. The locking provider stores the data to the null cache instance. I guess that's filelocking.enabled => false without a warning. Your decision if you want to run such a setup.

It'd be awesome, if one could use the filesystem (with tmpfs in RAM) for that, so no redis install is needed - but I guess that'd be a feature request.

We will not implement a cache adapter using a local or remote filesystem.
Redis is a good and reliable solution. I would even say the best option available today.

@heeplr
Copy link
Author

heeplr commented Jun 13, 2023

I guess that's filelocking.enabled => false without a warning. Your decision if you want to run such a setup.

So it sounds like it is a hazard for data corruption and not just a performance issue. The silenced warning might be a separate issue, then.

Redis is a good and reliable solution. I would even say the best option available today.

Except that it has a track record of security issues and adds overall complexity, thus attack surface. But I guess the devs thought about that.

If I'm getting the setup warning right, redis isn't optional anymore as of 27.x, which is why you might want to update the prerequesites in the documentation where redis is mentioned as optional.

@kesselb
Copy link
Contributor

kesselb commented Jun 13, 2023

If I'm getting the setup warning right, redis isn't optional anymore as of 27.x,

The warning is new. That's the only difference.

A contributor with a strong interest in performance optimizations brought to our attention that database locking is slow, and therefore we implemented a warning to inform the administrator about it: #37758

Database locking is also slow for Nextcloud 26, 25, 24, 23, etc.
If this was okay for you so far, then you can simply continue to use database locking.

@heeplr
Copy link
Author

heeplr commented Jun 13, 2023

If this was okay for you so far, then you can simply continue to use database locking.

The performance is ok for me, the warning is not.

Honestly I don't quite get why filelocking isn't using the already configured memcache adapter. It seems to use exactly the same API (?).
I've tried 'memcache.locking' => '\\OC\Memcache\\APCu', now and it throws no error or warning.

Is that imposing a risk for data corruption?

@daffydock
Copy link

Wondering the same thing.

I have a test instance and after adding @heeplr 's memcache.locking suggestion, I too no longer get the error.

@kesselb
Copy link
Contributor

kesselb commented Jun 15, 2023

Honestly I don't quite get why filelocking isn't using the already configured memcache adapter. It seems to use exactly the same API (?).

Well observed. local, distributed and locking all require a IMemcache instance. For a single server setup, it's common to configure local and distributed to use the same adapter. If distributed is undefined, local is used as fallback. A setup with multiple nodes often uses memcache or redis to share the distributed cache between the nodes.

How is locking different to local and distributed? For cache, it's usually okay if something is not there. Do you think that would work for your locks? ;)

I've tried 'memcache.locking' => '\OC\Memcache\APCu', now and it throws no error or warning

Keep in mind that APCu only works for web request.
That means occ or cron.php (if called from system cron) cannot access the locks.

Is that imposing a risk for data corruption?

Sorry, I don't know.

@Lawkss
Copy link

Lawkss commented Jun 19, 2023

I get this warning despite using apcu.

The official documentation states that this is ok for small single server.

How to disable the warning? What is the correct procedure now?

I want the green tick again and not an error/warning message for something that is optional.

This isnt like not wearing a seatbelt warning. This is a performance optimization message. There should be a way to disregard it and have the green tick.

EDIT: there a good and simple instructions to setup redis here:

https://help.nextcloud.com/t/warning-the-database-is-used-for-transactional-file-locking/164095/3

@jl-678
Copy link

jl-678 commented Jun 20, 2023

+1

I understand and appreciate the need for this warning on larger instances. However, disabling it should be possible for those running smaller instances. I get that file locking provides a load on the database, but that is fine for a smaller homelab instance.

For me installing Redis feels like way overkill. I am happy with the performance now and just want the warning message to go away.

@Strit
Copy link

Strit commented Jul 16, 2023

Any update on this?

This silent change would likely have caused a lot, if not most, admins of small Nextcloud instances to "panic", since there was no mention of it.

@joshtrichards joshtrichards added the feature: caching Related to our caching system: scssCacher, jsCombiner... label Aug 27, 2023
@joshtrichards
Copy link
Member

@Lawkss / @jl-678 / @Strit -

It's not an error nor warning. It's an informational level message on the setup checks page. It doesn't appear anywhere else. If you know you're on an instance where you're comfortable with db level locking you can ignore it.

For the record, Redis is simpler to install and activate than both your NC web and your db were (and requires minimal resources). And there is zero upkeep. It just... Works.

I've interacted with multiple people on the forums that avoided activating Redis for similar reasons. This change nudged them to trying it. It really was no big deal.

If you have reasons not to do so or simply still don't want to run it, that's fine. Just adding that tidbit of experience in case it's helpful.

@Strit
Copy link

Strit commented Aug 27, 2023

It's not an error nor warning. It's an informational level message on the setup checks page. It doesn't appear anywhere else. If you know you're on an instance where you're comfortable with db level locking you can ignore it.

Sure. I get that. But that's not what it looks like in the interface. It looks like a warning. Yellow icon and all.

@heeplr
Copy link
Author

heeplr commented Aug 27, 2023

It's not an error nor warning.

It literally spells "warning". The DOM id of the corresponding icon div is "security-warning-state-warning".

For the record, Redis is simpler to install and activate than both your NC web and your db were (and requires minimal resources).

I'm pretty sure you're forgetting people who just get PHP+APCu from their managed host and can't convince their hoster, to add redis to the service.

This change is going to make life unnecessarily harder for self-hoster and people without access to redis on their managed host.
At best, it will create numbness against warnings. That's not what "nudging" means.

@heeplr heeplr closed this as completed Aug 27, 2023
@joshtrichards
Copy link
Member

joshtrichards commented Aug 27, 2023

Sure. I get that. But that's not what it looks like in the interface. It looks like a warning. Yellow icon and all.

It literally spells "warning". The DOM id of the corresponding icon div is "security-warning-state-warning".

Sounds like there is room for improvement in the UI. But behind the scenes it's definitely not tagged as an error or warning, just informational:

if(data.hasDBFileLocking) {
messages.push({
msg: t('core', 'The database is used for transactional file locking. To enhance performance, please configure memcache, if available. See the {linkstart}documentation ↗{linkend} for more information.')
.replace('{linkstart}', '<a target="_blank" rel="noreferrer noopener" class="external" href="' + OC.theme.docPlaceholderUrl.replace('PLACEHOLDER', 'admin-transactional-locking') + '">')
.replace('{linkend}', '</a>'),
type: OC.SetupChecks.MESSAGE_TYPE_INFO

I'm pretty sure you're forgetting people who just get PHP+APCu from their managed host and can't convince their hoster, to add redis to the service.

Good point.

This change is going to make life unnecessarily harder for self-hoster and people without access to redis on their managed host.
At best, it will create numbness against warnings. That's not what "nudging" means.

Yes that seems an undesirable side effect. Perhaps a config option similar to this is justifiable:

https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/config_sample_php_parameters.html#check-for-working-wellknown-setup

The code change would be straightforward if someone feels like testing and submitting a PR. The code change would be here:

protected function hasDBFileLocking(): bool {

And the code - if you wanted to call the new option check_db_filelocking would be something like:


protected function hasDBFileLocking(): bool {
    if ($this->config->getSystemValue('check_db_filelocking, true) === false) {
        return true;
    } 
    return ($this->lockingProvider instanceof DBLockingProvider);
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap 27-feedback bug feature: caching Related to our caching system: scssCacher, jsCombiner... feature: locking
Projects
None yet
Development

No branches or pull requests

7 participants