Session lock record with PHP 7.0.9

[CassianoCarraro]

I'm using a Memcached session handler normally in PHP 5.6. But with PHP 7 session don't retrieved properly and the follow warning message was triggered in PHP error log: PHP Warning: session_start(): Unable to clear session lock record.

Any suggestion?

[datasage]

Session settings have changed quite a bit for php 7 version of the extension, you will probably need to tweak your settings a bit to get it to behave as it did in PHP 5.6

[sodabrew]

Please state the version of php-memcached that you are using (or the git revision if you're building from source).

[CassianoCarraro]

I tried to use repository and build php7 source branch, but in both case i have the same problem. The versions is as follows:

From repository
Memcached version: 3.0.0-dev
libmemcached version: 1.0.16

From php7 branch:
Memcached version: 3.0.0b1
libmemcached version: 1.0.18

[beniamin]

I had same problem and I resolved the issue by disabling session.lazy_write in php.ini

[arisro]

Hey guys,

You can set memcached.sess_lock_expire to your max_execution_time value to avoid that bug from #266. This way the extension won't fall back to the max_execution_time value - which it reads incorrectly.

[CassianoCarraro]

I disabled session.lazy_write and set memcached.sess_lock_expire according my max_execution_time, but the problem persists. I found that this problem occurs when many requests is sent in a short time to session_start() with a same session id. This can be simulated with a code that call session_start() and then add a sleep executing this script many times will throw this error. I think that have a problem to unlock session or another configuration is causing the problem.

[arisro]

@CassianoCarraro

In the case you describe, the error is normal.
Example:
0. See this: https://bugs.php.net/bug.php?id=72345. The maximum execution time is not affected by system calls, stream operations etc. So the time you wait for session_start() doesn't add to the execution time.

1. You have max_execution_time = memcached.sess_lock_expire = 30s
2. You fire a request that keeps the session locked 3s let's say. If you fire 11 requests, they are serialized by the session lock. Your requests will go through 1 by 1.
3. The first request goes in immediately and locks the session for 3s. The seconds request waits 3s then he locks it for another 3s. The third request will wait in total 6s, then lock for 3s, and so on.
4. The 11th request has to wait 30s. Since session_start() doesn't add to the exection time, you won't get a max execution time error. You will get instead a warning, because of the session locking timeouts (which you can control with memcached.sess_lock_retries, memcached.sess_lock_wait_min/max).
   Once this timeout happens, your request will go through and be executed - ignoring the lock so to say.

So this is another problem, and is not actually an issue cause by this extension. The issue #266 consists in the session lock being permanently set (with 0 TTL) so you reach the timeout limit on any request - and the lock is not removed even after the timeout, when the request is processed.

[beniamin]

My problem is with parallel requests. If I have an ajax request running and I send another request, both requests are stopped and receive 500 with error from first post.
I tried to set max_execution_time = memcached.sess_lock_expire, but didn't solve my problem.
Only disabling session.lazy_write solved my problem.

[idmssvep]

I have the same issue, i run a lot of paralell requests and i have repeating "Unable to clear session lock record" fatals. I disabled lazy write and tried following memcached config:

memcached.sess_lock_expire 7200
memcached.sess_lock_max_wait not set
memcached.sess_lock_retries 5
memcached.sess_lock_wait not set
memcached.sess_lock_wait_max 120000
memcached.sess_lock_wait_min 0
memcached.sess_locking 1

with:
Version 3.0.0-dev
libmemcached version 1.0.16

With older versions on php 5.4 base i have absolutely no issues. It gives me a feeling that on PHP7 they are processed much faster and due to this fact create more blocking requests. One question: Is this issue fixed to PHP 7.0.9, so a up or downgrade could help?

[arielkung]

Confirming here in php 7.0.12 if you set a fixed value for memcached.sess_lock_expire the issue goes away.
Message "PHP Warning: session_start(): Unable to clear session lock record." will appear anyway if your session is locked more than the time you set of course.

[bzabos]

Please read up on how these values are consumed: https://github.com/php-memcached-dev/php-memcached/blob/master/memcached.ini

Increasing sess_lock_wait_max is only increasing the maximum seconds to which php will perform its exponential back off routine for retries. Increasing this value does nothing when sess_lock_wait_min is left untouched. If we want php to continue retrying at an acceptable interval, I don't believe these are the keys we should be adjusting. Lowering sess_lock_expire is just highjacking the session from other, long running, requests — is that the desired behaviour? Disabling lazy_write also seems like a bad idea.

Can anyone see why the solution wouldn't be to simply increase sess_lock_retries to a number that would result in an acceptable wait time for your specific scenario?

Eg. Where default sess_lock_wait_min=1000 and default sess_lock_wait_max=2000 and increased sess_lock_retries=10, our retry wait will accumulate as 1s+2s+2s+... Ns, giving us a total wait of 19 seconds of waiting and retrying for session lock to be released by other requests; logically, it makes sense to formulate rough equality with your max_execution_time.

[woolardfa]

For our Moodle site, we use memcached for PHP sessions. Because of the way Moodle uses the session handler in its AJAX calls (several issued before main page request finishes) we often get blocked session_start() calls. See this bug and this bug for details.

To get the new version to behave like the old one, we set the lock_wait_min and _max to the same reduced value of 150. This makes the spin lock retry 150 msec, as was the default in 2.2.x. Moodle config default is to wait a max of 120 secs to attain a session lock, so we set the retry count to 800 (120 / .15).

One or two seconds seems to be much to long to wait between attempts; much better to make several frequent attempts waiting (sleeping) only 150 msec at a time.

[sodabrew]

I've read through this ticket and I'm not sure what problem or change request is still outstanding.

[andrerom]

Well somwhow it seems session locking might be broken with php7, similar issue here.

[sodabrew]

@andrerom As I wrote, I don't fully understand the specific problem(s?) that people are having in this thread. If the problem(s?) is also affecting you, can you help to explain what would resolve it?

[woolardfa]

I believe the session locking problem isn't so much an issue with the php_memcached as it is with PHP (all versions). See https://bugs.php.net/bug.php?id=71962.

The messages having to do with failure to clear lock records come about because competing processes using the same session (i.e. main page request, and any number of AJAX calls associated it), are all working with the same memcached item--the first to finish removes the lock item, and the rest all report they were unable to do so simply because it wasn't there any longer.

[sodabrew]

I still don't understand if there is something that needs to be fixed or changed or better documented.

@woolardfa The bug you linked is specific to the memcached extension, it just happens to be filed in the PHP bug tracker, and I don't see in what way the PHP version affects the bug.

[woolardfa]

@sodabrew, you are correct. As I said in previous message, the issue is not in php_memcached.

[idmssvep]

With the explanation from woolardfa i could successfully resolve the issue from my application. Thanks!

[idmssvep]

Can the ini file description be updated please:

https://github.com/php-memcached-dev/php-memcached/blob/master/memcached.ini

Section:
; The time, in seconds, before a lock should release itself.
; Setting to 0 results in the default behaviour, which is to
; use the memcached.sess_lock_max_wait setting. If that is
; also 0, max_execution_time will be used.
memcached.sess_lock_expire = 0;

"memcached.sess_lock_max_wait" is deprecated, i assume that "memcached.sess_lock_wait_max" will be i place but i´d like to know.

[mikealmond]

Any update on this?

[nitinsbuzz]

Any chance queing requests on the server side can solve this problem?

[nitinsbuzz]

If this is an issue regarding only ajax requests, how about session_write_close() a.s.ap. after the ajax call? Any experience with this option? Will it remove the lock?

http://php.net/manual/en/function.session-write-close.php
http://php.net/manual/en/function.session-write-close.php#63970

[woolardfa]

session_write_close will end up clobbering the session data. session_abort() is the way to leave it alone.

[nitinsbuzz]

@woolardfa would session_abort() has also work immediatly after session_write_close() in case session data needs update on ajax call?

[woolardfa]

No. Use one or the other. The point is that you have to detect whether or not PHP/session/memcached was truly successful in establishing the lock entry (semaphore), and thus acquiring the right to update the session content, OR whether it just gave up after trying the specified number of times (timed out), and returned a True (the bug in session_start()) anyway. In this latter case, the session content given to the usurping process will be empty... and when session_write_close is called, or the page is allowed to end normally, that bogus (empty) session data will be written back clobbering the valid data. Calling session_abort will prevent that.

[nitinsbuzz]

I think the problem is with async requests (using a service worker). Only if memcached can handle async requests would it solve the problem or move long processes to the background to make sure sync requests do not get blocked.

[woolardfa]

Just an FYI... have tested PHP 7.2 (7.2.6), the underlying issue where session_start returns true when it in fact failed to acquire the session lock, and subsequently corrupts the existing session data, has been fixed.

A call to session_start() now returns false when it failed to acquire the session lock.

[onassar]

Hmmm I'm running 7.2.7 and still running into the issue..

[Eimantas123]

php 7.2.15, libmemcached version 1.0.18
problems still occur.

[csev]

Just an update - I am using AWS Elasticache as my Memcache and PHP 7.1. Setting session.lazy_write = Off got rid of the message. I am 100% guessing here, but this behavior suggests that with lazy_write on, it holds the lock a little while longer - even though it knows the request is done and that the session is unmodified. When it writes it back the lock is released. Good thing my memcache is pretty fast / underutilized for now :)

[csev]

Update. I was able to watch error logs and tweak all these values live and figure out what worked and what did not. Turning off session.lazy_write only reduced the frequency of the errors. Also trying to set the _min and _max to 150 as described above does not stop the errors. The only thing that stopped the errors are to turn ini_set('memcached.sess_locking', '0'); - and then turn lazy_write back on. I read that Redis does not lock at all. So this feels like the simplest / best path. (This is PHP 7.1 and libmemcached 1.0.18 ). Hope this helps.

[woolardfa]

@csev Good news that you've determined your simplest/best path, and do not want to discourage you, but do want to offer some things to consider if you choose to keep using memcached for sessions.

Firstly, I'd recommend getting to a current version of PHP where the fundamental bug is fixed. The bug is not in libmemcached, or in php-memcached; it's in PHP.

The issue is not so much how php_memcached is configured (i.e. _min, _max), because like you said, changing how those are configured didn't eliminate the errors--and they weren't meant to, they were just meant to a) mimic the old session acquisition timings, and b) try to reduce the frequency of the errors based on how one particular application, Moodle, was designed.

The issue is more about how sessions are handled at the application level, and whether your application code uses PHP's session management on almost every browser request, like Moodle currently does, (page requests, and subsequent AJAX requests), or if it's more selective. Also, does the application code detect whether the $_SESSION super global is valid or not (most all the time we take that for granted, and that's the patch I added to Moodle's memcached session startup, until PHP was fixed).

So, you're right, going with no session locks will eliminate the errors, because that particular lock/semaphore entry won't be created to begin with, and no attempt will be made to remove it. If you're application will work with no concurrency checks (last to touch it wins), then yeah, that's the ticket.

[csev]

@woolardfa - Thanks a ton - My application is Tsugi - an EdTech app like Moodle. It can survive with "eventual consistency" and "last write wins". I am encouraged by your suggestion to move beyond 7.1 - what version of PHP is this fixed in? I am a little gun shy as, the road through and to 7.0 and 7.1 was kind of rocky, stuff was not getting fixed, and even the apt repos were wonky and non-standard. So when I got something working I just breathed a sigh of relief and froze my infrastructure. But that was a year ago and I am sure things are better so I am heartened to give it another go if I know what version to go to - I am nervous about jumping to the absolute latest. Again, thanks.

[SvenRtbg]

Nobody has been doing this in this issue, but I think in order to get the full picture of a problem report you should add at least all the settings I mentioned in my comment above: #269 (comment)

sess_lock_wait_min, sess_lock_wait_max, sess_lock_expire, sess_lock_retries. Note that the _min and _max times are basically irrelevant for the functioning of the session in general, they are being used to fine-tune timing behavior. The most important settings are lock_expire (also compared to your max_execution_time) and lock_retries. With these set the wrong way, you may end up overwriting your session unintentionally.

[woolardfa]

@csev The pain points in Moodle (all handled by moodlehq) were mostly in the 5.x -> 7.x migration. I imagine there ought to be very few going from 7.1 -> 7.3. We've tried to keep our sites fairly up-to-date, and didn't see any issues when we moved from 7.1.x. to 7.2.x.

I believe the bug was fixed in 7.2.6 onward, so the current stable of either 7.2 or 7.3 will be prudent... but then, like I said earlier, it will just mean that your application code can then correctly detect whether the session (lock) was acquired or not before it proceeds.

[csev]

@woolardfa I upgraded to 7.3 and everything is working fine and I am using the default settings and seeing no lock issues. Thanks for your advice.

[murraycollingwood]

I'm using php 7.3 and it's still a problem.
I've gone with the config changes to see how this impacts the incidence.
If it remains a problem then I'm seriously considering turning locking off.

[murraycollingwood]

I'm using php 7.3 and it's still a problem.
The config changes _wait _min _max made no difference, in fact I think it got worse.
I have now turned off locking and it seems to have solved the locking problem.
I also enabled lazy_write, although I'm not sure if that was necessary.

However, as discussed above, turning off locking is dependent upon how you use your session data. I think most of my usage should be fine, I guess the next few days will be telling.

[dytyniuk]

Hello everyone.

I have read carefully all the comments and solutions proposed: starting with memcached.sess_lock_* settings changed and up to sessions locks turned off.

Frankly, I'm strongly convinced, the issue is not PHP-related, while specific to this extension. Details are the following:

• an application is running in Docker image based on php/7.3.16-fpm-alpine with both memcache and memcached extensions installed (we need both, since this is our internal "base" image for all PHP apps)
• All off memcached.* and memcache.* settings are left default:

memcache support => enabled
memcache.allow_failover => 1 => 1
memcache.chunk_size => 32768 => 32768
memcache.compress_threshold => 20000 => 20000
memcache.default_port => 11211 => 11211
memcache.hash_function => crc32 => crc32
memcache.hash_strategy => consistent => consistent
memcache.lock_timeout => 15 => 15
memcache.max_failover_attempts => 20 => 20
memcache.prefix_host_key => 0 => 0
memcache.prefix_host_key_remove_subdomain => 0 => 0
memcache.prefix_host_key_remove_www => 1 => 1
memcache.prefix_static_key => no value => no value
memcache.protocol => ascii => ascii
memcache.redundancy => 1 => 1
memcache.session_prefix_host_key => 0 => 0
memcache.session_prefix_host_key_remove_subdomain => 0 => 0
memcache.session_prefix_host_key_remove_www => 1 => 1
memcache.session_prefix_static_key => no value => no value
memcache.session_redundancy => 2 => 2
memcache.session_save_path => no value => no value


memcached support => enabled
libmemcached version => 1.0.18
memcached.compression_factor => 1.3 => 1.3
memcached.compression_threshold => 2000 => 2000
memcached.compression_type => fastlz => fastlz
memcached.default_binary_protocol => Off => Off
memcached.default_connect_timeout => 0 => 0
memcached.default_consistent_hash => Off => Off
memcached.serializer => php => php
memcached.sess_binary_protocol => On => On
memcached.sess_connect_timeout => 0 => 0
memcached.sess_consistent_hash => On => On
memcached.sess_consistent_hash_type => ketama => ketama
memcached.sess_lock_expire => 0 => 0
memcached.sess_lock_max_wait => not set => not set
memcached.sess_lock_retries => 5 => 5
memcached.sess_lock_wait => not set => not set
memcached.sess_lock_wait_max => 150 => 150
memcached.sess_lock_wait_min => 150 => 150
memcached.sess_locking => On => On
memcached.sess_number_of_replicas => 0 => 0
memcached.sess_persistent => Off => Off
memcached.sess_prefix => memc.sess.key. => memc.sess.key.
memcached.sess_randomize_replica_read => Off => Off
memcached.sess_remove_failed_servers => Off => Off
memcached.sess_sasl_password => no value => no value
memcached.sess_sasl_username => no value => no value
memcached.sess_server_failure_limit => 0 => 0
memcached.store_retry_count => 2 => 2

• as soon as you set session.save_handler=memcached and put any memcached hostname to session.save_path, consecutive XHR calls to the backed start to fail with session_lock error
• but, as soon as you change your session.save_handler to "legacy" memcache (session.save_handler=memcache) and modily session.save_path a bit:

session.save_handler=memcache
session.save_path=tcp://<hostname>:<port>/ # <port> is still required even if you use default 11211

Everything works as it should.

So, either default settings for memcached are missing proper tuning compared to memcache ones. Or there's something wrong with sessions locking on memcached side. Again, I do think so, since "legacy" memcache works like a charm 🤷‍♂️

[michnovka]

I still dont think this is resolved. I had to fall back to using files for sessions

[martinsoenen]

Hello,

I also have this problem with PHP version 7.4.33 and memcached 1.6.19 .
Is there anything we can do ? Disabling session.lazy_write did not help us.

[Misosooup]

Same here, still getting this issue with 7.4.33