Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Symlink Race Condition #242

Closed
jaswrks opened this issue Jul 14, 2014 · 10 comments
Closed

Symlink Race Condition #242

jaswrks opened this issue Jul 14, 2014 · 10 comments
Labels
bug lite pro
Milestone
v140829

Comments

@jaswrks
Copy link

jaswrks commented Jul 14, 2014

I have seen this error pop up in log files recently...

PHP Fatal error: Uncaught exception 'Exception' with message 'Unable to create symlink. Possible permissions issue (or race condition), please check your cache directory. (advanced-cache.php on line 1316)

Other than symlink() calls when 404 caching is enabled, the rest of QC writes cache files atomically; i.e. creation followed by a rename. We need to improve the way symlinks are generated also; otherwise a race condition can occur on a busy site.

Referencing: https://github.com/websharks/quick-cache/blob/000000-dev/quick-cache/includes/advanced-cache.tpl.php#L914

Support Threads Referencing this Issue
@raamdev
Copy link
Contributor

raamdev commented Jul 14, 2014

Good catch!

I just tested some code to confirm that you can just use rename() on a symlink, the same way we use rename() for atomic cache file creation here.

So the following changes are all that should be necessary to improve the symlink() calls, correct?

$cache_file_tmp = $this->cache_file.'.'.uniqid('', TRUE).'.tmp'; // Cache creation is atomic; e.g. tmp file w/ rename.

if($this->is_404 && is_file($this->cache_file_404))
    if(!(symlink($this->cache_file_404, $cache_file_tmp) && rename($cache_file_tmp, $this->cache_file_404)))
        throw new \exception(sprintf(__('Unable to create symlink: `%1$s` » `%2$s`. Possible permissions issue (or race condition), please check your cache directory: `%3$s`.', $this->text_domain), $this->cache_file, $this->cache_file_404, QUICK_CACHE_DIR));
    else return (boolean)$this->maybe_set_debug_info($this::NC_DEBUG_1ST_TIME_404_SYMLINK);

@jaswrks
Copy link
Author

jaswrks commented Jul 15, 2014

Yes, looks great!

@raamdev raamdev added this to the Future Release milestone Jul 15, 2014
@raamdev
Copy link
Contributor

raamdev commented Jul 15, 2014

Assigning to Future Release milestone.

@jaswrks
Copy link
Author

jaswrks commented Jul 25, 2014

@raamdev Since discovering this, I disabled 404 caching on two sites that I run. However, oddly enough this error still pops into the logs every hour or two throughout the day (with 404 caching disabled). It's puzzling me. I have no answer for it at the moment, but thought I would add that to the report here.

[24-Jul-2014 20:21:44 UTC] PHP Warning:  symlink(): File exists in /var/www/mystery-site-x/htdocs/wp-content/advanced-cache.php on line 1288
[24-Jul-2014 20:21:44 UTC] PHP Fatal error:  Uncaught exception 'Exception' with message 'Unable to create symlink: `/var/www/mystery-site-x/htdocs/wp-content/cache/quick-cache/cache/http/www-mystery-site-x-com/videos/60abe396bbec4fd6.html` » `/var/www/mystery-site-x/htdocs/wp-content/cache/quick-cache/cache/http/www-mystery-site-x-com/----404----.html`. Possible permissions issue (or race condition), please check your cache directory: `/var/www/mystery-site-x/htdocs/wp-content/cache/quick-cache/cache`.' in /var/www/mystery-site-x/htdocs/wp-content/advanced-cache.php:1289
Stack trace:
#0 [internal function]: quick_cache\advanced_cache->output_buffer_callback_handler('<!DOCTYPE html>...', 9)
#1 /var/www/mystery-site-x/htdocs/wp-includes/functions.php(2935): ob_end_flush()
#2 [internal function]: wp_ob_end_flush_all('')
#3 /var/www/mystery-site-x/htdocs/wp-includes/plugin.php(470): call_user_func_array('wp_ob_end_flush...', Array)
#4 /var/www/mystery-site-x/htdocs/wp-includes/load.php(573): do_action('shutdown')
#5 [internal function]: shutdown_action_hook()
#6 {main}
  thrown in /var/www/mystery-site-x/htdocs/wp-content/advanced-cache.php on line 1289
[24-Jul-2014 20:25:10 UTC] PHP Warning:  symlink(): File exists in /var/www/mystery-site-x/htdocs/wp-content/advanced-cache.php on line 1288
[24-Jul-2014 20:25:10 UTC] PHP Fatal error:  Uncaught exception 'Exception' with message 'Unable to create symlink: `/var/www/mystery-site-x/htdocs/wp-content/cache/quick-cache/cache/http/www-mystery-site-x-com/codex/stable/mystery-site-x/api-functions/package-summary/rk-0/rs-tzg4h0rpgu1mze-ydkp9plnnav8-.html` » `/var/www/mystery-site-x/htdocs/wp-content/cache/quick-cache/cache/http/www-mystery-site-x-com/----404----.html`. Possible permissions issue (or race condition), please check your cache directory: `/var/www/mystery-site-x/htdocs/wp-content/cache/quick-cache/cache`.' in /var/www/mystery-site-x/htdocs/wp-content/advanced-cache.php:1289
Stack trace:
#0 [internal function]: quick_cache\advanced_cache->output_buffer_callback_handler('<!DOCTYPE html>...', 9)
#1 /var/www/mystery-site-x/htdocs/wp-includes/functions.php(2935): ob_end_flush()
#2 [internal function]: wp_ob_end_flush_all('')
#3 /var/www/mystery-site-x/htdocs/wp-includes/plugin.php(470): call_user_func_array('wp_ob_end_flush...', Array)
#4 /var/www/mystery-site-x/htdocs/wp-includes/load.php(573): do_action('shutdown')
#5 [internal function]: shutdow in /var/www/mystery-site-x/htdocs/wp-content/advanced-cache.php on line 1289

@jaswrks
Copy link
Author

jaswrks commented Jul 25, 2014

With 404 caching disabled, I also still have the ---------404----------.html cache file too. It's as if 404 caching is still enabled. I have checked, double-checked, and triple checked this to be sure it was not an oversight on my part; i.e. that 404 caching is in fact disabled. It is. I even inspected the resulting advanced-cache.php file on the server and it has the proper config flags in it. I just don't get it :-)

I will need to do more testing and make an attempt to reproduce this elsewhere. Hopefully I will have additional information that might help with this soon. For the moment, it is a complete mystery to me.

@raamdev
Copy link
Contributor

raamdev commented Jul 25, 2014

Very interesting! I'll do some investigation and testing of my own once I get the next release done.

@jaswrks
Copy link
Author

jaswrks commented Jul 28, 2014

It's as if 404 caching is still enabled.

The latest release seems to have resolved this. I won't pretend to know why, but it was probably an error on my part; or perhaps just something odd in the particular branch that I had running on those sites previously. Anyway, it seems fine now. We just need to resolve the potential for a race condition when 404 caching is enabled.

@raamdev
Copy link
Contributor

raamdev commented Jul 28, 2014

We just need to resolve the potential for a race condition when 404 caching is enabled.

Got it. Will do! I'll add this to the Next Release milestone when I work on triaging those. :) Thanks.

@raamdev
Copy link
Contributor

raamdev commented Aug 4, 2014

Also reported here: http://wordpress.org/support/topic/error-log-have-lots-of-errors-from-quick-cache

raamdev added a commit to wpsharks/comet-cache-pro that referenced this issue Aug 6, 2014
@raamdev
Symlink 404 cache files atomically to avoid potential race condition.
58edc82 
See wpsharks/comet-cache#242
raamdev added a commit that referenced this issue Aug 6, 2014
@raamdev
Symlink 404 cache files atomically to avoid potential race condition.
1c11f59 
See #242
This was referenced Aug 6, 2014
Closed
Closed
raamdev added a commit to wpsharks/comet-cache-pro that referenced this issue Aug 6, 2014
@raamdev
Fix bug and improve code in 404 symlink creation.
81b8813 
See wpsharks/comet-cache#242
raamdev added a commit that referenced this issue Aug 6, 2014
@raamdev
Fix bug and improve code in 404 symlink creation.
80da9a6 
See #242
@raamdev
Copy link
Contributor

raamdev commented Aug 6, 2014

Closed by PR #72.

@raamdev raamdev closed this as completed Aug 6, 2014
@raamdev raamdev mentioned this issue Aug 18, 2014
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug lite pro
Projects
None yet
Milestone
v140829
Development

No branches or pull requests
2 participants
@raamdev @jaswrks