New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use cURL as a fallback for the Auto-Cache Engine when allow_fopen_url is disabled #440

Closed
raamdev opened this Issue Mar 12, 2015 · 6 comments

Comments

Projects
None yet
2 participants
@raamdev
Contributor

raamdev commented Mar 12, 2015

Some web hosts disable allow_url_fopen in the PHP configuration on the server, which means the ZenCache Auto-Cache Engine will be unable to access the URL to the XML Sitemap (which it uses to figure out which pages it on the site it should generate a cache file for).

As a result, you may see an error like the following in your error log whenever the Auto-Cache Engine runs:

[Wed Mar 04 23:09:25 2015] [warn] [client 185.114.224.0] mod_fcgid: stderr: PHP Warning: XMLReader::open(http://example.com//sitemap_index.xml) [<a href='xmlreader.open'>xmlreader.open</a>]: failed to open stream: no suitable wrapper could be found in /home/example/www/wp-content/plugins/zencache-pro/includes/auto-cache.php on line 233

It should be possible to fetch a URL using cURL (which is more reliable than requiring allow_fopen_url, which many web hosts disable for security reasons) and then load the result into XMLReader().

@jaswsinc writes...

The XMLReader class reads the input XML from a stream, one node at a time. Making it possible for ZC to deal with extremely large (even multi-GB sitemaps). However, I don't see any reason why we couldn't have it open a stream from a local resource that we download via cURL. I'm not certain that will work, but definitely worth taking a look at, I agree.

Referencing: http://php.net/manual/en/xmlreader.open.php
See also: http://php.net/manual/en/xmlreader.read.php

@raamdev

This comment has been minimized.

Contributor

raamdev commented Dec 26, 2015

Referencing #644

@Ed-AITpro

This comment has been minimized.

Ed-AITpro commented Jun 14, 2016

allow_url_fopen is unfortunately is a common attack vector used by hackers so it should be disabled/turned off in a php.ini file for security reasons. I just purchased the Pro unlimited version and will unfortunately have to request a refund for now since I am not willing to open up this attack vector and allow allow_url_fopen on any of my websites. I will repurchase Comet Cache Pro again at a later time if you do provide a cURL or other safe method alternative to using allow_url_fopen. I'm bummed out about this, but security takes priority so that leaves me no choice here. I will continue to use your free Comet Cache version since it is awesome, but so wanted the Pro version. bummer.

@raamdev raamdev added this to the Next Release milestone Jun 17, 2016

@raamdev raamdev self-assigned this Jun 17, 2016

raamdev added a commit to websharks/comet-cache-pro that referenced this issue Jun 24, 2016

raamdev added a commit to websharks/comet-cache-pro that referenced this issue Jun 24, 2016

raamdev added a commit to websharks/comet-cache-pro that referenced this issue Jun 24, 2016

raamdev added a commit to websharks/comet-cache-pro that referenced this issue Jun 24, 2016

raamdev added a commit to websharks/comet-cache-pro that referenced this issue Jun 24, 2016

raamdev added a commit to websharks/comet-cache-pro that referenced this issue Jun 24, 2016

@raamdev

This comment has been minimized.

Contributor

raamdev commented Jun 24, 2016

The work on this issue includes a new filter that allows you to force the use of the fallback when fetching the XML Sitemap. The fallback uses the WP HTTP API to download the XML Sitemap and then parses it from the local file. The WP HTTP API includes a fallback to cURL, so if allow_fopen_url is disabled, it will use cURL instead.

If allow_fopen_url is enabled and you want to force the use of the fallback anyway, you can create an MU-Plugin and add the following override:

add_filter('comet_cache_auto_cache_sitemap_force_wp_http_api', function() { return true; });
@raamdev

This comment has been minimized.

Contributor

raamdev commented Jun 24, 2016

Next Release Changelog:

  • Enhancement (Pro): The Auto-Cache Engine now supports a fallback to cURL using the WP HTTP API. If your PHP configuration has allow_fopen_url=0, the Auto-Cache Engine will use the fallback to download the XML Sitemap and parse it from a temporary file. If you want to force the use of this fallback even when allow_fopen_url=1, you can use a filter. See Issue #440.

@raamdev raamdev closed this Jun 24, 2016

@Ed-AITpro

This comment has been minimized.

Ed-AITpro commented Jun 24, 2016

Awesome! Very impressed with how quickly you created the additional options/features to get the cURL alternative caching method implemented. Woo Hoo!!!

raamdev added a commit that referenced this issue Jul 7, 2016

Phing release of v160706 with the following changes:
- **New Feature! Apache Optimizations.** This release includes a completely new option panel for Apache Performance Tuning. Current options for Apache tuning include GZIP Compression, Leverage Browser Caching, Enforce Canonical URLs, and Send Access-Control-Allow-Origin Header (for Static CDN Filters). These options automatically add or remove from your `.htaccess` file the appropriate configuration based on the options you enable or disable (all options are disabled by default, so your `.htaccess` file is not modified unless you say so). If you prefer to update your `.htaccess` file manually, the necessary configuration can be viewed beneath each option. Props @jaswsinc, @renzms. See [Issue #789](#789).
- **New Feature!** A new "Enable GZIP Compression" option has been added to the new Apache Optimizations panel. This option will automatically add the appropriate configuration to your `.htaccess` file to enable GZIP compression. This option is disabled by default. The old "GZIP Compression" panel has been removed in favor of the new option inside Apache Optimizations. Props @renzms, @jaswsinc. See [Issue #764](#764).
- **New Feature!** Multisite Host Exclusion Patterns. It's now possible to exclude entire sites from the cache in a Multisite Network environment. Domain mapping is also supported! See _Comet Cache → Plugin Options → Host Exclusion Patterns_. If you're running a Multisite Network with Sub-Directories, you can exclude sites using the existing URI Exclusion Patterns feature. Props @kristineds. See [Issue #754](#754).
- **New Feature (Pro)!** A new "Leverage Browser Caching" option has been added to the new Apache Optimizations panel. This option will automatically add the appropriate configuration to your `.htaccess` file to enable Browser Caching. This option is disabled by default. Props @renzms, @jaswsinc. See [Issue #764](#764).
- **New Feature (Pro)!** A new "Enforce Canonical URLs" option has been added to the new Apache Optimizations panel. This options adds the appropriate `.htaccess` code to enforce the correct canonical URLs according to your WordPress Permalink settings (Comet Cache detects if the Permalink Structure ends with a trailing slash, or without a trailing slash). Props @renzms, @jaswsinc. See [Issue #554](#554).
- **Bug Fix**: In some scenarios the Cron Event that cleans up expired cache files (`_cron_comet_cache_cleanup`) would never run, or the Next Run time would constantly reset to 1 minute away from running every time a page was reloaded. We suspect this is a race condition and in attempt to work around this issue we now skip all of our Cron-related checks if Cron is currently in the middle of running a process. Props @xberg and @lkraav for help reporting. See [Issue #653](#653).
- **Bug Fix**: If your site uses aliased domains, Comet Cache now properly considers all possible domain variations when it clears the cache on WP Standard installations. Props @kristineds, @jaswsinc, @yoffe, and @VR51. See [Issue #608](#608).
- **Bug Fix** (Pro): Fixed a bug where Comet Cache would appear to prevent WordPress from redirecting Permalinks that don't include a trailing slash, to the URL that does include a trailing slash. This was due to the fact that Comet Cache loads very early on (for caching purposes) and as a result the WordPress `redirect_canonical()` function never gets run. This was fixed by adding an option to the new Apache Optimizations panel that allows you to Enforce Canonical URLs. Props @renzms, @jaswsinc. See [Issue #554](#554).
- **UX Bug Fix** (Pro): If you had your WordPress Dashboard login details saved by your browser, the browser autofill would automatically fill in the Pro Plugin Updater fields with those details, which then needed to be replaced with your actual Pro license details. The browser autofill has been disabled for those fields (tested in Chrome, Firefox, and Safari). Props @renzms. See [Issue #741](#741).
- **Enhancement**: Added links the Options Page for the Comet Cache [Twitter](http://twitter.com/cometcache) and [Facebook](http://facebook.com/cometcache) accounts. Props @renzms. [Issue #771](#771).
- **Enhancement:** Added full support for UTF-8 (multibyte strings). This release adds full support for UTF-8 throughout the Comet Cache codebase, greatly enhancing Comet Cache's ability to deal with file paths and URLs that may contain UTF-8 characters. Props @jaswsinc. [Issue #703](#703).
- **UI Enhancements**: Improved the Logged-In Users and the Client-Side Caching options panels to dim additional options when the feature is disabled. Additionally, the "Enable HTML Compression for Logged-In Users?" option has been relocated from the HTML Compressor option panel to the more appropriate Logged-In Users option panel. See [Issue #768](#768).
- **UX Enhancement**: Improved the inline docs for Auto-Clear List of Custom URLs to clarify that full URLs must be provided. Props @renzms. See [Issue #781](#781).
- **Enhancement** (Pro): The Pro Plugin Updater has been improved to allow for better compatibility with hosting platforms that use Apache's ModSecurity. In some cases, site owners were seeing a 404 error when attempting to update the Pro version using the Pro Plugin updater because certain ModSecurity rules were blocking the Pro Updater requests. The Pro Plugin Updater now uses WP Transients to store the necessary metadata, which works around the issue with ModSecurity. Props to @seozones for reporting and @jaswsinc for help fixing this. [Issue #416](#416).
- **Enhancement** (Pro): When Static CDN Filters are enabled, it's now possible to disable the automatic insertion of rules into your `.htaccess` file that are designed to prevent issues with [CORS](https://cometcache.com/kb-article/what-are-cross-origin-request-blocked-errors/). See _Apache Optimizations → Send Access-Control-Allow-Origin Header?_ See [Issue #787](#787).
- **Enhancement** (Pro): The HTML Notes added to the bottom of a cached page now specify if the page was cached as the result of an HTTP Request or if it was cached by the Auto-Cache Engine. Props @kristineds. See [Issue #292](#292).
- **Enhancement** (Pro): The Auto-Cache Engine now supports a fallback to cURL using the WP HTTP API. If your PHP configuration has `allow_fopen_url=0`, the Auto-Cache Engine will use the fallback to download the XML Sitemap and parse it from a temporary file. If you want to force the use of this fallback even when `allow_fopen_url=1`, you can use [a filter](#440 (comment)). See [Issue #440](#440).
- **UI Enhancement** (Pro): A second button has been added to the bottom of the Pro Plugin Updater page that allows you to "Save and Update Comet Cache Pro" in one step. Props @renzms. See [Issue #741](#741).
- **UI Enhancement** (Pro): The "Cache Stats" button in Admin Bar is now linked to the Cache Stats page. Instead of hovering over the button and then clicking "More Info" inside the popup panel, you can now just click the "Cache Stats" button to go directly to the Cache Stats page. Props @Presskopp, @renzms. See [Issue #780](#780).
- **Comment Mail Compatibility:** Improved compatibility with the Comment Mail plugin by automatically clearing the cache whenever Comment Mail options are changed. Many of the Comment Mail options affect front-end portions of the site, so it's important that the cache is cleared whenever Comment Mail options change. See [Comment Mail Issue #278](websharks/comment-mail#278 (comment)).
- **PHP Compatibility:** Improved compatibility back to PHP 5.2 (the lowest version allowed by WordPress). Comet Cache still requires PHP 5.4+, but if you install Comet Cache on a site running PHP 5.2, it will now fail gracefully with a Dashboard notice indicating PHP 5.4+ is required, instead of producing a fatal error. See [Issue #784](#784).
- **WP-CLI Compatibility**: Fixed a bug with deactivating Comet Cache using WP-CLI. Doing so was producing a "Invalid argument; host token empty!" error message. This has been resolved. Props @MarioKnight @jaswsinc @renzms. See [Issue #728](#728).
- Renamed `COMET_CACHE_ALLOW_BROWSER_CACHE` constant to `COMET_CACHE_ALLOW_CLIENT_SIDE_CACHE`. Backwards compatibility has been maintained.
- Renamed `allow_browser_cache` plugin option to `allow_client_side_cache`.
@raamdev

This comment has been minimized.

Contributor

raamdev commented Jul 7, 2016

Comet Cache v160706 has been released and includes changes from this GitHub Issue. See the v160706 announcement for further details.


This issue will now be locked to further updates. If you have something to add related to this GitHub Issue, please open a new GitHub Issue and reference this one (#440).

@websharks websharks locked and limited conversation to collaborators Jul 7, 2016

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.