Feature Request: CDN

[jaswrks]

@raamdev I'd like to request the following feature in QC Pro.

• I would like to be able to serve WordPress media from a CDN of my choosing. For instance, so I can cache static media at edge locations with CloudFront.

Here is a clip from a custom plugin that I've used for quite awhile now. Perhaps it could serve as a starting point for a whole new set of functionality that's offered by QC Pro when it comes to CDNs.

add_filter('pre_option_upload_path', 'cdn::uploads_path');
add_filter('pre_option_upload_url_path', 'cdn::uploads_url_path');

class cdn // Content delivery network for this site.
{
    public static function uploads_path()
        {
            return '/amazon-s3-bucket-mount';
            // Files uploaded through the WP Media Library go here.
        }

    public static function uploads_url_path()
        {
            return 'http://d1v41qemfjie0l.cloudfront.net/';
            // Files served by WordPress that reside in my `upload_path`; are served with this URL as the base.
        }
}

---

This code assumes that an S3 Bucket has been mounted locally, so it won't work unless a site owner can do that. Ideally, Quick Cache Pro could simplify this part further and make it easier for a novice site owner to set things up on their own.

---

Other Considerations...

• It's probably worth having a discussion about how far QC Pro is going to take it's CDN integration — if and when it's added as a pro feature.

  The feature request here only asks for the Media Library. However, there are plugins like CloudFlare that deal with ALL static files served by WordPress, which is even better IMO.

• Which CDNs can we support? Some, or all of them? The code sample above could potentially work with any CDN that could be mounted locally. However, that's still problematic because not all of them can be of course.

• Multisite compatibility.

---

A QC user recently wrote...

I just installed your quick cache plugin last night and I just wanted to give you a quick suggestion that I think would improve sales significantly. I was looking at the difference in the free and pro versions and seeing if I wanted to upgrade, the thing that would have made me pay the $15 upgrade right away I didn't see in the pro version. That feature is CDN image rewriting. The feature is present in both your top 2 competitors for free but your product is the easiest to use of the 3 products. I googled for the feature and found many people requesting the same feature from your product saying it's the only thing that it's missing. I also tried several other plugins along with your plugin to accomplish this feature and they wouldn't point the images to my Amazon Cloudfront CDN properly while your plugin cached the pages. As a first time user that spent an hour analyzing solutions to speed up Wordpress last night I wanted to let you know that you would have
the perfect combination product if you added that one feature as caching and CDN are the top two methods to increase speed, why not make them into one super easy plugin? To keep things as simple as you have them all I would want is a single field to enter the new address to change the local image path into the CDN image path.

---

TODOs

• Create repo for AWS SDK. https://github.com/websharks/aws-php-sdk
• Create repo for MaxCDN SDK. https://github.com/websharks/maxcdn-php-sdk
• Add submodule for AWS SDK. quick-cache-pro/submodules/aws-php-sdk
• Add submodule for MaxCDN SDK. quick-cache-pro/submodules/maxcdn-php-sdk
• Rename htmlc submodule to quick-cache-pro/submodules/html-compressor
• Initial URL filter-related methods implemented.
• Implement content filtering methods; i.e. content_filter() in the cdn_base class.
• Test MS Network compat. against cdn_base::local_file().
• Work on create_distro() method for CloudFront.
• Work on get_distro() method for CloudFront.
• Work on update_distro() method for CloudFront.
• Work on create_distro() method for MaxCDN.
• Work on get_distro() method for MaxCDN.
• Work on update_distro() method for MaxCDN.
• Need to discuss cache invalidations again with @raamdev.

[Slobork]

Hi,

I want to say that I've tried the CDN-Linker plugin with the Quick Cache and it mixes great. Maybe you can add its functionality to your plugin, I know that one of your competitor already does this.
This is the CDN plugin - https://github.com/wmark/CDN-Linker

Alternatively you can just keep your plugin compatible with the CDN-Linker.

[raamdev]

@SBOQ Thank you very much for that info! I'll be working on this soon and will try to maintain support for that plugin as well.

[bridgeport]

One thing I noticed with the various CDN plugins (whether they use output buffering regex or not) is that they don't provide the ability to switch CDN URL's based on whether the current page is HTTP or HTTPS (SSL). This can become an issue if you're using a CNAME CDN (e.g., cdn.example.com), but also need to CDNify your HTTPS pages. If your SSL pages aren't pointing to a URL that has a valid security certificate, you'll get browser security errors.

For example, using the CloudFront CDN with origin pull, you'd have to use uglySubdomain.cloudfront.net for SSL pages, as the cdn.example.com CNAME wouldn't match the SSL certificate. Now, amazon does offer SSL CNAME certificates for around $600/month, but that's probably cost-prohibitive for a lot of folks. You could probably get it to work locally if you acquired a wildcard SSL certificate, but that's more expensive than a non-wildcard. So just using the ugly SSL CDN URL is the least expensive option.

What's interesting with Quick Cache implementing CDN support (SSL page or not) is that CDN URL conversion could be performed whether or not caching is being performed. For instance, if someone excludes certain pages from caching, it's probable that they'll still want their static files CDNified. So Quick Cache would be more of a hybrid plugin where the CDNification operates independently from the caching.

[Slobork]

@raamdev I'm configuring a new site and I wonder should I install the existing CDN plugin or wait for your built-in with QC. It's not that I want to be pushy, I just wonder how fast this can be expected?

[raamdev]

@SBOQ I don't have any precise ETA, but it will be at least a few weeks before I have this feature integrated, tested, and ready for release.

[Slobork]

@raamdev Well since you've decided to implement the CDN into the Quick Cache I have to say that I cheer for this feature to come fast because I'll have to reconfigure things later when it becomes available.
The challenge would probably be with Multisite where QC is designed to be network active, while CDN plugin should be enabled and configured site by site. That's the reason the current situation with 2 separate plugins is working very well. I just wanted to remind you about this.

Edit: Maybe setting the CDN plugin to work like a plugin for the QC could be a good approach, then it's functionality can be accessed on a site by site basis.

[Slobork]

@raamdev it's 2 months since the hint about the CDN addition. Is this feature any time close?
Thanks.

[raamdev]

@SBOQ My apologies for the delay. I'm a bit behind on things, but this feature, along with Branched Cache Structure, are at the top of my list for new features that I'm working on for Quick Cache.

[jaswrks]

This code assumes that an S3 Bucket has been mounted locally, so it won't work unless a site owner can do that. Ideally, Quick Cache Pro could simplify this part further and make it easier for a novice site owner to set things up on their own.

@raamdev In my original feature request I noted the above.

I see there is an easy solution to this, which I will describe briefly...

Using the AWS-SDK (which we can redistribute w/ Quick Cache); we can add a stream wrapper like s3://[bucket]/uploads. That would be (pretty much) the same as having an S3 bucket mounted locally, but in a much more compatible way that can be accomplished via PHP code alone.

See: http://docs.aws.amazon.com/aws-sdk-php/guide/latest/feature-s3-stream-wrapper.html

---

So with that in place... the example I posted earlier might look like this...

<?php
require_once 'aws.phar';
use Aws\S3\S3Client;

add_action('init', 'cdn::init');
add_filter('pre_option_upload_path', 'cdn::uploads_path');
add_filter('pre_option_upload_url_path', 'cdn::uploads_url_path');

class cdn // Content delivery network for this site.
{
    public static function init()
        {
            $client = S3Client::factory(array(
                                            'key'    => '<aws access key>',
                                            'secret' => '<aws secret key>'
                                        ));
            $client->registerStreamWrapper(); // Register `s3://` stream wrapper.
        }

    public static function uploads_path()
        {
            return 's3://[bucket]/uploads';
            // Files uploaded through the WP Media Library go here.
        }

    public static function uploads_url_path()
        {
            return 'http://d1v41qemfjie0l.cloudfront.net/';
            // Files served by WordPress that reside in my `upload_path`; are served with this URL as the base.
        }
}

[jaswrks]

@raamdev The AWS-SDK can be downloaded here.
http://aws.amazon.com/sdkforphp/

[raamdev]

@JasWSInc Perfect! Thanks so much. I'll work on getting this integrated so that we can start testing it for a future release.

[jaswrks]

@raamdev I just ran some tests against the S3 stream wrapper in my work on another project. There are two issues that I ran into which I thought I would share with you here.

1. The WordPress filters I mentioned previously in this topic: pre_option_upload_path and pre_option_upload_url_path; will NOT work with an s3:// stream wrapper.

   WP forces an ABSPATH prefix on the values. So this won't work because you will end up with /abs/path/to/wordpress/s3://bucket/uploads. Not good.

   The solution is to filter the entire array of data that WP uses. See filter upload_dir here: https://github.com/WordPress/WordPress/blob/master/wp-includes/functions.php#L1658

   A quick example...

     add_filter('upload_dir', 'cdn::upload_dir');
   
     class cdn // Content delivery network for this site.
     {
         public static function upload_dir($details)
             {
                 $details['basedir'] = 's3://[BUCKET]/uploads';
                 $details['path']    = $details['basedir'].$details['subdir'];
   
                 $details['baseurl'] = 'http://d1v41qemfjie0l.cloudfront.net/uploads';
                 $details['url']     = $details['baseurl'].$details['subdir'];
   
                 return $details; // After CDN filters.
             }
     }
   

2. You MUST use the latest release of the AWS-SDK. I was testing this the other day and ran into a major limitation in the S3 stream wrapper related to mkdir() not working properly. I contacted AWS and they've had the issue corrected in the latest release (available today).

   See: https://github.com/aws/aws-sdk-php/releases

[raamdev]

@JasWSInc Thanks! That will definitely come in handy when I tackle CDN support. I'm currently working on #3.

[ethanpil]

+! on CDN. I use CDN-Linker and its great, but if we can combine that with S3 support for the cache files... AMAZING!!

[raamdev]

Adding this to the Future Release milestone to work on next, after the Next Release.

[raamdev]

Also requested here: http://wordpress.org/support/topic/cdn-support?replies=1

[raamdev]

Punting to Future Release milestone so that the bugs in the Next Release milestone can be released sooner rather than later. CDN integration will be my top priority feature to work on for the following release.

[ioerjfoige0439i]

Not sure if my comment showed up.

Two questions for you:

1. Is this CDN feature going to be as good as W3 Total Cache's CDN with MaxCDN? I intend to use it with MaxCDN.
2. Will Quick Cache & the CDN feature be fully compatible with s2Member and iDevAffiliate? (because W3 Total Cache is not 100% compatible with both)

Thanks!

[raamdev]

Is this CDN feature going to be as good as W3 Total Cache's CDN with MaxCDN? I intend to use it with MaxCDN.

Yes.

Will Quick Cache & the CDN feature be fully compatible with s2Member and iDevAffiliate? (because W3 Total Cache is not 100% compatible with both)

Yes. It will be fully compatible with s2Member and iDevAffiliate.

[ioerjfoige0439i]

Raam you're going to make me cry bro. Lol, I'm so excited. Now I can finally get rid of W3 Total Cache! Weeee!

[jaswrks]

😿 : I am already there. Tears of happiness brother. I can't wait!

[ethanpil]

Very excited here on this as well...

But to completely replace other (bloated) caching plugins, would need #47 too!! (Just a little plug and reminder!! :))))

[ioerjfoige0439i]

@raamdev @jaswsinc Holy moly, DreamSpeed CDN (Part of DreamObjects) just came out! Can you please make sure your release (I'm assuming is on the 19th) is compatible with this as well!! It's much better than MaxCDN so we're going to be switching over to it and I really want to delete as many plugins as I'm using as possible so we're only using WebSharks products and iDevAffiliate. Here's the link: https://www.dreamhost.com/cloud/cdn/ Thank you!! And here's a wiki on it http://wiki.dreamhost.com/DreamSpeed_CDN_Overview

[raamdev]

Request for Comments

What is the intention for Quick Cache CDN integration? @jaswsinc your initial comment when you opened this feature request indicated the following:

I would like to be able to serve WordPress media from a CDN of my choosing. For instance, so I can cache static media at edge locations with CloudFront.

But "static media" here implies only media uploaded through the WordPress Media Library, correct?

There are lots of ways Quick Cache could utilize CDN integration:

• Host attachments (files uploaded to the WordPress Media Library)
• Host minified CSS/JS files (via the HTML Compressor)
• Host WordPress Core files (e.g., wp-includes/*)
• Host Theme Files (e.g., wp-content/themes/<active-theme>/*)

I feel that it would be nice to eventually support all of these, but obviously that will take some time, planning, and testing. I'm thinking it will make sense to start with CDN support for hosting attachments (files uploaded through the Media Library) and then work on the other features, maybe in the order listed above.

Thoughts?

[jaswrks]

@raamdev I'm going to respond to this privately and share my thoughts about what we could do here.

[ioerjfoige0439i]

Any updates guys? Here's the TRIFECTA pictured above:
Quick Cache / Zen Cache perfectly integrated with:
s2Member
iDevAffiliate
MaxCDN

[ioerjfoige0439i]

Let me know your thoughts on my above Trifecta @clavaque @raamdev @BruceCaldwell @jaswsinc @ethanpil @Reedyseth I think this would be the dream for all owners of Quick Cache / Zen Cache.

[ioerjfoige0439i]

iDevAffiliate works with s2Member now already but unfortunately, iDevAffiliate and s2Member don't work together properly when W3 Total Cache is turned on, but W3 Total Cache does MaxCDN perfectly. If W3 Total Cache worked properly with iDevAffiliate and s2Member then I would just use it instead because all that's missing is MaxCDN. If Quick Cache / Zen Cache could work nicely and fully tested together with all 3 then that would eliminate our need for W3 Total Cache completely and allow everyone to see the full potential of all these technologies working together!

[ioerjfoige0439i]

Of course being that iDevAffiliate has it's own seperate install folder, but just that it can properly communicate with s2Member when Quick Cache / Zen Cache is turned on and connected to MaxCDN. iDevAffiliate would not need to be using any CDN features because it is pretty lite weight ( I mean unless you wanted to do that in the future but for right now that's not very important ).

[raamdev]

@ioerjfoige0439i thank you for the ideas and the suggestions. :)

We already have plans to launch ZenCache with MaxCDN support within the next few weeks / early January. We'll ensure that ZenCache + s2Member + iDevAffiliate work together as expected!

[ioerjfoige0439i]

@raamdev Great, I can't wait to finally dump W3 Total Cache and consolidate my plugins. It's much easier for me to buy all my plugins with Web Sharks Inc. or those that work in conjunction with Web Sharks Inc. plugins (such as iDevAffiliate) than to have other 3rd party plugins that will introduce compatibility issues into my workflow. It makes it much easier to know that you guys are handling everything in house and have your best interest to squash any bugs that arise between different plugins. Good work.

[ioerjfoige0439i]

@clavaque @raamdev @BruceCaldwell @jaswsinc @ethanpil @Reedyseth Don't forget about Amazon S3 and Amazon Cloudfront CDN options as well :-)!

[ioerjfoige0439i]

@clavaque @raamdev @BruceCaldwell @jaswsinc @ethanpil @Reedyseth As well as a “Generic” origin-pull Mirror in case we want to add other CDN's like CDN77 or CDN.net

[tokar86a]

How is it going? Any ETA?

[raamdev]

@tokar86a We'll have CDN functionality released as part of ZenCache, which is coming this month (the soft launch is likely happening this week).

If you're interested in testing a beta version when that goes out, please make sure you're on the beta testers list.

[ethanpil]

Are we going to lose our registration when you switch over?

[raamdev]

Are we going to lose our registration when you switch over?

No, you won't lose anything. All existing Quick Cache Pro customers will automatically have access to ZenCache Pro.

[ioerjfoige0439i]

Thanks guys, just signed up for the pro beta testers list

[tokar86a]

Hove is it going whit this soft release?

[raamdev]

@tokar86a The soft release is going well. :-) We should be doing a public release next week. I'll update here again when that happens.

[raamdev]

CDN Integration has been released as part of the changeover to ZenCache.

See Introduction to Static CDN Filters.