From 88e879d18dcb8b280a9559548e807d14d9c13263 Mon Sep 17 00:00:00 2001 From: Tim Fischbach Date: Thu, 10 Aug 2017 11:34:20 +0200 Subject: [PATCH] Import external services doc from wiki Allow versioning the contents together with gem releases. --- README.md | 4 +- doc/setting_up_external_services.md | 181 +++++++++++++++++++++++++++ doc/setting_up_s3_bucket_policies.md | 95 ++++++++++++++ 3 files changed, 278 insertions(+), 2 deletions(-) create mode 100644 doc/setting_up_external_services.md create mode 100644 doc/setting_up_s3_bucket_policies.md diff --git a/README.md b/README.md index 8c8566e0b5..9576834a1e 100644 --- a/README.md +++ b/README.md @@ -124,8 +124,8 @@ things do not look too blank in development mode. ## Configuration Pageflow stores files in S3 buckets also in development -mode. Otherwise there's no way to have Zencoder encode them. See the -wiki page [Setting up external services](https://github.com/codevise/pageflow/wiki/Setting-up-External-Services). +mode. Otherwise there's no way to have Zencoder encode them. See +[setting up external services](./doc/setting_up_external_services.md). The host application can utilize environment variables to configure the API keys for S3 and Zencoder. The variables can be found in the generated Pageflow initializer. diff --git a/doc/setting_up_external_services.md b/doc/setting_up_external_services.md new file mode 100644 index 0000000000..73ceed8822 --- /dev/null +++ b/doc/setting_up_external_services.md @@ -0,0 +1,181 @@ +# Setting up External Services + +Pageflow uses some external services to process and store media files: + + * [Amazon S3](http://aws.amazon.com/s3/) to host its videos, images + and audios. + * [Zencoder](https://zencoder.com/) to encode the content to fit the + different formats needed. + * (In production environment) A + [CDN](http://en.wikipedia.org/wiki/Content_delivery_network) to + speed up access to the content + (e.g. [Amazon Cloudfront](http://aws.amazon.com/cloudfront/)). + +Please create accounts for [Amazon AWS](http://aws.amazon.com/) and +[Zencoder](https://zencoder.com/). + +Files that are uploaded to be used in Pageflow will be stored in an S3 +bucket. Zencoder takes them from there, encodes the files and stores +them in a second bucket. The website displays the content from this +second bucket via the CDN. + +## Amazon S3 + +Two Amazon S3 buckets are needed are needed for each environment. + + * Main bucket for paperclip attachments and Zencoder to read from. + * Output bucket for Zencoder to write encoded files to. + +For example: + + * `de-mycompany-pageflow-production` + * `de-mycompany-pageflow-production-out` + * `de-mycompany-pageflow-development` + * `de-mycompany-pageflow-development-out` + +You can add buckets for `staging` or other deployment environments. +Multiple developers can share the same development buckets because +files in this buckets are namespaced by the hostname of the +developer's machine. + +Hint: Its better to use dashes `-` than dots `.`. Dots can cause +trouble when using https. + +### Bucket Configuration + +* Configure static website hosting for each bucket. + + * Go to Properties -> Static Website Hosting + * Click "Enable Static Website Hosting". + * Enter arbitrary string into the field "Index Document" + (i.e. "foo", you can't save without this) + * Click "Save" + +* Grant access to the buckets by adding + [bucket policies](./setting_up_s3_bucket_policies.md). + + * Our policies also enable public read access to your bucket. This + is required. + * Go to Properties -> Permissions. + * Click "Bucket Policy". + * Copy the correct JSON snippet from + [bucket policies](./setting_up_s3_bucket_policies.md) (main or + output) + * Replace the \
or \ placeholders with + the full bucket name. + * Click "Save" + +* You need to enable Cross Origin Resource Sharing (CORS). + + * Still in the Permissions screen, click _CORS Configuration_. + * Paste the code below. This grants everyone GET requests, which is + what Pageflow needs. + +``` + + + * + GET + 28800 + * + + +``` + +You can tweak these rules if you want to. See the +[Amazon Developer Guide](https://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html#how-do-i-enable-cors). + +### Bandwidth Detection + +Pageflow measures the download times of some static files to detect +the bandwidth of the client. + +The following files have to be placed in the Output ("-out") bucket: + + * [app/assets/images/pageflow/bandwidth_probe_large.png](https://raw.githubusercontent.com/codevise/pageflow/master/app/assets/images/pageflow/bandwidth_probe_large.png) + * [app/assets/images/pageflow/bandwidth_probe_small.png](https://raw.githubusercontent.com/codevise/pageflow/master/app/assets/images/pageflow/bandwidth_probe_small.png). + +### Configuring Paperclip + +Pageflow uses the +[Paperclip](https://github.com/thoughtbot/paperclip/) gem, to upload +files to S3. Edit the `paperclip_s3_default_options` settings in +`config/initializers/pageflow.rb`: + +```ruby + config.paperclip_s3_default_options.merge!( + s3_credentials: { + bucket: 'com-example-pageflow-development', + access_key_id: 'xxx', + secret_access_key: 'xxx', + s3_host_name: 's3-eu-west-1.amazonaws.com' + }, + s3_host_alias: 'com-example-pageflow.s3-website-eu-west-1.amazonaws.com', + s3_protocol: 'http' + ) +``` + +The required options are: + +* `bucket`: The name of the main S3 bucket you chose above. + +* `access_key_id`/`secret_access_key`: + [IAM](http://aws.amazon.com/de/iam/) credentials that grant write + access to your main bucket. + +* `s3_host_name`: The host name that shall be used on the server side + to connect to the S3 REST API to upload files. The correct value + depends on the region of your bucket. See the list of + [S3 AWS endpoints](http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) + for more information. **Important**: Be sure NOT to enter an S3 + Website endpoint (`s3-website-*.amazonaws.com`) here. Those enpoints + can only be used for read access to files from your buckets. + +* `s3_host_alias`: The host name that shall be used for image URLs in + your published entries. This can be the hostname of some CDN you + have configured to deliver files from your bucket. Even though use + of a CDN is strongly recommended, you can also use an S3 Website + endpoint here, as in the example above. + +* `s3_protocol`: The protocol to use for public image URLs. Note that + S3 Website endpoints only support `http`. + +Refer to the +[Paperclip documentation](http://www.rubydoc.info/gems/paperclip/Paperclip/Storage/S3) +for a full list of supported options. + +## Zencoder + +Get your API Key from [Zencoder](https://zencoder.com) and make sure +it is used in the `zencoder_options` in +`config/initializers/pageflow.rb`: + +```ruby + config.zencoder_options.merge!( + api_key: 'xxx', + output_bucket: 'com-example-pageflow-out', + s3_host_alias: 'com-example-pageflow-out.s3-website-eu-west-1.amazonaws.com', + s3_protocol: 'http', + attachments_version: 'v1' + ) +``` + +Just like in the previous section, the `s3_host_alias` and +`s3_protocol` settings are used to build video and audio URLs inside +your published entries. + +Note that Zencoder offers a special "Integration API Key" that can be +used free of charge during development. Encoded files are cropped at +five seconds. + +## Amazon Cloudfront + +Create three distributions: + +- one for each of the two S3 buckets +- one for which has the rails app itself as origin + +Configure CNAMES and make sure they are used in +`config/initializers/pageflow.rb` as `s3_host_alias` in production +mode. Please see the inline docs and examples in +`config/initializers/pageflow.rb`. diff --git a/doc/setting_up_s3_bucket_policies.md b/doc/setting_up_s3_bucket_policies.md new file mode 100644 index 0000000000..85c7fa280d --- /dev/null +++ b/doc/setting_up_s3_bucket_policies.md @@ -0,0 +1,95 @@ +# Setting up S3 Bucket Policies + +For Zencoder to be able to access S3 buckets of your AWS account, the +following bucket policies have to be configured. In addition we're +granting public read access to the entire bucket, which is needed +before we can start using S3 as a web server. + +Note that `
` and `` have to be replaced +with the correct bucket names below. + +Grant read access to main bucket: + + { + "Version": "2017-06-20", + "Id": "PageflowMainBucketPolicy", + "Statement": [ + { + "Sid": "Stmt1497951043738", + "Action": [ + "s3:GetObject" + ], + "Effect": "Allow", + "Resource": "arn:aws:s3:::
/*", + "Principal": "*" + }, + { + "Sid": "Stmt1295042087538", + "Effect": "Allow", + "Principal": { + "AWS": "arn:aws:iam::395540211253:root" + }, + "Action": "s3:GetObject", + "Resource": "arn:aws:s3:::
/*" + }, + { + "Sid": "Stmt1295042087538", + "Effect": "Allow", + "Principal": { + "AWS": "arn:aws:iam::395540211253:root" + }, + "Action": [ + "s3:ListBucketMultipartUploads", + "s3:GetBucketLocation" + ], + "Resource": "arn:aws:s3:::
" + } + ] + } + +Grant full access to output bucket: + + { + "Version": "2017-06-20", + "Id": "PageflowOutputBucketPolicy", + "Statement": [ + { + "Sid": "Stmt1497951043738", + "Action": [ + "s3:GetObject" + ], + "Effect": "Allow", + "Resource": "arn:aws:s3:::/*", + "Principal": "*" + }, + { + "Sid": "Stmt1295042087538", + "Effect": "Allow", + "Principal": { + "AWS": "arn:aws:iam::395540211253:root" + }, + "Action": [ + "s3:GetObject", + "s3:PutObjectAcl", + "s3:ListMultipartUploadParts", + "s3:PutObject" + ], + "Resource": "arn:aws:s3:::/*" + }, + { + "Sid": "Stmt1295042087538", + "Effect": "Allow", + "Principal": { + "AWS": "arn:aws:iam::395540211253:root" + }, + "Action": [ + "s3:ListBucketMultipartUploads", + "s3:GetBucketLocation" + ], + "Resource": "arn:aws:s3:::" + } + ] + } + +See also +[Using Zencoder with S3](https://app.zencoder.com/docs/guides/getting-started/working-with-s3)