Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support Azure Blob storage #363

Merged
merged 11 commits into from Dec 12, 2017
Copy path View file
@@ -33,6 +33,14 @@ gem "asset_sync"
gem "fog-aws"
```

Or, to use Azure Blob storage, configure as this.

``` ruby
gem "asset_sync"
gem "fog-azure-rm"
```


### Extended Installation (Faster sync with turbosprockets)

It's possible to improve **asset:precompile** time if you are using Rails 3.2.x
@@ -64,6 +72,12 @@ Or, to use Google Storage Cloud, configure as this.
config.action_controller.asset_host = "//#{ENV['FOG_DIRECTORY']}.storage.googleapis.com"
```

Or, to use Azure Blob storage, configure as this.

``` ruby
#config/environments/production.rb
config.action_controller.asset_host = "//#{ENV['AZURE_STORAGE_ACCOUNT_NAME']}.blob.core.windows.net/#{ENV['FOG_DIRECTORY']}"
```


On **HTTPS**: the exclusion of any protocol in the asset host declaration above will allow browsers to choose the transport mechanism on the fly. So if your application is available under both HTTP and HTTPS the assets will be served to match.
@@ -73,11 +87,20 @@ On **HTTPS**: the exclusion of any protocol in the asset host declaration above
``` ruby
config.action_controller.asset_host = "//s3.amazonaws.com/#{ENV['FOG_DIRECTORY']}"
```
Or

Or, to use Google Storage Cloud, configure as this.

``` ruby
config.action_controller.asset_host = "//storage.googleapis.com/#{ENV['FOG_DIRECTORY']}"
```

Or, to use Azure Blob storage, configure as this.

``` ruby
#config/environments/production.rb
config.action_controller.asset_host = "//#{ENV['AZURE_STORAGE_ACCOUNT_NAME']}.blob.core.windows.net/#{ENV['FOG_DIRECTORY']}"
```

On **non default S3 bucket region**: If your bucket is set to a region that is not the default US Standard (us-east-1) you must use the first style of url ``//#{ENV['FOG_DIRECTORY']}.s3.amazonaws.com`` or amazon will return a 301 permanently moved when assets are requested. Note the caveat above about bucket names and periods.

If you wish to have your assets sync to a sub-folder of your bucket instead of into the root add the following to your ``production.rb`` file
@@ -167,6 +190,7 @@ Run the included Rake task to generate a starting point.

rails g asset_sync:install --provider=Rackspace
rails g asset_sync:install --provider=AWS
rails g asset_sync:install --provider=AzureRM

The generator will create a Rails initializer at `config/initializers/asset_sync.rb`.

@@ -220,6 +244,7 @@ Run the included Rake task to generate a starting point.

rails g asset_sync:install --use-yml --provider=Rackspace
rails g asset_sync:install --use-yml --provider=AWS
rails g asset_sync:install --use-yml --provider=AzureRM

The generator will create a YAML file at `config/asset_sync.yml`.

@@ -316,12 +341,12 @@ end
The blocks are run when local files are being scanned and uploaded

#### Fog (Required)
* **fog\_provider**: your storage provider *AWS* (S3) or *Rackspace* (Cloud Files) or *Google* (Google Storage)
* **fog\_provider**: your storage provider *AWS* (S3) or *Rackspace* (Cloud Files) or *Google* (Google Storage) or *AzureRM* (Azure Blob)
* **fog\_directory**: your bucket name

#### Fog (Optional)

* **fog\_region**: the region your storage bucket is in e.g. *eu-west-1* (AWS), *ord* (Rackspace)
* **fog\_region**: the region your storage bucket is in e.g. *eu-west-1* (AWS), *ord* (Rackspace), *japanwest* (Azure Blob)
* **fog\_path\_style**: To use buckets with dot in names, check https://github.com/fog/fog/issues/2381#issuecomment-28088524

#### AWS
@@ -338,6 +363,10 @@ The blocks are run when local files are being scanned and uploaded
* **google\_storage\_access\_key\_id**: your Google Storage access key
* **google\_storage\_secret\_access\_key**: your Google Storage access secret

#### Azure Blob
* **azure\_storage\_account\_name**: your Azure Blob access key
* **azure\_storage\_access\_key**: your Azure Blob access secret

#### Rackspace (Optional)

* **rackspace\_auth\_url**: Rackspace auth URL, for Rackspace London use: `https://lon.identity.api.rackspacecloud.com/v2.0`
Copy path View file
@@ -27,6 +27,7 @@ Gem::Specification.new do |s|
s.add_development_dependency "jeweler"

s.add_development_dependency "fog-aws"
s.add_development_dependency "fog-azure-rm"

s.add_development_dependency "uglifier"
s.add_development_dependency "appraisal"
Copy path View file
@@ -42,6 +42,10 @@ class Invalid < StandardError; end
# Google Storage
attr_accessor :google_storage_secret_access_key, :google_storage_access_key_id

# Azure Blob with Fog::AzureRM
attr_accessor :azure_storage_account_name
attr_accessor :azure_storage_access_key

validates :existing_remote_files, :inclusion => { :in => %w(keep delete ignore) }

validates :fog_provider, :presence => true
@@ -120,6 +124,10 @@ def google?
fog_provider =~ /google/i
end

def azure_rm?
fog_provider =~ /azurerm/i
end

def cache_asset_regexp=(cache_asset_regexp)
self.cache_asset_regexps = [cache_asset_regexp]
end
@@ -175,6 +183,9 @@ def load_yml!
self.cdn_distribution_id = yml['cdn_distribution_id'] if yml.has_key?("cdn_distribution_id")
self.cache_asset_regexps = yml['cache_asset_regexps'] if yml.has_key?("cache_asset_regexps")

self.azure_storage_account_name = yml['azure_storage_account_name'] if yml.has_key?("azure_storage_account_name")
self.azure_storage_access_key = yml['azure_storage_access_key'] if yml.has_key?("azure_storage_access_key")

# TODO deprecate the other old style config settings. FML.
self.aws_access_key_id = yml["aws_access_key"] if yml.has_key?("aws_access_key")
self.aws_secret_access_key = yml["aws_access_secret"] if yml.has_key?("aws_access_secret")
@@ -222,11 +233,20 @@ def fog_options
:google_storage_secret_access_key => google_storage_secret_access_key,
:google_storage_access_key_id => google_storage_access_key_id
})
elsif azure_rm?
require 'fog/azurerm'
options.merge!({
:azure_storage_account_name => azure_storage_account_name,
:azure_storage_access_key => azure_storage_access_key,
})
options.merge!({
:environment => fog_region
}) if fog_region
else
raise ArgumentError, "AssetSync Unknown provider: #{fog_provider} only AWS, Rackspace and Google are supported currently."
end

options.merge!({:region => fog_region}) if fog_region && !rackspace?
options.merge!({:region => fog_region}) if fog_region && !rackspace? && !azure_rm?

This comment has been minimized.

Copy link
@PikachuEXE

PikachuEXE Dec 11, 2017

Member

Should really put this line into every if except rackspace

This comment has been minimized.

Copy link
@PikachuEXE

PikachuEXE Dec 11, 2017

Member

I can refactor this later you can leave this unchanged :P

This comment has been minimized.

Copy link
@devchick

devchick Dec 11, 2017

Author Contributor

Do you mean like below?

if aws?
  ...
  options.merge!({:region => fog_region}) if fog_region
elsif rackspace?
  ...
  options.merge!({:rackspace_region => fog_region}) if fog_region
elsif google?
  ...
  options.merge!({:region => fog_region}) if fog_region
elsif azure_rm?
  ...
  options.merge!({:environment => fog_region}) if fog_region
else
  ...
end
return options
end

Copy path View file
@@ -31,6 +31,9 @@ class Engine < Rails::Engine
config.google_storage_access_key_id = ENV['GOOGLE_STORAGE_ACCESS_KEY_ID'] if ENV.has_key?('GOOGLE_STORAGE_ACCESS_KEY_ID')
config.google_storage_secret_access_key = ENV['GOOGLE_STORAGE_SECRET_ACCESS_KEY'] if ENV.has_key?('GOOGLE_STORAGE_SECRET_ACCESS_KEY')

config.azure_storage_account_name = ENV['AZURE_STORAGE_ACCOUNT_NAME'] if ENV.has_key?('AZURE_STORAGE_ACCOUNT_NAME')
config.azure_storage_access_key = ENV['AZURE_STORAGE_ACCESS_KEY'] if ENV.has_key?('AZURE_STORAGE_ACCESS_KEY')

config.enabled = (ENV['ASSET_SYNC_ENABLED'] == 'true') if ENV.has_key?('ASSET_SYNC_ENABLED')

config.existing_remote_files = ENV['ASSET_SYNC_EXISTING_REMOTE_FILES'] || "keep"
Copy path View file
@@ -202,6 +202,10 @@ def upload_file(f)
})
end

if config.azure_rm?
file[:content_type] = file[:content_type].content_type if file[:content_type].is_a?(::MIME::Type)

This comment has been minimized.

Copy link
@PikachuEXE

PikachuEXE Dec 11, 2017

Member

Don't quite understand why this is needed

This comment has been minimized.

Copy link
@devchick

devchick Dec 11, 2017

Author Contributor

I'm not sure, sorry...
but, I got an error as below without the content_type conversion.

rake aborted!
NoMethodError: undefined method `length' for #<MIME::Type:0x007fda42e79040>
vendor/bundle/ruby/2.3.0/gems/azure-storage-0.11.5.preview/lib/azure/storage/core/utility.rb:112:in `parse_charset_from_content_type'
vendor/bundle/ruby/2.3.0/gems/azure-storage-0.11.5.preview/lib/azure/storage/blob/blob_service.rb:54:in `call'
vendor/bundle/ruby/2.3.0/gems/azure-storage-0.11.5.preview/lib/azure/storage/blob/block.rb:114:in `create_block_blob'
vendor/bundle/ruby/2.3.0/gems/fog-azure-rm-0.3.8/lib/fog/azurerm/requests/storage/create_block_blob.rb:31:in `create_block_blob'
vendor/bundle/ruby/2.3.0/gems/fog-azure-rm-0.3.8/lib/fog/azurerm/models/storage/file.rb:250:in `save_blob'
vendor/bundle/ruby/2.3.0/gems/fog-azure-rm-0.3.8/lib/fog/azurerm/models/storage/file.rb:93:in `save'
vendor/bundle/ruby/2.3.0/gems/fog-core-1.45.0/lib/fog/core/collection.rb:50:in `create'
vendor/bundle/ruby/2.3.0/bundler/gems/asset_sync-2fa4d474d97e/lib/asset_sync/storage.rb:205:in `upload_file'
vendor/bundle/ruby/2.3.0/bundler/gems/asset_sync-2fa4d474d97e/lib/asset_sync/storage.rb:220:in `block in upload_files'
vendor/bundle/ruby/2.3.0/bundler/gems/asset_sync-2fa4d474d97e/lib/asset_sync/storage.rb:218:in `each'
vendor/bundle/ruby/2.3.0/bundler/gems/asset_sync-2fa4d474d97e/lib/asset_sync/storage.rb:218:in `upload_files'
vendor/bundle/ruby/2.3.0/bundler/gems/asset_sync-2fa4d474d97e/lib/asset_sync/storage.rb:234:in `sync'
vendor/bundle/ruby/2.3.0/bundler/gems/asset_sync-2fa4d474d97e/lib/asset_sync/asset_sync.rb:29:in `block in sync'
vendor/bundle/ruby/2.3.0/bundler/gems/asset_sync-2fa4d474d97e/lib/asset_sync/asset_sync.rb:51:in `with_config'
vendor/bundle/ruby/2.3.0/bundler/gems/asset_sync-2fa4d474d97e/lib/asset_sync/asset_sync.rb:28:in `sync'
vendor/bundle/ruby/2.3.0/bundler/gems/asset_sync-2fa4d474d97e/lib/tasks/asset_sync.rake:5:in `block (2 levels) in <top (required)>'
vendor/bundle/ruby/2.3.0/bundler/gems/asset_sync-2fa4d474d97e/lib/tasks/asset_sync.rake:28:in `block in <top (required)>'

I think Azure::Storage::Core::Utility#parse_charset_from_content_type needs content_type as a String, not as a MIME::Type, because it calls length and split(';') methods of content_type. so, I put the code to convert from MIME::Type to String.

module Azure::Storage
  module Core
    module Utility
      ...
      def parse_charset_from_content_type(content_type)
        if (content_type && content_type.length > 0)
          charset = content_type.split(';').delete_if { |attribute| !attribute.lstrip.start_with?('charset=') }.map { |x| x.lstrip }[0]
          charset['charset='.length...charset.length] if charset
        end
      end

This comment has been minimized.

Copy link
@PikachuEXE

PikachuEXE Dec 12, 2017

Member

I guess add a code comment to briefly explain this

This comment has been minimized.

Copy link
@devchick

devchick Dec 12, 2017

Author Contributor

I added a comment!

end

bucket.files.create( file ) unless ignore
file_handle.close
gzip_file_handle.close if gzip_file_handle
@@ -5,7 +5,7 @@ class InstallGenerator < Rails::Generators::Base

# Commandline options can be defined here using Thor-like options:
class_option :use_yml, :type => :boolean, :default => false, :desc => "Use YML file instead of Rails Initializer"
class_option :provider, :type => :string, :default => "AWS", :desc => "Generate with support for 'AWS', 'Rackspace', or 'Google'"
class_option :provider, :type => :string, :default => "AWS", :desc => "Generate with support for 'AWS', 'Rackspace', 'Google', or 'AzureRM"

def self.source_root
@source_root ||= File.join(File.dirname(__FILE__), 'templates')
@@ -23,6 +23,10 @@ def rackspace?
options[:provider] == 'Rackspace'
end

def azure_rm?
options[:provider] == 'AzureRM'
end

def aws_access_key_id
"<%= ENV['AWS_ACCESS_KEY_ID'] %>"
end
@@ -47,6 +51,14 @@ def rackspace_api_key
"<%= ENV['RACKSPACE_API_KEY'] %>"
end

def azure_storage_account_name
"<%= ENV['AZURE_STORAGE_ACCOUNT_NAME'] %>"
end

def azure_storage_access_key
"<%= ENV['AZURE_STORAGE_ACCESS_KEY'] %>"
end

def app_name
@app_name ||= Rails.application.is_a?(Rails::Application) && Rails.application.class.name.sub(/::Application$/, "").downcase
end
@@ -26,6 +26,12 @@
# if you need to change rackspace_auth_url (e.g. if you need to use Rackspace London)
# config.rackspace_auth_url = "lon.auth.api.rackspacecloud.com"
<%- elsif azure_rm? -%>
config.fog_provider = 'AzureRM'
config.azure_storage_account_name = ENV['AZURE_STORAGE_ACCOUNT_NAME']
config.azure_storage_access_key = ENV['AZURE_STORAGE_ACCESS_KEY']

# config.fog_directory specifies container name of Azure Blob storage
<%- end -%>
config.fog_directory = ENV['FOG_DIRECTORY']
@@ -24,6 +24,12 @@ defaults: &defaults
rackspace_api_key: "<%= rackspace_api_key %>"
# if you need to change rackspace_auth_url (e.g. if you need to use Rackspace London)
# rackspace_auth_url: "https://lon.identity.api.rackspacecloud.com/v2.0"
<%- elsif azure_rm? -%>
fog_provider: 'AzureRM'
azure_storage_account_name: "<%= azure_storage_account_name %>"
azure_storage_access_key: "<%= azure_storage_access_key %>"

# fog_directory specifies container name of Azure Blob storage
<%- end -%>
fog_directory: "<%= app_name %>-assets"
# You may need to specify what region your storage bucket is in
@@ -0,0 +1,19 @@
defaults: &defaults
fog_provider: "AzureRM"
azure_storage_account_name: 'xxxx'
azure_storage_access_key: 'zzzz'

development:
<<: *defaults
fog_directory: "rails_app_development"
existing_remote_files: keep

test:
<<: *defaults
fog_directory: "rails_app_test"
existing_remote_files: keep

production:
<<: *defaults
fog_directory: "rails_app_production"
existing_remote_files: delete
Copy path View file
@@ -0,0 +1,142 @@
require File.dirname(__FILE__) + '/../spec_helper'

describe AssetSync do
include_context "mock Rails without_yml"

describe 'with initializer' do
before(:each) do
AssetSync.config = AssetSync::Config.new
AssetSync.configure do |config|
config.fog_provider = 'AzureRM'
config.azure_storage_account_name = 'aaaa'
config.azure_storage_access_key = 'bbbb'
config.fog_directory = 'mybucket'
config.existing_remote_files = "keep"
end
end

it "should configure provider as AzureRM" do
expect(AssetSync.config.fog_provider).to eq('AzureRM')
expect(AssetSync.config).to be_azure_rm
end

it "should should keep existing remote files" do
expect(AssetSync.config.existing_remote_files?).to eq(true)
end

it "should configure azure_storage_account_name" do
expect(AssetSync.config.azure_storage_account_name).to eq("aaaa")
end

it "should configure azure_storage_access_key" do
expect(AssetSync.config.azure_storage_access_key).to eq("bbbb")
end

it "should configure fog_directory" do
expect(AssetSync.config.fog_directory).to eq("mybucket")
end

it "should configure existing_remote_files" do
expect(AssetSync.config.existing_remote_files).to eq("keep")
end

it "should default gzip_compression to false" do
expect(AssetSync.config.gzip_compression).to be_falsey
end

it "should default manifest to false" do
expect(AssetSync.config.manifest).to be_falsey
end
end

describe 'from yml' do
before(:each) do
set_rails_root('azure_rm_with_yml')
AssetSync.config = AssetSync::Config.new
end

it "should configure azure_storage_account_name" do
expect(AssetSync.config.azure_storage_account_name).to eq("xxxx")
end

it "should configure azure_storage_access_key" do
expect(AssetSync.config.azure_storage_access_key).to eq("zzzz")
end

it "should configure fog_directory" do
expect(AssetSync.config.fog_directory).to eq("rails_app_test")
end

it "should configure existing_remote_files" do
expect(AssetSync.config.existing_remote_files).to eq("keep")
end

it "should default gzip_compression to false" do
expect(AssetSync.config.gzip_compression).to be_falsey
end

it "should default manifest to false" do
expect(AssetSync.config.manifest).to be_falsey
end
end

describe 'with no configuration' do
before(:each) do
AssetSync.config = AssetSync::Config.new
end

it "should be invalid" do
expect{ AssetSync.sync }.to raise_error(::AssetSync::Config::Invalid)
end
end

describe 'with fail_silent configuration' do
before(:each) do
allow(AssetSync).to receive(:stderr).and_return(StringIO.new)
AssetSync.config = AssetSync::Config.new
AssetSync.configure do |config|
config.fail_silently = true
end
end

it "should not raise an invalid exception" do
expect{ AssetSync.sync }.not_to raise_error
end
end

describe 'with gzip_compression enabled' do
before(:each) do
AssetSync.config = AssetSync::Config.new
AssetSync.config.gzip_compression = true
end

it "config.gzip? should be true" do
expect(AssetSync.config.gzip?).to be_truthy
end
end

describe 'with manifest enabled' do
before(:each) do
AssetSync.config = AssetSync::Config.new
AssetSync.config.manifest = true
end

it "config.manifest should be true" do
expect(AssetSync.config.manifest).to be_truthy
end

it "config.manifest_path should default to public/assets.." do
expect(AssetSync.config.manifest_path).to match(/public\/assets\/manifest.yml/)
end

it "config.manifest_path should default to public/assets.." do
Rails.application.config.assets.manifest = "/var/assets"
expect(AssetSync.config.manifest_path).to eq("/var/assets/manifest.yml")
end

it "config.manifest_path should default to public/custom_assets.." do
Rails.application.config.assets.prefix = 'custom_assets'
expect(AssetSync.config.manifest_path).to match(/public\/custom_assets\/manifest.yml/)
end
end
end
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.