Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Add Aliyun OSS storage option. #419

Closed
wants to merge 5 commits into from

2 participants

@huacnlee

Aliyun OSS is one of most used storage solutions in China, like S3.

We need store files in local storage host, it will faster that Dropbox, S3...

This pull request add :aliyun option for storage.

Usage:

store_with Aliyun do |aliyun|
  aliyun.access_key_id = 'my_access_id'
  aliyun.access_key_secret = 'my_access_key'
  aliyun.bucket = 'foo'
  aliyun.path = ''
  aliyun.keep = 2
end
@benmccann

I think it might be better to call the class Aliyun or AliyunOSS instead of OSS.

Also, you will need to add a template under templates/cli/storage

@huacnlee

@benmccann Storage name changed.

@benmccann

You probably have to change spec/cli_spec.rb as well

@huacnlee

@benmccann cli test has fixed.

@ghost Show outdated diff Hide outdated diff Unknown commented on an outdated diff
lib/backup/storage/aliyun.rb
@@ -0,0 +1,55 @@
+Backup::Dependency.load('carrierwave-aliyun')
+
+module Backup
+ module Storage
+ class Aliyun < Base
+ attr_accessor :bucket,:access_key_id,:access_key_secret, :path
+
+ attr_deprecate "carrierwave-aliyun", :version => '>= 0.1.3'
@ghost
ghost added a note

???

@huacnlee
huacnlee added a note

Because it need that to access Aliyun OSS

@ghost
ghost added a note

Ok. Will look into it as soon as I get a chance.

Hi @burns , Why did not merge this pull request.
We want this feature to store backup files in locally Cloud hosting, to instead store my backup in Dropbox, because our Web Host in China connection to Dropbox is so hard, so slow.

And I want to write the another Cloud hosting for Backup store interface.

Would you please tell me the reason of not merge this pull request in you free time? and then I can fix it.

@ghost
ghost added a note

All Storages, including their tests, have been updated since this PR. So it needs to be re-written.
But I would still like for you to explain the purpose of Line#8.

Line#8 is a mistake use, I will fix it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@huacnlee

@burns New fixes has committed, and this has it test in stroage/aliyun_spec.rb

@huacnlee

Not need merge this pull_request now, I found a new way to solution this.

I write to plugin for Backup

Just only install them and require in Backup model file.

@huacnlee huacnlee closed this
@ghost

OK. Just so you're aware, calling file.read could consume an enormous amount of memory.
And after reviewing rest-client, even if you were to only pass the file as the "payload",
rest-client will simply do the same (call #read). It's simply not designed for large file uploads.

@huacnlee

@burns It's have to use file.read , because Aliyun OSS API need send file content MD5 digest and Content-Length, so...

@huacnlee

Digest::MD5.file(File.open("xxx")) can instead file.read to MD5, now large file no problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Mar 12, 2013
  1. @huacnlee
Commits on Mar 13, 2013
  1. @huacnlee
  2. @huacnlee

    Fix cli test.

    huacnlee authored
Commits on Sep 13, 2013
  1. @huacnlee

    merge from main repo.

    huacnlee authored
  2. @huacnlee
This page is out of date. Refresh to see the latest.
View
1  README.md
@@ -87,6 +87,7 @@ Once your final backup package is ready, you can use any number of the following
- Dropbox Web Service
- Remote Servers _(Available Protocols: FTP, SFTP, SCP and RSync)_
- Local Storage _(including network mounted locations)_
+- Aliyun (OSS)
All of the above Storages _(except RSync)_ support:
View
1  lib/backup.rb
@@ -47,6 +47,7 @@ module Storage
autoload :SCP, File.join(STORAGE_PATH, 'scp')
autoload :RSync, File.join(STORAGE_PATH, 'rsync')
autoload :Local, File.join(STORAGE_PATH, 'local')
+ autoload :Aliyun, File.join(STORAGE_PATH, 'aliyun')
end
##
View
2  lib/backup/config.rb
@@ -111,7 +111,7 @@ def add_dsl_constants!
['MySQL', 'PostgreSQL', 'MongoDB', 'Redis', 'Riak'],
# Storages
['S3', 'CloudFiles', 'Ninefold', 'Dropbox', 'FTP',
- 'SFTP', 'SCP', 'RSync', 'Local'],
+ 'SFTP', 'SCP', 'RSync', 'Local', "Aliyun"],
# Compressors
['Gzip', 'Bzip2', 'Custom', 'Pbzip2', 'Lzma'],
# Encryptors
View
53 lib/backup/storage/aliyun.rb
@@ -0,0 +1,53 @@
+begin
+ require "carrierwave-aliyun"
+rescue LoadError
+ puts "Aliyun OSS storage gem has not install\n\n"
+ puts "gem install carrierwave-aliyun"
+end
+
+module Backup
+ module Storage
+ class Aliyun < Base
+ attr_accessor :bucket,:access_key_id,:access_key_secret, :path
+
+ def initialize(model, storage_id = nil, &block)
+ super(model, storage_id)
+
+ @path ||= 'backups'
+
+ instance_eval(&block) if block_given?
+ end
+
+ private
+
+ def connection
+ return @connection if @connection
+ opts = {
+ :aliyun_access_id => self.access_key_id,
+ :aliyun_access_key => self.access_key_secret,
+ :aliyun_bucket => self.bucket
+ }
+ @connection = CarrierWave::Storage::Aliyun::Connection.new(opts)
+ end
+
+ def transfer!
+ remote_path = remote_path_for(@package)
+
+ @package.filenames.each do |filename|
+ src = File.join(Config.tmp_path, filename)
+ dest = File.join(remote_path, filename)
+ Logger.info "#{storage_name} uploading '#{ dest }'..."
+ File.open(src, 'r') do |file|
+ connection.put(dest, file.read)
+ end
+ end
+ end
+
+ def remove!(package)
+ remote_path = remote_path_for(package)
+ Logger.info "#{storage_name} removing '#{remote_path}'..."
+ connection.delete(remote_path)
+ end
+ end
+ end
+end
View
172 spec/storage/aliyun_spec.rb
@@ -0,0 +1,172 @@
+# encoding: utf-8
+
+require File.expand_path('../../spec_helper.rb', __FILE__)
+
+describe Backup::Storage::Aliyun do
+ let(:model) { Backup::Model.new(:test_trigger, 'test label') }
+ let(:storage) do
+ Backup::Storage::Aliyun.new(model) do |db|
+ db.access_key_id = 'my_access_id'
+ db.access_key_secret = 'my_access_key'
+ db.bucket = 'foo'
+ db.keep = 5
+ end
+ end
+
+ it 'should be a subclass of Storage::Base' do
+ Backup::Storage::Aliyun.
+ superclass.should == Backup::Storage::Base
+ end
+
+ describe '#initialize' do
+ after { Backup::Storage::Aliyun.clear_defaults! }
+
+ it 'should load pre-configured defaults through Base' do
+ Backup::Storage::Aliyun.any_instance.expects(:load_defaults!)
+ storage
+ end
+
+ it 'should pass the model reference to Base' do
+ storage.instance_variable_get(:@model).should == model
+ end
+
+ it 'should pass the storage_id to Base' do
+ storage = Backup::Storage::Aliyun.new(model, 'my_storage_id')
+ storage.storage_id.should == 'my_storage_id'
+ end
+
+ context 'when no pre-configured defaults have been set' do
+ it 'should use the values given' do
+ storage.access_key_id.should == 'my_access_id'
+ storage.access_key_secret.should == 'my_access_key'
+ storage.bucket.should == "foo"
+ storage.path.should == 'backups'
+
+ storage.storage_id.should be_nil
+ storage.keep.should == 5
+ end
+
+ it 'should use default values if none are given' do
+ storage = Backup::Storage::Aliyun.new(model)
+ storage.access_key_id.should be_nil
+ storage.access_key_secret.should be_nil
+ storage.bucket.should be_nil
+ storage.path.should == 'backups'
+
+ storage.storage_id.should be_nil
+ storage.keep.should be_nil
+ end
+ end # context 'when no pre-configured defaults have been set'
+
+ context 'when pre-configured defaults have been set' do
+ before do
+ Backup::Storage::Aliyun.defaults do |s|
+ s.access_key_id = 'some_api_key'
+ s.access_key_secret = 'some_api_secret'
+ s.bucket = 'some_bucket'
+ s.path = 'some_path'
+ s.keep = 15
+ end
+ end
+
+ it 'should use pre-configured defaults' do
+ storage = Backup::Storage::Aliyun.new(model)
+
+ storage.access_key_id.should == 'some_api_key'
+ storage.access_key_secret.should == 'some_api_secret'
+ storage.bucket.should == 'some_bucket'
+ storage.path.should == 'some_path'
+
+ storage.storage_id.should be_nil
+ storage.keep.should == 15
+ end
+
+ it 'should override pre-configured defaults' do
+ storage = Backup::Storage::Aliyun.new(model) do |s|
+ s.access_key_id = 'new_api_key'
+ s.access_key_secret = 'new_api_secret'
+ s.bucket = 'new_bucket'
+ s.path = 'new_path'
+ s.keep = 10
+ end
+
+ storage.access_key_id.should == 'new_api_key'
+ storage.access_key_secret.should == 'new_api_secret'
+ storage.bucket.should == 'new_bucket'
+ storage.path.should == 'new_path'
+
+ storage.storage_id.should be_nil
+ storage.keep.should == 10
+ end
+ end # context 'when pre-configured defaults have been set'
+ end # describe '#initialize'
+
+ describe '#transfer!' do
+ let(:connection) { mock }
+ let(:package) { mock }
+ let(:file) { mock }
+ let(:s) { sequence '' }
+
+ before do
+ storage.instance_variable_set(:@package, package)
+ storage.stubs(:storage_name).returns('Storage::Aliyun')
+ Backup::Config.stubs(:tmp_path).returns('/local/path')
+ storage.stubs(:connection).returns(connection)
+ file.stubs(:read).returns("foo")
+ end
+
+ it 'should transfer the package files' do
+
+ storage.expects(:remote_path_for).in_sequence(s).with(package).
+ returns('remote/path')
+ package.stubs(:filenames).returns(["backup.tar.enc-aa","backup.tar.enc-ab"])
+ # first yield
+ Backup::Logger.expects(:info).in_sequence(s).with(
+ "Storage::Aliyun uploading 'remote/path/backup.tar.enc-aa'..."
+ )
+ File.expects(:open).in_sequence(s).with(
+ File.join('/local/path', 'backup.tar.enc-aa'), 'r'
+ ).yields(file)
+ connection.expects(:put).in_sequence(s).with(
+ File.join('remote/path', 'backup.tar.enc-aa'), file.read
+ )
+ # second yield
+ Backup::Logger.expects(:info).in_sequence(s).with(
+ "Storage::Aliyun uploading 'remote/path/backup.tar.enc-ab'..."
+ )
+ File.expects(:open).in_sequence(s).with(
+ File.join('/local/path', 'backup.tar.enc-ab'), 'r'
+ ).yields(file)
+ connection.expects(:put).in_sequence(s).with(
+ File.join('remote/path', 'backup.tar.enc-ab'), file.read
+ )
+
+ storage.send(:transfer!)
+ end
+ end # describe '#transfer!'
+
+ describe '#remove!' do
+ let(:package) { mock }
+ let(:connection) { mock }
+ let(:s) { sequence '' }
+
+ before do
+ storage.stubs(:storage_name).returns('Storage::Aliyun')
+ storage.stubs(:connection).returns(connection)
+ end
+
+ it 'should remove the package files' do
+ storage.expects(:remote_path_for).in_sequence(s).with(package).
+ returns('remote/path')
+ # after both yields
+ Backup::Logger.expects(:info).in_sequence(s).with(
+ "Storage::Aliyun removing 'remote/path'..."
+ )
+ connection.expects(:delete).in_sequence(s).with('remote/path')
+
+ storage.send(:remove!, package)
+ end
+ end # describe '#remove!'
+
+
+end
View
21 templates/cli/storage/aliyun
@@ -0,0 +1,21 @@
+ ##
+ # Aliyun OSS File Hosting Service [Storage]
+ #
+ # Access Type:
+ #
+ # - :aliyun
+ #
+ # Note:
+ #
+ # Initial backup must be performed manually to authorize
+ # this machine with your Aliyun OSS account.
+ #
+ store_with Aliyun do |db|
+ # You can found you access key in here:
+ # http://i.aliyun.com/access_key
+ db.access_key_id = "my_access_key_id"
+ db.access_key_secret = "my_access_key_secret"
+ db.bucket = "my_bucket"
+ db.path = "/path/to/my/backups"
+ db.keep = 25
+ end
Something went wrong with that request. Please try again.