Skip to content
Browse files

Adding an SSH/SCP backend for uploading to remote media servers.

  • Loading branch information...
1 parent 6bea121 commit 72bc37a6760eff3656f117121270b325fa5e04cd @JEG2 committed Oct 27, 2010
View
3 CHANGELOG
@@ -1,3 +1,6 @@
+* Oct 27 2010 *
+* Added an SSH/SCP backend for managing a remote media server [James Edward Gray II]
+
* Apr 17 2008 *
* amazon_s3.yml is now passed through ERB before being passed to AWS::S3 [François Beausoleil]
View
58 README
@@ -14,45 +14,46 @@ There are four storage options for files uploaded through attachment_fu:
Database file
Amazon S3
Rackspace (Mosso) Cloud Files
+ SSH/SCP Remote Media Server
-Each method of storage many options associated with it that will be covered in the following section. Something to note, however, is that the Amazon S3 storage requires you to modify config/amazon_s3.yml, the Rackspace Cloud Files storage requires you to modify config/rackspace_cloudfiles.yml, and the Database file storage requires an extra table.
+Each method of storage many options associated with it that will be covered in the following section. Something to note, however, is that the Amazon S3 storage requires you to modify config/amazon_s3.yml, the Rackspace Cloud Files storage requires you to modify config/rackspace_cloudfiles.yml, the Database file storage requires an extra table, and the SSH/SCP Remote Media Server storage requires you to modify config/ssh.yml.
attachment_fu models
====================
-For all three of these storage options a table of metadata is required. This table will contain information about the file (hence the 'meta') and its location. This table has no restrictions on naming, unlike the extra table required for database storage, which must have a table name of db_files (and by convention a model of DbFile).
+For all of these storage options a table of metadata is required. This table will contain information about the file (hence the 'meta') and its location. This table has no restrictions on naming, unlike the extra table required for database storage, which must have a table name of db_files (and by convention a model of DbFile).
In the model there are two methods made available by this plugins: has_attachment and validates_as_attachment.
has_attachment(options = {})
This method accepts the options in a hash:
- :content_type # Allowed content types.
- # Allows all by default. Use :image to allow all standard image types.
- :min_size # Minimum size allowed.
- # 1 byte is the default.
- :max_size # Maximum size allowed.
- # 1.megabyte is the default.
- :size # Range of sizes allowed.
- # (1..1.megabyte) is the default. This overrides the :min_size and :max_size options.
- :resize_to # Used by RMagick to resize images.
- # Pass either an array of width/height, or a geometry string.
- :thumbnails # Specifies a set of thumbnails to generate.
- # This accepts a hash of filename suffixes and RMagick resizing options.
- # This option need only be included if you want thumbnailing.
- :thumbnail_class # Set which model class to use for thumbnails.
- # This current attachment class is used by default.
- :path_prefix # Path to store the uploaded files in.
- # Uses public/#{table_name} by default for the filesystem, and just #{table_name} for the S3 and Cloud Files backend.
- # Setting this sets the :storage to :file_system.
- :partition # Whether to partiton files in directories like /0000/0001/image.jpg. Default is true. Only applicable to the :file_system backend.
- :storage # Specifies the storage system to use..
- # Defaults to :db_file. Options are :file_system, :db_file, :s3, and :cloud_files.
- :cloudfront # If using S3 for storage, this option allows for serving the files via Amazon CloudFront.
- # Defaults to false.
- :processor # Sets the image processor to use for resizing of the attached image.
- # Options include ImageScience, Rmagick, and MiniMagick. Default is whatever is installed.
- :uuid_primary_key # If your model's primary key is a 128-bit UUID in hexadecimal format, then set this to true.
+ :content_type # Allowed content types.
+ # Allows all by default. Use :image to allow all standard image types.
+ :min_size # Minimum size allowed.
+ # 1 byte is the default.
+ :max_size # Maximum size allowed.
+ # 1.megabyte is the default.
+ :size # Range of sizes allowed.
+ # (1..1.megabyte) is the default. This overrides the :min_size and :max_size options.
+ :resize_to # Used by RMagick to resize images.
+ # Pass either an array of width/height, or a geometry string.
+ :thumbnails # Specifies a set of thumbnails to generate.
+ # This accepts a hash of filename suffixes and RMagick resizing options.
+ # This option need only be included if you want thumbnailing.
+ :thumbnail_class # Set which model class to use for thumbnails.
+ # This current attachment class is used by default.
+ :path_prefix # Path to store the uploaded files in.
+ # Uses public/#{table_name} by default for the filesystem, and just #{table_name} for the S3 and Cloud Files backend.
+ # Setting this sets the :storage to :file_system.
+ :partition # Whether to partiton files in directories like /0000/0001/image.jpg. Default is true. Only applicable to the :file_system backend.
+ :storage # Specifies the storage system to use..
+ # Defaults to :db_file. Options are :file_system, :db_file, :s3, :cloud_files, and :ssh.
+ :cloudfront # If using S3 for storage, this option allows for serving the files via Amazon CloudFront.
+ # Defaults to false.
+ :processor # Sets the image processor to use for resizing of the attached image.
+ # Options include ImageScience, Rmagick, and MiniMagick. Default is whatever is installed.
+ :uuid_primary_key # If your model's primary key is a 128-bit UUID in hexadecimal format, then set this to true.
:association_options # attachment_fu automatically defines associations with thumbnails with has_many and belongs_to. If there are any additional options that you want to pass to these methods, then specify them here.
@@ -72,6 +73,7 @@ has_attachment(options = {})
has_attachment :storage => :s3
has_attachment :store => :s3, :cloudfront => true
has_attachment :storage => :cloud_files
+ has_attachment :storage => :ssh
validates_as_attachment
This method prevents files outside of the valid range (:min_size to :max_size, or the :size range) from being saved. It does not however, halt the upload of such files. They will be uploaded into memory regardless of size before validation.
View
4 install.rb
@@ -4,4 +4,6 @@
FileUtils.cp File.dirname(__FILE__) + '/amazon_s3.yml.tpl', s3_config unless File.exist?(s3_config)
cloudfiles_config = File.dirname(__FILE__) + '/../../../config/rackspace_cloudfiles.yml'
FileUtils.cp File.dirname(__FILE__) + '/rackspace_cloudfiles.yml.tpl', cloudfiles_config unless File.exist?(cloudfiles_config)
-puts IO.read(File.join(File.dirname(__FILE__), 'README'))
+ssh_config = File.dirname(__FILE__) + '/../../../config/ssh.yml'
+FileUtils.cp File.dirname(__FILE__) + '/ssh.yml.tpl', ssh_config unless File.exist?(ssh_config)
+puts IO.read(File.join(File.dirname(__FILE__), 'README'))
View
7 lib/technoweenie/attachment_fu.rb
@@ -83,7 +83,7 @@ def has_attachment(options = {})
options[:thumbnail_class] ||= self
options[:s3_access] ||= :public_read
options[:cloudfront] ||= false
- options[:content_type] = [options[:content_type]].flatten.collect! { |t| t == :image ? Technoweenie::AttachmentFu.content_types : t }.flatten unless options[:content_type].nil?
+ options[:content_type] = [options[:content_type]].flatten.collect! { |t| t == :image ? Technoweenie::AttachmentFu.content_types : t }.flatten unless options[:content_type].nil?
unless options[:thumbnails].is_a?(Hash)
raise ArgumentError, ":thumbnails option should be a hash: e.g. :thumbnails => { :foo => '50x50' }"
@@ -103,9 +103,8 @@ def has_attachment(options = {})
attachment_options[:path_prefix] ||= attachment_options[:file_system_path]
if attachment_options[:path_prefix].nil?
attachment_options[:path_prefix] = case attachment_options[:storage]
- when :s3 then table_name
- when :cloud_files then table_name
- else File.join("public", table_name)
+ when :s3, :cloud_files, :ssh then table_name
+ else File.join("public", table_name)
end
end
attachment_options[:path_prefix] = attachment_options[:path_prefix][1..-1] if options[:path_prefix].first == '/'
View
197 lib/technoweenie/attachment_fu/backends/ssh_backend.rb
@@ -0,0 +1,197 @@
+module Technoweenie # :nodoc:
+ module AttachmentFu # :nodoc:
+ module Backends
+ # Methods for SSH/SCP backed attachments (uploaded to a media server).
+ module SshBackend
+ # A global ID cycled 1 to 4 for asset URL's.
+ def self.cycled_asset_id
+ @cycled_asset_id ||= -1
+ @cycled_asset_id += 1
+ @cycled_asset_id % 4 + 1
+ end
+
+ def self.included(base) #:nodoc:
+ begin
+ require "net/ssh"
+ rescue LoadError
+ raise RequiredLibraryNotFoundError.
+ new("Net::SSH could not be loaded")
+ end
+ begin
+ require "net/scp"
+ rescue LoadError
+ raise RequiredLibraryNotFoundError.
+ new("Net::SCP could not be loaded")
+ end
+
+ class << base
+ attr_accessor :ssh_config
+ end
+ begin
+ ssh_config_path = base.attachment_options.fetch(
+ :ssh_config_path,
+ "#{RAILS_ROOT}/config/ssh.yml"
+ )
+ base.ssh_config = YAML.load(
+ ERB.new(File.read(ssh_config_path)).result
+ )[RAILS_ENV].symbolize_keys
+ rescue
+ # raise ConfigFileNotFoundError.
+ # new("File %s not found" % @@ssh_config_path)
+ end
+
+ base.before_update :rename_file
+ end
+
+ # Gets the full path to the filename (on the server) in this format:
+ #
+ # # This assumes a model name like MyModel
+ # ssh_config_dir/my_models/0000/0005/blah.jpg
+ #
+ # The optional thumbnail argument will output the thumbnail's filename.
+ def full_filename(thumbnail = nil)
+ File.join( *[ self.class.ssh_config[:directory],
+ base_path(thumbnail ? thumbnail_class : self),
+ thumbnail_name_for(thumbnail) ].compact )
+ end
+
+ # The pseudo hierarchy containing the file relative to the SSH
+ # directory. Example: <tt>:table_name/:partitioned_id</tt>.
+ #
+ # If a block is passed, each chunk of the path filtered through that
+ # block for escaping.
+ def base_path(prefix_class = self, &escape)
+ escape ||= lambda { |path| path }
+ File.join( *[ prefix_class.attachment_options[:path_prefix].to_s,
+ *partitioned_id ].map(&escape) )
+ end
+
+ # The attachment ID used in the full path of a file.
+ def attachment_path_id
+ ((respond_to?(:parent_id) and parent_id) or id) or 0
+ end
+
+ # Partitions the ID into an array of path components.
+ #
+ # For example, given an ID of 1, it will return
+ # <tt>["0000", "0001"]</tt>.
+ #
+ # If the id is not an integer, then path partitioning will be performed
+ # by hashing the string value of the id with SHA-512, and splitting the
+ # result into four components. If the id a 128-bit UUID (as set by
+ # <tt>:uuid_primary_key => true</tt>) then it will be split into two
+ # components.
+ #
+ # To turn this off entirely, set <tt>:partition => false</tt>.
+ def partitioned_id
+ if respond_to?(:attachment_options) and
+ attachment_options[:partition] == false
+ [ ]
+ elsif attachment_options[:uuid_primary_key]
+ # Primary key is a 128-bit UUID in hex format.
+ # Split it into 2 components.
+ path_id = attachment_path_id.to_s
+ component1 = path_id[0..15] || "-"
+ component2 = path_id[16..-1] || "-"
+ [component1, component2]
+ else
+ path_id = attachment_path_id
+ if path_id.is_a?(Integer)
+ # Primary key is an integer. Split it after padding it with 0.
+ ("%08d" % path_id).scan(/..../)
+ else
+ # Primary key is a String. Hash it and split it into 4 components.
+ hash = Digest::SHA512.hexdigest(path_id.to_s)
+ [hash[0..31], hash[32..63], hash[64..95], hash[96..127]]
+ end
+ end
+ end
+
+ # Gets the public path to the file (based on the URL from the SSH
+ # config.) The optional thumbnail argument will output the thumbnail's
+ # filename.
+ #
+ # If the SSH URL includes includes a %d, it will be replaced with
+ # cycling ID's from 1-4.
+ def public_filename(thumbnail = nil)
+ [ self.class.ssh_config[:url].to_s %
+ Technoweenie::AttachmentFu::Backends::SshBackend.cycled_asset_id,
+ base_path(thumbnail ? thumbnail_class : self) { |path|
+ ERB::Util.url_encode(path)
+ },
+ ERB::Util.url_encode(thumbnail_name_for(thumbnail)) ].
+ reject(&:blank?).join("/")
+ end
+
+ # Overwrites the base filename writer in order to store the old
+ # filename.
+ def filename=(value)
+ @old_filename = full_filename unless filename.nil? or @old_filename
+ write_attribute :filename, sanitize_filename(value)
+ end
+
+ protected
+
+ # Destroys the file. Called in the after_destroy() callback.
+ def destroy_file
+ start_ssh do |ssh|
+ ssh.exec!("rm #{e full_filename}")
+ ssh.exec!("rm -r #{e File.dirname(File.dirname(full_filename))}")
+ end
+ end
+
+ # Renames the given file before saving.
+ def rename_file
+ return unless @old_filename and @old_filename != full_filename
+ start_ssh do |ssh|
+ if save_attachment?
+ ssh.exec!("rm #{e @old_filename}")
+ else
+ ssh.exec!("mv #{e @old_filename} #{e full_filename}")
+ end
+ end
+ @old_filename = nil
+ true
+ end
+
+ # Saves the file to the file system.
+ def save_to_storage
+ if save_attachment?
+ start_ssh do |ssh|
+ ssh.exec!("mkdir -p #{e File.dirname(full_filename)}")
+ ssh.scp.upload!(temp_path, full_filename)
+ ssh.exec!( "chmod #{attachment_options.fetch(:chmod, '0644')}" +
+ " #{e full_filename}" )
+ end
+ end
+ @old_filename = nil
+ true
+ end
+
+ # Opens an SSH connection to the server based on the configured
+ # settings.
+ def start_ssh(&session)
+ config = self.class.ssh_config
+ Net::SSH.start( config[:host],
+ config[:user],
+ config.fetch(:options, { }),
+ &session )
+ end
+
+ # Escape the passed content before handing it to the shell.
+ def e(str)
+ str.to_s.gsub(/(?=[^a-zA-Z0-9_.\/\-\x7F-\xFF\n])/n, '\\').
+ gsub(/\n/, "'\n'").
+ sub(/^$/, "''")
+ end
+
+ # Returns the current contents of the file.
+ def current_data
+ start_ssh do |ssh|
+ return ssh.scp.download!(full_filename)
+ end
+ end
+ end
+ end
+ end
+end
View
20 ssh.yml.tpl
@@ -0,0 +1,20 @@
+development:
+ host: ssh_development_host
+ user: deploy
+ options: { }
+ directory: /var/uploads
+ url: http://media%d.example.com
+
+test:
+ host: ssh_test_host
+ user: deploy
+ options: { }
+ directory: /var/uploads
+ url: http://media%d.example.com
+
+production:
+ host: ssh_production_host
+ user: deploy
+ options: { }
+ directory: /var/uploads
+ url: http://media%d.example.com
View
18 test/backends/remote/ssh_test.rb
@@ -0,0 +1,18 @@
+require File.expand_path(File.join(File.dirname(__FILE__), '..', '..', 'test_helper'))
+
+class SshTest < ActiveSupport::TestCase
+ def self.test_ssh?
+ true unless ENV["TEST_SSH"] == "false"
+ end
+
+ if test_ssh? and
+ File.exist? File.join( File.dirname(__FILE__),
+ "../../../../../../config/ssh.yml" )
+ include BaseAttachmentTests
+ attachment_model SshAttachment
+ else
+ def test_flunk_ssh
+ puts "SSH config file not loaded, tests not running"
+ end
+ end
+end
View
124 test/base_attachment_tests.rb
@@ -12,66 +12,66 @@ def test_should_create_file_from_uploaded_file
end
end
- # def test_should_create_file_from_merb_temp_file
- # assert_created do
- # attachment = upload_merb_file :filename => '/files/foo.txt'
- # assert_valid attachment
- # assert !attachment.db_file.new_record? if attachment.respond_to?(:db_file)
- # assert attachment.image?
- # assert !attachment.size.zero?
- # #assert_equal 3, attachment.size
- # assert_nil attachment.width
- # assert_nil attachment.height
- # end
- # end
- #
- # def test_reassign_attribute_data
- # assert_created 1 do
- # attachment = upload_file :filename => '/files/rails.png'
- # assert_valid attachment
- # assert attachment.size > 0, "no data was set"
- #
- # attachment.set_temp_data 'wtf'
- # assert attachment.save_attachment?
- # attachment.save!
- #
- # assert_equal 'wtf', attachment_model.find(attachment.id).send(:current_data)
- # end
- # end
- #
- # def test_no_reassign_attribute_data_on_nil
- # assert_created 1 do
- # attachment = upload_file :filename => '/files/rails.png'
- # assert_valid attachment
- # assert attachment.size > 0, "no data was set"
- #
- # attachment.set_temp_data nil
- # assert !attachment.save_attachment?
- # end
- # end
- #
- # def test_should_overwrite_old_contents_when_updating
- # attachment = upload_file :filename => '/files/rails.png'
- # assert_not_created do # no new db_file records
- # use_temp_file 'files/rails.png' do |file|
- # attachment.filename = 'rails2.png'
- # attachment.temp_paths.unshift File.join(fixture_path, file)
- # attachment.save!
- # end
- # end
- # end
- #
- # def test_should_save_without_updating_file
- # attachment = upload_file :filename => '/files/foo.txt'
- # assert_valid attachment
- # assert !attachment.save_attachment?
- # assert_nothing_raised { attachment.save! }
- # end
- #
- # def test_should_handle_nil_file_upload
- # attachment = attachment_model.create :uploaded_data => ''
- # assert_raise ActiveRecord::RecordInvalid do
- # attachment.save!
- # end
- # end
+ def test_should_create_file_from_merb_temp_file
+ assert_created do
+ attachment = upload_merb_file :filename => '/files/foo.txt'
+ assert_valid attachment
+ assert !attachment.db_file.new_record? if attachment.respond_to?(:db_file)
+ assert attachment.image?
+ assert !attachment.size.zero?
+ #assert_equal 3, attachment.size
+ assert_nil attachment.width
+ assert_nil attachment.height
+ end
+ end
+
+ def test_reassign_attribute_data
+ assert_created 1 do
+ attachment = upload_file :filename => '/files/rails.png'
+ assert_valid attachment
+ assert attachment.size > 0, "no data was set"
+
+ attachment.set_temp_data 'wtf'
+ assert attachment.save_attachment?
+ attachment.save!
+
+ assert_equal 'wtf', attachment_model.find(attachment.id).send(:current_data)
+ end
+ end
+
+ def test_no_reassign_attribute_data_on_nil
+ assert_created 1 do
+ attachment = upload_file :filename => '/files/rails.png'
+ assert_valid attachment
+ assert attachment.size > 0, "no data was set"
+
+ attachment.set_temp_data nil
+ assert !attachment.save_attachment?
+ end
+ end
+
+ def test_should_overwrite_old_contents_when_updating
+ attachment = upload_file :filename => '/files/rails.png'
+ assert_not_created do # no new db_file records
+ use_temp_file 'files/rails.png' do |file|
+ attachment.filename = 'rails2.png'
+ attachment.temp_paths.unshift File.join(fixture_path, file)
+ attachment.save!
+ end
+ end
+ end
+
+ def test_should_save_without_updating_file
+ attachment = upload_file :filename => '/files/foo.txt'
+ assert_valid attachment
+ assert !attachment.save_attachment?
+ assert_nothing_raised { attachment.save! }
+ end
+
+ def test_should_handle_nil_file_upload
+ attachment = attachment_model.create :uploaded_data => ''
+ assert_raise ActiveRecord::RecordInvalid do
+ attachment.save!
+ end
+ end
end
View
7 test/fixtures/attachment.rb
@@ -224,3 +224,10 @@ class CloudFilesWithPathPrefixAttachment < CloudFilesAttachment
rescue
puts "S3 error: #{$!}"
end
+
+
+
+class SshAttachment < ActiveRecord::Base
+ has_attachment :storage => :ssh
+ validates_as_attachment
+end
View
12 test/schema.rb
@@ -130,5 +130,17 @@
t.column :type, :string
t.column :aspect_ratio, :float
end
+
+ create_table :ssh_attachments, :force => true do |t|
+ t.column :parent_id, :integer
+ t.column :thumbnail, :string
+ t.column :filename, :string, :limit => 255
+ t.column :content_type, :string, :limit => 255
+ t.column :size, :integer
+ t.column :width, :integer
+ t.column :height, :integer
+ t.column :type, :string
+ t.column :aspect_ratio, :float
+ end
end
View
1 test/test_helper.rb
@@ -84,7 +84,6 @@ def self.test_against_subclass(test_method, klass)
protected
def upload_file(options = {})
use_temp_file options[:filename] do |file|
- p attachment_model.name
att = attachment_model.create :uploaded_data => fixture_file_upload(file, options[:content_type] || 'image/png')
att.reload unless att.new_record?
return att

0 comments on commit 72bc37a

Please sign in to comment.
Something went wrong with that request. Please try again.