Permalink
Browse files

(#329) Create right_aws initial release; publish to RubyForge

git-svn-id: https://wush.net/svn/rightscale/right_aws/release_1_1_0@1514 9f0cbaf6-ce18-0410-ad37-d14a22affa91
  • Loading branch information...
1 parent 8c23098 commit 6f1258fe4d43fc0e441144e96c2c9c6fc1cff066 @trbryan trbryan committed Aug 15, 2007
View
@@ -16,6 +16,6 @@ test/s3/test_helper.rb
test/s3/test_right_s3.rb
test/sqs/test_helper.rb
test/sqs/test_right_sqs.rb
-test/test_right_aws.rb
-test/test_right_aws.rb
+test/ts_right_aws.rb
+test/test_credentials.rb
View
@@ -3,24 +3,74 @@ RightAws
www.RightScale.com
== DESCRIPTION:
+
+RightAws::Ec2 is a Ruby library for the Amazon EC2 (Elastic Compute Cloud)
+service.
+
+RightAws::S3 and RightAws::S3Interface are Ruby libraries for the Amazon S3
+(Simple Storage Service) service.
+
+RightAws::Sqs and RightAws::SqsInterface is a Ruby library for the Amazon SQS (Simple Queue Service)
+service.
+
-FIX (describe your package)
+All RightAws interfaces work in one of two ways:
+1) They use a single persistent HTTP connection per
+process or 2) per Ruby thread. Foir example, it doesn't matter how many RightAws::S3
+objects you create, they all use the same per-program or per-thread
+connection. The purpose of sharing the connection is to keep a single
+persistent HTTP connection open to avoid paying connection
+overhead on every request. However, if you have multiple concurrent
+threads, you may want or need an HTTP connection per thread to enable
+concurrent requests to S3. The way this plays out in practice is:
+1) If you have a non-multithreaded Ruby program, use the non-multithreaded setting for Gem.
+2) If you have a multi-threaded Ruby program, use the multithreaded setting to enable
+concurrent requests to S3 (SQS, EC2).
+3) For running under Mongrel/Rails, use thhe non-multithreaded setting for Gem even though
+Mongrel is multithreaded. This is because only one Rails handler is invoked at
+any time (i.e. it acts like a single-threaded program)
+
+By default, Ec2/S3/Sqs interface instances are created in single-threaded mode. Set
+"params[:multi_thread]" to "true" in the initialization arguments to use
+multithreaded mode.
== FEATURES/PROBLEMS:
-
-* FIX (list of features or problems)
+- Full programmmatic access to Ec2, S3, and Sqs
+- Robust network-level retry layer (using Rightscale::HttpConnection). This includes
+ socket connect and read timeouts and retries.
+- Robust HTTP-level retry layer. Certain (user-adjustable) HTTP errors are
+ classified as temporary errors. These errors are automaticallly retried
+ over exponentially increasing intervals. The number of retries is
+ user-configurable.
+- Support for large S3 list operations. Buckets and key subfolders containing
+ many (> 1000) keys are listed in entirety. Operations based on list (like
+ bucket clear) work on arbitrary numbers of keys.
+- Support for streaming PUTs to S3 if the data source is a file.
+- Support for streaming GETs from S3.
+- Interfaces for HTML link generation.
+
+Known Problems:
+
+- Amazon recently (8/07) changed the semantics of the SQS service. A
+ new queue may not be created within 60 seconds of the destruction of any
+ older queue with the same name. Certain methods of RightAws::Sqs and
+ RightAws::SqsInterface will fail with the message:
+ "AWS.SimpleQueueService.QueueDeletedRecently: You must wait 60 seconds after deleting a queue before you can create another with the same name."
+
== SYNOPSIS:
- FIX (code sample of usage)
+
+
+
== REQUIREMENTS:
-* FIX (list of requirements)
+RightAws requires activesupport and RightScale's right_http_connection gem.
== INSTALL:
-* FIX (sudo gem install, anything else)
+sudo gem install
== LICENSE:
View
@@ -17,4 +17,9 @@ Hoe.new('right_aws', RightAws::VERSION::STRING) do |p|
['right_http_connection','>= 0.1.4']]
end
+task :test do
+ puts "Hi There"
+ require './test/ts_right_aws'
+end
+
# vim: syntax=Ruby
View
@@ -23,27 +23,6 @@
module RightAws
-=begin rdoc
-All RightScale AWS interfaces work in one of two ways:
-1) They use a single persistent HTTP connection per
-process or 2) per Ruby thread. It doesn't matter how many Rightscale::S3
-objects you create, they all use the same per-program or per-thread
-connection. The purpose of sharing the connection is to keep a single
-persistent HTTP connection open to avoid paying connection
-overhead on every request. However, if you have multiple concurrent
-threads, you may want or need an HTTP connection per thread to enable
-concurrent requests to S3. The way this plays out in practice is:
-- If you have a non-multithreaded Ruby program, use the non-multithreaded setting for Gem.
-- If you have a multi-threaded Ruby program, use the multithreaded setting to enable
-concurrent requests to S3 (SQS, EC2).
-- For running under Mongrel/Rails, use thhe non-multithreaded setting for Gem even though
-Mongrel is multithreaded. This is because only one Rails handler is invoked at
-any time (i.e. it acts like a single-threaded program)
-
-By default, S3 instances are created in single-threaded mode. Set
-"params[:multi_thread]" to "true" in the initialization argumnts to use
-multithreaded mode.
-=end
class Ec2
SIGNATURE_VERSION = "1"
View
@@ -55,7 +55,8 @@ module VERSION #:nodoc:
end
end
-# We also want everything available in the Rightscale namespace...
+# We also want everything available in the Rightscale namespace for backward
+# compatibility reasons.
module Rightscale
include RightAws
extend RightAws
View
@@ -23,27 +23,6 @@
module RightAws
-=begin rdoc
-All RightAws AWS interfaces work in one of two ways:
-1) They use a single persistent HTTP connection per
-process or 2) per Ruby thread. It doesn't matter how many RightAws::S3
-objects you create, they all use the same per-program or per-thread
-connection. The purpose of sharing the connection is to keep a single
-persistent HTTP connection open to avoid paying connection
-overhead on every request. However, if you have multiple concurrent
-threads, you may want or need an HTTP connection per thread to enable
-concurrent requests to S3. The way this plays out in practice is:
-- If you have a non-multithreaded Ruby program, use the non-multithreaded setting for Gem.
-- If you have a multi-threaded Ruby program, use the multithreaded setting to enable
-concurrent requests to S3 (SQS, EC2).
-- For running under Mongrel/Rails, use thhe non-multithreaded setting for Gem even though
-Mongrel is multithreaded. This is because only one Rails handler is invoked at
-any time (i.e. it acts like a single-threaded program)
-
-By default, S3 instances are created in single-threaded mode. Set
-"params[:multi_thread]" to "true" in the initialization argumnts to use
-multithreaded mode.
-=end
class S3
attr_reader :interface
@@ -81,7 +81,7 @@ def self.amazon_problems=(problems_list)
# Creates new RightS3 instance.
#
- # s3 = RightS3.new('1E3GDYEOGFJPIT75KDT40','hgTHt68JY07JKUY08ftHYtERkjgtfERn57DFE379', {:multi_thread => true, :logger => Logger.new('/tmp/x.log')}) #=> #<RightS3:0xb7b3c27c>
+ # s3 = RightAws::S3Interface.new('1E3GDYEOGFJPIT7XXXXXX','hgTHt68JY07JKUY08ftHYtERkjgtfERn57XXXXXX', {:multi_thread => true, :logger => Logger.new('/tmp/x.log')}) #=> #<RightS3:0xb7b3c27c>
#
# Params is a hash:
#
@@ -557,10 +557,13 @@ def force_delete_bucket(bucket)
#
def delete_folder(bucket, folder_key, separator='/')
folder_key.chomp!(separator)
+ allkeys = []
incrementally_list_bucket(bucket, { 'prefix' => folder_key }) do |results|
keys = results[:contents].map{ |s3_key| s3_key[:key][/^#{folder_key}($|#{separator}.*)/] ? s3_key[:key] : nil}.compact
keys.each{ |key| delete(bucket, key) }
+ allkeys << keys
end
+ allkeys
rescue
on_exception
end
View
@@ -47,27 +47,6 @@ module RightAws
# grantee2 = queue.grantees('another_cool_guy@email.address')
# grantee2.revoke('SENDMESSAGE')
#
-=begin rdoc
-All RightGrid AWS interface gems work in one of two ways:
-1) They use a single persistent HTTP connection per
-process or 2) per Ruby thread. It doesn't matter how many RightAws::S3
-objects you create, they all use the same per-program or per-thread
-connection. The purpose of sharing the connection is to keep a single
-persistent HTTP connection open to avoid paying connection
-overhead on every request. However, if you have multiple concurrent
-threads, you may want or need an HTTP connection per thread to enable
-concurrent requests to S3. The way this plays out in practice is:
-- If you have a non-multithreaded Ruby program, use the non-multithreaded setting for Gem.
-- If you have a multi-threaded Ruby program, use the multithreaded setting to enable
-concurrent requests to S3 (SQS, EC2).
-- For running under Mongrel/Rails, use thhe non-multithreaded setting for Gem even though
-Mongrel is multithreaded. This is because only one Rails handler is invoked at
-any time (i.e. it acts like a single-threaded program)
-
-By default, S3 instances are created in single-threaded mode. Set
-"params[:multi_thread]" to "true" in the initialization argumnts to use
-multithreaded mode.
-=end
class Sqs
attr_reader :interface
@@ -282,8 +261,8 @@ def grantees(grantee_email_address=nil, permission = nil)
class Message
- attr_reader :queue, :id, :body
- attr_accessor :sent_at, :received_at, :visibility
+ attr_reader :queue, :id, :body, :visibility
+ attr_accessor :sent_at, :received_at
def initialize(queue, id=nil, body=nil, visibility=nil)
@queue = queue
@@ -0,0 +1,2 @@
+require 'test/unit'
+require File.dirname(__FILE__) + '/../../lib/right_aws'
@@ -1,2 +1,2 @@
require 'test/unit'
-require File.dirname(__FILE__) + '/../lib/right_ec2'
+require File.dirname(__FILE__) + '/../../lib/right_aws'
@@ -1,26 +1,21 @@
require File.dirname(__FILE__) + '/test_helper.rb'
require 'pp'
-class TestT < Test::Unit::TestCase
+class TestEc2 < Test::Unit::TestCase
- # Please, change the constants below to your own AWS credentials
-
- USER_AWS_ID = 'xxxxxxxxxxxx'
- AWS_ACCESS_KEY_ID = 'xxxxxxxxxxxxxxxxxxxx'
- AWS_SECRET_ACCESS_KEY = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
-
# Some of RightEc2 instance methods concerning instance launching and image registration
# are not tested here due to their potentially risk.
def setup
- @ec2 = Rightscale::Ec2.new(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
+ @ec2 = Rightscale::Ec2.new(TestCredentials.aws_access_key_id,
+ TestCredentials.aws_secret_access_key)
@key = 'right_ec2_awesome_test_key'
@group = 'right_ec2_awesome_test_security_group'
end
def test_01_create_describe_key_pairs
new_key = @ec2.create_key_pair(@key)
- assert new_key[:aws_material][/BEGIN RSA PRIVATE KEY/], "New key material is abcent"
+ assert new_key[:aws_material][/BEGIN RSA PRIVATE KEY/], "New key material is absent"
keys = @ec2.describe_key_pairs
assert keys.map{|key| key[:aws_key_name] }.include?(@key), "#{@key} must exist"
end
@@ -32,7 +27,7 @@ def test_02_create_security_group
end
def test_03_perms_add
- assert @ec2.authorize_security_group_named_ingress(@group, USER_AWS_ID, 'default')
+ assert @ec2.authorize_security_group_named_ingress(@group, TestCredentials.account_number, 'default')
assert @ec2.authorize_security_group_IP_ingress(@group, 80,80,'udp','192.168.1.0/8')
end
@@ -42,7 +37,8 @@ def test_04_check_new_perms_exist
def test_05_perms_remove
assert @ec2.revoke_security_group_IP_ingress(@group, 80,80,'udp','192.168.1.0/8')
- assert @ec2.revoke_security_group_named_ingress(@group, USER_AWS_ID, 'default')
+ assert @ec2.revoke_security_group_named_ingress(@group,
+ TestCredentials.account_number, 'default')
end
def test_06_describe_images
@@ -58,7 +54,7 @@ def test_07_describe_instanses
assert_raise(Rightscale::AwsError){ @ec2.describe_instances(['i-ABCDEFGH'])}
end
- def test_08_delete_securiry_group
+ def test_08_delete_security_group
assert @ec2.delete_security_group(@group), 'Delete_security_group fail'
end
@@ -1,2 +1,2 @@
require 'test/unit'
-require File.dirname(__FILE__) + '/../lib/right_s3'
+require File.dirname(__FILE__) + '/../../lib/right_aws'
@@ -1,19 +1,15 @@
require File.dirname(__FILE__) + '/test_helper.rb'
-class TestT < Test::Unit::TestCase
-
- # Please, change the constants below to your own AWS credentials
+class TestS3 < Test::Unit::TestCase
RIGHT_OBJECT_TEXT = 'Right test message'
- AWS_ACCESS_KEY_ID = 'XXXXXXXXXXXXXXXXXXXX'
- AWS_SECRET_ACCESS_KEY = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
def setup
- @s3 = Rightscale::S3Interface.new(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
+ @s3 = Rightscale::S3Interface.new(TestCredentials.aws_access_key_id, TestCredentials.aws_secret_access_key)
@bucket = 'right_s3_awesome_test_bucket'
@key1 = 'test/woohoo1'
@key2 = 'test1/key/woohoo2'
- @s = Rightscale::S3.new(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
+ @s = Rightscale::S3.new(TestCredentials.aws_access_key_id, TestCredentials.aws_secret_access_key)
end
#---------------------------
@@ -1,2 +1,2 @@
require 'test/unit'
-require File.dirname(__FILE__) + '/../lib/right_sqs'
+require File.dirname(__FILE__) + '/../../lib/right_aws'
Oops, something went wrong.

0 comments on commit 6f1258f

Please sign in to comment.