Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Browse files

(#329) Create right_aws initial release; publish to RubyForge

git-svn-id: https://wush.net/svn/rightscale/right_aws/release_1_1_0@1514 9f0cbaf6-ce18-0410-ad37-d14a22affa91
  • Loading branch information...
commit 6f1258fe4d43fc0e441144e96c2c9c6fc1cff066 1 parent 8c23098
@trbryan trbryan authored
View
4 Manifest.txt
@@ -16,6 +16,6 @@ test/s3/test_helper.rb
test/s3/test_right_s3.rb
test/sqs/test_helper.rb
test/sqs/test_right_sqs.rb
-test/test_right_aws.rb
-test/test_right_aws.rb
+test/ts_right_aws.rb
+test/test_credentials.rb
View
62 README.txt
@@ -3,24 +3,74 @@ RightAws
www.RightScale.com
== DESCRIPTION:
+
+RightAws::Ec2 is a Ruby library for the Amazon EC2 (Elastic Compute Cloud)
+service.
+
+RightAws::S3 and RightAws::S3Interface are Ruby libraries for the Amazon S3
+(Simple Storage Service) service.
+
+RightAws::Sqs and RightAws::SqsInterface is a Ruby library for the Amazon SQS (Simple Queue Service)
+service.
+
-FIX (describe your package)
+All RightAws interfaces work in one of two ways:
+1) They use a single persistent HTTP connection per
+process or 2) per Ruby thread. Foir example, it doesn't matter how many RightAws::S3
+objects you create, they all use the same per-program or per-thread
+connection. The purpose of sharing the connection is to keep a single
+persistent HTTP connection open to avoid paying connection
+overhead on every request. However, if you have multiple concurrent
+threads, you may want or need an HTTP connection per thread to enable
+concurrent requests to S3. The way this plays out in practice is:
+1) If you have a non-multithreaded Ruby program, use the non-multithreaded setting for Gem.
+2) If you have a multi-threaded Ruby program, use the multithreaded setting to enable
+concurrent requests to S3 (SQS, EC2).
+3) For running under Mongrel/Rails, use thhe non-multithreaded setting for Gem even though
+Mongrel is multithreaded. This is because only one Rails handler is invoked at
+any time (i.e. it acts like a single-threaded program)
+
+By default, Ec2/S3/Sqs interface instances are created in single-threaded mode. Set
+"params[:multi_thread]" to "true" in the initialization arguments to use
+multithreaded mode.
== FEATURES/PROBLEMS:
-
-* FIX (list of features or problems)
+- Full programmmatic access to Ec2, S3, and Sqs
+- Robust network-level retry layer (using Rightscale::HttpConnection). This includes
+ socket connect and read timeouts and retries.
+- Robust HTTP-level retry layer. Certain (user-adjustable) HTTP errors are
+ classified as temporary errors. These errors are automaticallly retried
+ over exponentially increasing intervals. The number of retries is
+ user-configurable.
+- Support for large S3 list operations. Buckets and key subfolders containing
+ many (> 1000) keys are listed in entirety. Operations based on list (like
+ bucket clear) work on arbitrary numbers of keys.
+- Support for streaming PUTs to S3 if the data source is a file.
+- Support for streaming GETs from S3.
+- Interfaces for HTML link generation.
+
+Known Problems:
+
+- Amazon recently (8/07) changed the semantics of the SQS service. A
+ new queue may not be created within 60 seconds of the destruction of any
+ older queue with the same name. Certain methods of RightAws::Sqs and
+ RightAws::SqsInterface will fail with the message:
+ "AWS.SimpleQueueService.QueueDeletedRecently: You must wait 60 seconds after deleting a queue before you can create another with the same name."
+
== SYNOPSIS:
- FIX (code sample of usage)
+
+
+
== REQUIREMENTS:
-* FIX (list of requirements)
+RightAws requires activesupport and RightScale's right_http_connection gem.
== INSTALL:
-* FIX (sudo gem install, anything else)
+sudo gem install
== LICENSE:
View
5 Rakefile
@@ -17,4 +17,9 @@ Hoe.new('right_aws', RightAws::VERSION::STRING) do |p|
['right_http_connection','>= 0.1.4']]
end
+task :test do
+ puts "Hi There"
+ require './test/ts_right_aws'
+end
+
# vim: syntax=Ruby
View
21 lib/ec2/right_ec2.rb
@@ -23,27 +23,6 @@
module RightAws
-=begin rdoc
-All RightScale AWS interfaces work in one of two ways:
-1) They use a single persistent HTTP connection per
-process or 2) per Ruby thread. It doesn't matter how many Rightscale::S3
-objects you create, they all use the same per-program or per-thread
-connection. The purpose of sharing the connection is to keep a single
-persistent HTTP connection open to avoid paying connection
-overhead on every request. However, if you have multiple concurrent
-threads, you may want or need an HTTP connection per thread to enable
-concurrent requests to S3. The way this plays out in practice is:
-- If you have a non-multithreaded Ruby program, use the non-multithreaded setting for Gem.
-- If you have a multi-threaded Ruby program, use the multithreaded setting to enable
-concurrent requests to S3 (SQS, EC2).
-- For running under Mongrel/Rails, use thhe non-multithreaded setting for Gem even though
-Mongrel is multithreaded. This is because only one Rails handler is invoked at
-any time (i.e. it acts like a single-threaded program)
-
-By default, S3 instances are created in single-threaded mode. Set
-"params[:multi_thread]" to "true" in the initialization argumnts to use
-multithreaded mode.
-=end
class Ec2
SIGNATURE_VERSION = "1"
View
3  lib/right_aws.rb
@@ -55,7 +55,8 @@ module VERSION #:nodoc:
end
end
-# We also want everything available in the Rightscale namespace...
+# We also want everything available in the Rightscale namespace for backward
+# compatibility reasons.
module Rightscale
include RightAws
extend RightAws
View
21 lib/s3/right_s3.rb
@@ -23,27 +23,6 @@
module RightAws
-=begin rdoc
-All RightAws AWS interfaces work in one of two ways:
-1) They use a single persistent HTTP connection per
-process or 2) per Ruby thread. It doesn't matter how many RightAws::S3
-objects you create, they all use the same per-program or per-thread
-connection. The purpose of sharing the connection is to keep a single
-persistent HTTP connection open to avoid paying connection
-overhead on every request. However, if you have multiple concurrent
-threads, you may want or need an HTTP connection per thread to enable
-concurrent requests to S3. The way this plays out in practice is:
-- If you have a non-multithreaded Ruby program, use the non-multithreaded setting for Gem.
-- If you have a multi-threaded Ruby program, use the multithreaded setting to enable
-concurrent requests to S3 (SQS, EC2).
-- For running under Mongrel/Rails, use thhe non-multithreaded setting for Gem even though
-Mongrel is multithreaded. This is because only one Rails handler is invoked at
-any time (i.e. it acts like a single-threaded program)
-
-By default, S3 instances are created in single-threaded mode. Set
-"params[:multi_thread]" to "true" in the initialization argumnts to use
-multithreaded mode.
-=end
class S3
attr_reader :interface
View
5 lib/s3/right_s3_interface.rb
@@ -81,7 +81,7 @@ def self.amazon_problems=(problems_list)
# Creates new RightS3 instance.
#
- # s3 = RightS3.new('1E3GDYEOGFJPIT75KDT40','hgTHt68JY07JKUY08ftHYtERkjgtfERn57DFE379', {:multi_thread => true, :logger => Logger.new('/tmp/x.log')}) #=> #<RightS3:0xb7b3c27c>
+ # s3 = RightAws::S3Interface.new('1E3GDYEOGFJPIT7XXXXXX','hgTHt68JY07JKUY08ftHYtERkjgtfERn57XXXXXX', {:multi_thread => true, :logger => Logger.new('/tmp/x.log')}) #=> #<RightS3:0xb7b3c27c>
#
# Params is a hash:
#
@@ -557,10 +557,13 @@ def force_delete_bucket(bucket)
#
def delete_folder(bucket, folder_key, separator='/')
folder_key.chomp!(separator)
+ allkeys = []
incrementally_list_bucket(bucket, { 'prefix' => folder_key }) do |results|
keys = results[:contents].map{ |s3_key| s3_key[:key][/^#{folder_key}($|#{separator}.*)/] ? s3_key[:key] : nil}.compact
keys.each{ |key| delete(bucket, key) }
+ allkeys << keys
end
+ allkeys
rescue
on_exception
end
View
25 lib/sqs/right_sqs.rb
@@ -47,27 +47,6 @@ module RightAws
# grantee2 = queue.grantees('another_cool_guy@email.address')
# grantee2.revoke('SENDMESSAGE')
#
-=begin rdoc
-All RightGrid AWS interface gems work in one of two ways:
-1) They use a single persistent HTTP connection per
-process or 2) per Ruby thread. It doesn't matter how many RightAws::S3
-objects you create, they all use the same per-program or per-thread
-connection. The purpose of sharing the connection is to keep a single
-persistent HTTP connection open to avoid paying connection
-overhead on every request. However, if you have multiple concurrent
-threads, you may want or need an HTTP connection per thread to enable
-concurrent requests to S3. The way this plays out in practice is:
-- If you have a non-multithreaded Ruby program, use the non-multithreaded setting for Gem.
-- If you have a multi-threaded Ruby program, use the multithreaded setting to enable
-concurrent requests to S3 (SQS, EC2).
-- For running under Mongrel/Rails, use thhe non-multithreaded setting for Gem even though
-Mongrel is multithreaded. This is because only one Rails handler is invoked at
-any time (i.e. it acts like a single-threaded program)
-
-By default, S3 instances are created in single-threaded mode. Set
-"params[:multi_thread]" to "true" in the initialization argumnts to use
-multithreaded mode.
-=end
class Sqs
attr_reader :interface
@@ -282,8 +261,8 @@ def grantees(grantee_email_address=nil, permission = nil)
class Message
- attr_reader :queue, :id, :body
- attr_accessor :sent_at, :received_at, :visibility
+ attr_reader :queue, :id, :body, :visibility
+ attr_accessor :sent_at, :received_at
def initialize(queue, id=nil, body=nil, visibility=nil)
@queue = queue
View
2  test/awsbase/test_helper.rb
@@ -0,0 +1,2 @@
+require 'test/unit'
+require File.dirname(__FILE__) + '/../../lib/right_aws'
View
2  test/ec2/test_helper.rb
@@ -1,2 +1,2 @@
require 'test/unit'
-require File.dirname(__FILE__) + '/../lib/right_ec2'
+require File.dirname(__FILE__) + '/../../lib/right_aws'
View
20 test/ec2/test_right_ec2.rb
@@ -1,26 +1,21 @@
require File.dirname(__FILE__) + '/test_helper.rb'
require 'pp'
-class TestT < Test::Unit::TestCase
+class TestEc2 < Test::Unit::TestCase
- # Please, change the constants below to your own AWS credentials
-
- USER_AWS_ID = 'xxxxxxxxxxxx'
- AWS_ACCESS_KEY_ID = 'xxxxxxxxxxxxxxxxxxxx'
- AWS_SECRET_ACCESS_KEY = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
-
# Some of RightEc2 instance methods concerning instance launching and image registration
# are not tested here due to their potentially risk.
def setup
- @ec2 = Rightscale::Ec2.new(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
+ @ec2 = Rightscale::Ec2.new(TestCredentials.aws_access_key_id,
+ TestCredentials.aws_secret_access_key)
@key = 'right_ec2_awesome_test_key'
@group = 'right_ec2_awesome_test_security_group'
end
def test_01_create_describe_key_pairs
new_key = @ec2.create_key_pair(@key)
- assert new_key[:aws_material][/BEGIN RSA PRIVATE KEY/], "New key material is abcent"
+ assert new_key[:aws_material][/BEGIN RSA PRIVATE KEY/], "New key material is absent"
keys = @ec2.describe_key_pairs
assert keys.map{|key| key[:aws_key_name] }.include?(@key), "#{@key} must exist"
end
@@ -32,7 +27,7 @@ def test_02_create_security_group
end
def test_03_perms_add
- assert @ec2.authorize_security_group_named_ingress(@group, USER_AWS_ID, 'default')
+ assert @ec2.authorize_security_group_named_ingress(@group, TestCredentials.account_number, 'default')
assert @ec2.authorize_security_group_IP_ingress(@group, 80,80,'udp','192.168.1.0/8')
end
@@ -42,7 +37,8 @@ def test_04_check_new_perms_exist
def test_05_perms_remove
assert @ec2.revoke_security_group_IP_ingress(@group, 80,80,'udp','192.168.1.0/8')
- assert @ec2.revoke_security_group_named_ingress(@group, USER_AWS_ID, 'default')
+ assert @ec2.revoke_security_group_named_ingress(@group,
+ TestCredentials.account_number, 'default')
end
def test_06_describe_images
@@ -58,7 +54,7 @@ def test_07_describe_instanses
assert_raise(Rightscale::AwsError){ @ec2.describe_instances(['i-ABCDEFGH'])}
end
- def test_08_delete_securiry_group
+ def test_08_delete_security_group
assert @ec2.delete_security_group(@group), 'Delete_security_group fail'
end
View
2  test/s3/test_helper.rb
@@ -1,2 +1,2 @@
require 'test/unit'
-require File.dirname(__FILE__) + '/../lib/right_s3'
+require File.dirname(__FILE__) + '/../../lib/right_aws'
View
10 test/s3/test_right_s3.rb
@@ -1,19 +1,15 @@
require File.dirname(__FILE__) + '/test_helper.rb'
-class TestT < Test::Unit::TestCase
-
- # Please, change the constants below to your own AWS credentials
+class TestS3 < Test::Unit::TestCase
RIGHT_OBJECT_TEXT = 'Right test message'
- AWS_ACCESS_KEY_ID = 'XXXXXXXXXXXXXXXXXXXX'
- AWS_SECRET_ACCESS_KEY = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
def setup
- @s3 = Rightscale::S3Interface.new(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
+ @s3 = Rightscale::S3Interface.new(TestCredentials.aws_access_key_id, TestCredentials.aws_secret_access_key)
@bucket = 'right_s3_awesome_test_bucket'
@key1 = 'test/woohoo1'
@key2 = 'test1/key/woohoo2'
- @s = Rightscale::S3.new(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
+ @s = Rightscale::S3.new(TestCredentials.aws_access_key_id, TestCredentials.aws_secret_access_key)
end
#---------------------------
View
2  test/sqs/test_helper.rb
@@ -1,2 +1,2 @@
require 'test/unit'
-require File.dirname(__FILE__) + '/../lib/right_sqs'
+require File.dirname(__FILE__) + '/../../lib/right_aws'
View
37 test/sqs/test_right_sqs.rb
@@ -1,20 +1,16 @@
require File.dirname(__FILE__) + '/test_helper.rb'
-class TestT < Test::Unit::TestCase
+class TestSqs < Test::Unit::TestCase
- # Please, change the constants below to your own AWS credentials
-
- AWS_ACCESS_KEY_ID = 'xxxxxxxxxxxxxxxxxxxx'
- AWS_SECRET_ACCESS_KEY = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
- GRANTEE_EMAIL_ADDRESS = 'x.xxxxxxx@gmail.com'
+ GRANTEE_EMAIL_ADDRESS = 'fester@example.com'
RIGHT_MESSAGE_TEXT = 'Right test message'
def setup
- @sqs = Rightscale::SqsInterface.new(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
+ @sqs = Rightscale::SqsInterface.new(TestCredentials.aws_access_key_id, TestCredentials.aws_secret_access_key)
@queue_name = 'right_sqs_test_awesome_queue'
# for classes
- @s = Rightscale::Sqs.new(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
+ @s = Rightscale::Sqs.new(TestCredentials.aws_access_key_id, TestCredentials.aws_secret_access_key)
end
#---------------------------
@@ -100,7 +96,7 @@ def test_20_sqs_create_delete_queue
# get queues list
queues = @s.queues
# create new queue
- queue = @s.queue(@queue_name, true)
+ queue = @s.queue("#{@queue_name}_20", true)
# check that it is created
assert queue.is_a?(Rightscale::Sqs::Queue)
# check that amount of queues has increased
@@ -111,13 +107,15 @@ def test_20_sqs_create_delete_queue
def test_21_queue_create
# create new queue
- queue = Rightscale::Sqs::Queue.create(@s, @queue_name, true)
+ queue = Rightscale::Sqs::Queue.create(@s, "#{@queue_name}_21", true)
# check that it is created
assert queue.is_a?(Rightscale::Sqs::Queue)
+ # Don't test this - just for cleanup purposes
+ queue.delete
end
def test_22_queue_attributes
- queue = Rightscale::Sqs::Queue.create(@s, @queue_name, false)
+ queue = Rightscale::Sqs::Queue.create(@s, "#{@queue_name}_22", false)
# get a list of attrinutes
attributes = queue.get_attribute
assert attributes.is_a?(Hash) && attributes.size>0
@@ -135,10 +133,12 @@ def test_22_queue_attributes
queue.visibility += 10
# make sure that it is changed
assert v.to_i + 10, queue.visibility
+ # Don't test this - just for cleanup purposes
+ queue.delete
end
def test_23_grantees
- queue = Rightscale::Sqs::Queue.create(@s, @queue_name, false)
+ queue = Rightscale::Sqs::Queue.create(@s, "#{@queue_name}_23", false)
# get a list of grantees
grantees = queue.grantees
# create new grantee
@@ -155,10 +155,12 @@ def test_23_grantees
assert_equal 1, grantee.perms.size
# remove grantee
assert grantee.drop
+ # Don't test this - just for cleanup purposes
+ queue.delete
end
def test_24_send_size
- queue = Rightscale::Sqs::Queue.create(@s, @queue_name, false)
+ queue = Rightscale::Sqs::Queue.create(@s, "#{@queue_name}_24", false)
# send 5 messages
assert queue.push('a1')
assert queue.push('a2')
@@ -171,10 +173,13 @@ def test_24_send_size
assert queue.push('a6')
# check queue size again
assert_equal 6, queue.size
+ # Don't test this - just for cleanup purposes
+ queue.clear
+ queue.delete
end
def test_25_message_receive_pop_peek_delete
- queue = Rightscale::Sqs::Queue.create(@s, @queue_name, false)
+ queue = Rightscale::Sqs::Queue.create(@s, "#{@queue_name}_25", false)
# get queue size
size = queue.size
# get first message
@@ -196,10 +201,12 @@ def test_25_message_receive_pop_peek_delete
assert m1.delete
# make sure that queue size has decreased again
assert_equal size-2, queue.size
+ # Don't test this - just for cleanup purposes
+ queue.delete
end
def test_26
- queue = Rightscale::Sqs::Queue.create(@s, @queue_name, false)
+ queue = Rightscale::Sqs::Queue.create(@s, "#{@queue_name}_26", false)
# lock message
queue.receive(100)
# clear queue
View
0  test/test_right_aws.rb
No changes.
View
9 test/ts_right_aws.rb
@@ -0,0 +1,9 @@
+require 'test/unit'
+require 'test_credentials'
+TestCredentials.get_credentials
+
+require 'awsbase/test_right_awsbase.rb'
+require 'ec2/test_right_ec2.rb'
+require 's3/test_right_s3.rb'
+require 'sqs/test_right_sqs.rb'
+
Please sign in to comment.
Something went wrong with that request. Please try again.