Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

#1760 Create RightAws 1.8.1 release branch

git-svn-id: https://wush.net/svn/rightscale/right_aws/release_1_8_1@5974 9f0cbaf6-ce18-0410-ad37-d14a22affa91
  • Loading branch information...
commit a88f9ac99fc946c5a3ab3932836726d7228c0ef0 2 parents 4559fdc + d8e3e7f
@trbryan trbryan authored
View
31 History.txt
@@ -174,8 +174,33 @@ Initial release.
- Removed the 1.7.2 monkey-patch of the Ruby File class on Windows. This patch broke Rails 2.0.
The patch is now included in the README for anyone to use at their own risk.
-
+
+== 1.8.0
+
+ Release Notes:
+
+ This release adds major new features to RightAws to support Amazon's new
+ Elastic Block Store (EBS). Via the RightAws::Ec2 module, users can create
+ and delete EBS volumes, attach and detach them from instances, snapshot
+ volumes, and list the available volumes and snapshots.
+
+ Bug fixes include correction of RightAws::S3 copy's failure to url-encode
+the source key.
+
== 1.8.1
-* r5111, konstantin, 2008-08-19 16:19:06 +0400
- * #1526, fixed: Right_S3 copy fails to url-encode source
+ Release Notes:
+
+ RightScale::SdbInterface & ::ActiveSdb have several enhancements, including:
+ - RightAws::SdbInterface#last_query_expression added for debug puposes
+ - RightAws::ActiveSdb::Base#query :order and :auto_load options added to support query
+ result sorting and attributes auto loading
+ - RightAws::ActiveSdb::Base#find_all_by_ and find_by_ helpers improved to support
+ :order, :auto_load, :limit and :next_token options
+ - RightAws::SdbInterface#delete_attributes bug fixed
+ - SdbInterface allows specification of a string value to use for
+ representing Ruby nil in SDB.
+ - Sdb tests fixed and improved
+
+ The ::S3 interface now has support for S3's server access logging.
+ Amazon considers server access logging to be a beta or provisional feature.
View
9 README.txt
@@ -5,9 +5,10 @@ For information about RightScale, see http://www.rightscale.com
== DESCRIPTION:
-The RightScale AWS gems have been designed to provide a robust, fast, and secure interface to Amazon EC2, Amazon S3, Amazon SQS, and Amazon SDB. These gems have been used in production by RightScale since late 2006 and are being maintained to track enhancements made by Amazon. The RightScale AWS gems comprise:
+The RightScale AWS gems have been designed to provide a robust, fast, and secure interface to Amazon EC2, EBS, S3, SQS, and SDB. These gems have been used in production by RightScale since late 2006 and are being maintained to track enhancements made by Amazon. The RightScale AWS gems comprise:
-- RightAws::Ec2 -- interface to Amazon EC2 (Elastic Compute Cloud)
+- RightAws::Ec2 -- interface to Amazon EC2 (Elastic Compute Cloud) and the
+ associated EBS (Elastic Block Store)
- RightAws::S3 and RightAws::S3Interface -- interface to Amazon S3 (Simple Storage Service)
- RightAws::Sqs and RightAws::SqsInterface -- interface to first-generation Amazon SQS (Simple Queue Service) (API version 2007-05-01)
- RightAws::SqsGen2 and RightAws::SqsGen2Interface -- interface to second-generation Amazon SQS (Simple Queue Service) (API version 2008-01-01)
@@ -15,7 +16,7 @@ The RightScale AWS gems have been designed to provide a robust, fast, and secure
== FEATURES:
-- Full programmmatic access to EC2, S3, SQS, and SDB.
+- Full programmmatic access to EC2, EBS, S3, SQS, and SDB.
- Complete error handling: all operations check for errors and report complete
error information by raising an AwsError.
- Persistent HTTP connections with robust network-level retry layer using
@@ -123,7 +124,7 @@ multithreaded mode.
== REQUIREMENTS:
-RightAws requires REXML and the RightHttpConnection gem.
+RightAws requires REXML and the right_http_connection gem.
If libxml and its Ruby bindings (distributed in the libxml-ruby gem) are
present, RightAws can be configured to use them:
RightAws::RightAWSParser.xml_lib = 'libxml'
View
18 lib/awsbase/right_awsbase.rb
@@ -24,6 +24,7 @@
# Test
module RightAws
require 'md5'
+ require 'pp'
class AwsUtils #:nodoc:
@@digest = OpenSSL::Digest::Digest.new("sha1")
@@ -44,6 +45,23 @@ def self.URLencode(raw)
e = URI.escape(raw)
e.gsub(/\+/, "%2b")
end
+
+ def self.allow_only(allowed_keys, params)
+ bogus_args = []
+ params.keys.each {|p| bogus_args.push(p) unless allowed_keys.include?(p) }
+ raise AwsError.new("The following arguments were given but are not legal for the function call #{caller_method}: #{bogus_args.inspect}") if bogus_args.length > 0
+ end
+
+ def self.mandatory_arguments(required_args, params)
+ rargs = required_args.dup
+ params.keys.each {|p| rargs.delete(p)}
+ raise AwsError.new("The following mandatory arguments were not provided to #{caller_method}: #{rargs.inspect}") if rargs.length > 0
+ end
+
+ def self.caller_method
+ caller[1]=~/`(.*?)'/
+ $1
+ end
end
View
11 lib/ec2/right_ec2.rb
@@ -25,7 +25,8 @@ module RightAws
# = RightAWS::EC2 -- RightScale Amazon EC2 interface
# The RightAws::EC2 class provides a complete interface to Amazon's
- # Elastic Compute Cloud service.
+ # Elastic Compute Cloud service, as well as the associated EBS (Elastic Block
+ # Store).
# For explanations of the semantics
# of each call, please refer to Amazon's documentation at
# http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=87
@@ -67,7 +68,7 @@ class Ec2 < RightAwsBase
include RightAwsBaseInterface
# Amazon EC2 API version being used
- API_VERSION = "2008-02-01"
+ API_VERSION = "2008-05-05"
DEFAULT_HOST = "ec2.amazonaws.com"
DEFAULT_PATH = '/'
DEFAULT_PROTOCOL = 'https'
@@ -1404,7 +1405,7 @@ def tagend(name)
when 'device' then @result[:aws_device] = @text
when 'status' then @result[:aws_attachment_status] = @text
when 'attachTime' then @result[:aws_attached_at] = Time.parse(@text)
- end
+ end
end
def reset
@result = {}
@@ -1416,8 +1417,8 @@ def tagstart(name, attributes)
case name
when 'item'
case @xmlpath
- when 'DescribeVolumesResponse/volumeSet' then @volume = {}
-end
+ when 'DescribeVolumesResponse/volumeSet' then @volume = {}
+ end
end
end
def tagend(name)
View
38 lib/s3/right_s3.rb
@@ -53,6 +53,14 @@ class S3
# Create a new handle to an S3 account. All handles share the same per process or per thread
# HTTP connection to Amazon S3. Each handle is for a specific account.
# The +params+ are passed through as-is to RightAws::S3Interface.new
+ #
+ # Params is a hash:
+ #
+ # {:server => 's3.amazonaws.com' # Amazon service host: 's3.amazonaws.com'(default)
+ # :port => 443 # Amazon service port: 80 or 443(default)
+ # :protocol => 'https' # Amazon service protocol: 'http' or 'https'(default)
+ # :multi_thread => true|false # Multi-threaded (connection per each thread): true or false(default)
+ # :logger => Logger Object} # Logger instance: logs to STDOUT if omitted }
def initialize(aws_access_key_id=nil, aws_secret_access_key=nil, params={})
@interface = S3Interface.new(aws_access_key_id, aws_secret_access_key, params)
end
@@ -161,6 +169,36 @@ def public_link
def location
@location ||= @s3.interface.bucket_location(@name)
end
+
+ # Retrieves the logging configuration for a bucket.
+ # Returns a hash of {:enabled, :targetbucket, :targetprefix}
+ #
+ # bucket.logging_info()
+ # => {:enabled=>true, :targetbucket=>"mylogbucket", :targetprefix=>"loggylogs/"}
+ def logging_info
+ @s3.interface.get_logging_parse(:bucket => @name)
+ end
+
+ # Enables S3 server access logging on a bucket. The target bucket must have been properly configured to receive server
+ # access logs.
+ # Params:
+ # :targetbucket - either the target bucket object or the name of the target bucket
+ # :targetprefix - the prefix under which all logs should be stored
+ #
+ # bucket.enable_logging(:targetbucket=>"mylogbucket", :targetprefix=>"loggylogs/")
+ # => true
+ def enable_logging(params)
+ AwsUtils.mandatory_arguments([:targetbucket, :targetprefix], params)
+ AwsUtils.allow_only([:targetbucket, :targetprefix], params)
+ xmldoc = "<?xml version=\"1.0\" encoding=\"UTF-8\"?><BucketLoggingStatus xmlns=\"http://doc.s3.amazonaws.com/2006-03-01\"><LoggingEnabled><TargetBucket>#{params[:targetbucket]}</TargetBucket><TargetPrefix>#{params[:targetprefix]}</TargetPrefix></LoggingEnabled></BucketLoggingStatus>"
+ @s3.interface.put_logging(:bucket => @name, :xmldoc => xmldoc)
+ end
+
+ # Disables S3 server access logging on a bucket. Takes no arguments.
+ def disable_logging
+ xmldoc = "<?xml version=\"1.0\" encoding=\"UTF-8\"?><BucketLoggingStatus xmlns=\"http://doc.s3.amazonaws.com/2006-03-01\"></BucketLoggingStatus>"
+ @s3.interface.put_logging(:bucket => @name, :xmldoc => xmldoc)
+ end
# Retrieve a group of keys from Amazon.
# +options+ is a hash: { 'prefix'=>'', 'marker'=>'', 'max-keys'=>5, 'delimiter'=>'' }).
View
198 lib/s3/right_s3_interface.rb
@@ -95,7 +95,7 @@ def canonical_string(method, path, headers={}, expires=nil) # :nodoc:
out_string << '?acl' if path[/[&?]acl($|&|=)/]
out_string << '?torrent' if path[/[&?]torrent($|&|=)/]
out_string << '?location' if path[/[&?]location($|&|=)/]
-# out_string << '?logging' if path[/[&?]logging($|&|=)/] # this one is beta, no support for now
+ out_string << '?logging' if path[/[&?]logging($|&|=)/] # this one is beta, no support for now
out_string
end
@@ -211,6 +211,37 @@ def bucket_location(bucket, headers={})
on_exception
end
+ # Retrieves the logging configuration for a bucket.
+ # Returns a hash of {:enabled, :targetbucket, :targetprefix}
+ #
+ # s3.interface.get_logging_parse(:bucket => "asset_bucket")
+ # => {:enabled=>true, :targetbucket=>"mylogbucket", :targetprefix=>"loggylogs/"}
+ #
+ #
+ def get_logging_parse(params)
+ AwsUtils.mandatory_arguments([:bucket], params)
+ AwsUtils.allow_only([:bucket, :headers], params)
+ params[:headers] = {} unless params[:headers]
+ req_hash = generate_rest_request('GET', params[:headers].merge(:url=>"#{params[:bucket]}?logging"))
+ request_info(req_hash, S3LoggingParser.new)
+ rescue
+ on_exception
+ end
+
+ # Sets logging configuration for a bucket from the XML configuration document.
+ # params:
+ # :bucket
+ # :xmldoc
+ def put_logging(params)
+ AwsUtils.mandatory_arguments([:bucket,:xmldoc], params)
+ AwsUtils.allow_only([:bucket,:xmldoc, :headers], params)
+ params[:headers] = {} unless params[:headers]
+ req_hash = generate_rest_request('PUT', params[:headers].merge(:url=>"#{params[:bucket]}?logging", :data => params[:xmldoc]))
+ request_info(req_hash, S3TrueParser.new)
+ rescue
+ on_exception
+ end
+
# Deletes new bucket. Bucket must be empty! Returns +true+ or an exception.
#
# s3.delete_bucket('my_awesome_bucket') #=> true
@@ -349,6 +380,7 @@ def under_max_keys(internal_options)
# a text mode IO object is passed to PUT, it will be converted to binary
# mode.
#
+
def put(bucket, key, data=nil, headers={})
# On Windows, if someone opens a file in text mode, we must reset it so
# to binary mode for streaming to work properly
@@ -364,6 +396,86 @@ def put(bucket, key, data=nil, headers={})
rescue
on_exception
end
+
+
+
+ # New experimental API for uploading objects, introduced in RightAws 1.8.1.
+ # store_object is similar in function to the older function put, but returns the full response metadata. It also allows for optional verification
+ # of object md5 checksums on upload. Parameters are passed as hash entries and are checked for completeness as well as for spurious arguments.
+ # The hash of the response headers contains useful information like the Amazon request ID and the object ETag (MD5 checksum).
+ #
+ # If the optional :md5 argument is provided, store_object verifies that the given md5 matches the md5 returned by S3. The :verified_md5 field in the response hash is
+ # set true or false depending on the outcome of this check. If no :md5 argument is given, :verified_md5 will be false in the response.
+ #
+ # The optional argument of :headers allows the caller to specify arbitrary request header values.
+ #
+ # s3.store_object(:bucket => "foobucket", :key => "foo", :md5 => "a507841b1bc8115094b00bbe8c1b2954", :data => "polemonium" )
+ # => {"x-amz-id-2"=>"SVsnS2nfDaR+ixyJUlRKM8GndRyEMS16+oZRieamuL61pPxPaTuWrWtlYaEhYrI/",
+ # "etag"=>"\"a507841b1bc8115094b00bbe8c1b2954\"",
+ # "date"=>"Mon, 29 Sep 2008 18:57:46 GMT",
+ # :verified_md5=>true,
+ # "x-amz-request-id"=>"63916465939995BA",
+ # "server"=>"AmazonS3",
+ # "content-length"=>"0"}
+ #
+ # s3.store_object(:bucket => "foobucket", :key => "foo", :data => "polemonium" )
+ # => {"x-amz-id-2"=>"MAt9PLjgLX9UYJ5tV2fI/5dBZdpFjlzRVpWgBDpvZpl+V+gJFcBMW2L+LBstYpbR",
+ # "etag"=>"\"a507841b1bc8115094b00bbe8c1b2954\"",
+ # "date"=>"Mon, 29 Sep 2008 18:58:56 GMT",
+ # :verified_md5=>false,
+ # "x-amz-request-id"=>"3B25A996BC2CDD3B",
+ # "server"=>"AmazonS3",
+ # "content-length"=>"0"}
+
+ def store_object(params)
+ AwsUtils.allow_only([:bucket, :key, :data, :headers, :md5], params)
+ AwsUtils.mandatory_arguments([:bucket, :key, :data], params)
+ params[:headers] = {} unless params[:headers]
+
+ params[:data].binmode if(params[:data].respond_to?(:binmode)) # On Windows, if someone opens a file in text mode, we must reset it to binary mode for streaming to work properly
+ if (params[:data].respond_to?(:lstat) && params[:data].lstat.size >= USE_100_CONTINUE_PUT_SIZE) ||
+ (params[:data].respond_to?(:size) && params[:data].size >= USE_100_CONTINUE_PUT_SIZE)
+ params[:headers]['expect'] = '100-continue'
+ end
+
+ req_hash = generate_rest_request('PUT', params[:headers].merge(:url=>"#{params[:bucket]}/#{CGI::escape params[:key]}", :data=>params[:data]))
+ resp = request_info(req_hash, S3HttpResponseHeadParser.new)
+ if(params[:md5])
+ resp[:verified_md5] = (resp['etag'].gsub(/\"/, '') == params[:md5]) ? true : false
+ else
+ resp[:verified_md5] = false
+ end
+ resp
+ rescue
+ on_exception
+ end
+
+ # Identical in function to store_object, but requires verification that the returned ETag is identical to the checksum passed in by the user as the 'md5' argument.
+ # If the check passes, returns the response metadata with the "verified_md5" field set true. Raises an exception if the checksums conflict.
+ # This call is implemented as a wrapper around put_object and the user may gain different semantics by creating a custom wrapper.
+ #
+ # s3.store_object_and_verify(:bucket => "foobucket", :key => "foo", :md5 => "a507841b1bc8115094b00bbe8c1b2954", :data => "polemonium" )
+ # => {"x-amz-id-2"=>"IZN3XsH4FlBU0+XYkFTfHwaiF1tNzrm6dIW2EM/cthKvl71nldfVC0oVQyydzWpb",
+ # "etag"=>"\"a507841b1bc8115094b00bbe8c1b2954\"",
+ # "date"=>"Mon, 29 Sep 2008 18:38:32 GMT",
+ # :verified_md5=>true,
+ # "x-amz-request-id"=>"E8D7EA4FE00F5DF7",
+ # "server"=>"AmazonS3",
+ # "content-length"=>"0"}
+ #
+ # s3.store_object_and_verify(:bucket => "foobucket", :key => "foo", :md5 => "a507841b1bc8115094b00bbe8c1b2953", :data => "polemonium" )
+ # RightAws::AwsError: Uploaded object failed MD5 checksum verification: {"x-amz-id-2"=>"HTxVtd2bf7UHHDn+WzEH43MkEjFZ26xuYvUzbstkV6nrWvECRWQWFSx91z/bl03n",
+ # "etag"=>"\"a507841b1bc8115094b00bbe8c1b2954\"",
+ # "date"=>"Mon, 29 Sep 2008 18:38:41 GMT",
+ # :verified_md5=>false,
+ # "x-amz-request-id"=>"0D7ADE09F42606F2",
+ # "server"=>"AmazonS3",
+ # "content-length"=>"0"}
+ def store_object_and_verify(params)
+ AwsUtils.mandatory_arguments([:md5], params)
+ r = put_object(params)
+ r[:verified_md5] ? (return r) : (raise AwsError.new("Uploaded object failed MD5 checksum verification: #{r.inspect}"))
+ end
# Retrieves object data from Amazon. Returns a +hash+ or an exception.
#
@@ -398,6 +510,68 @@ def get(bucket, key, headers={}, &block)
rescue
on_exception
end
+
+ # New experimental API for retrieving objects, introduced in RightAws 1.8.1.
+ # retrieve_object is similar in function to the older function get. It allows for optional verification
+ # of object md5 checksums on retrieval. Parameters are passed as hash entries and are checked for completeness as well as for spurious arguments.
+ #
+ # If the optional :md5 argument is provided, retrieve_object verifies that the given md5 matches the md5 returned by S3. The :verified_md5 field in the response hash is
+ # set true or false depending on the outcome of this check. If no :md5 argument is given, :verified_md5 will be false in the response.
+ #
+ # The optional argument of :headers allows the caller to specify arbitrary request header values.
+ # Mandatory arguments:
+ # :bucket - the bucket in which the object is stored
+ # :key - the object address (or path) within the bucket
+ # Optional arguments:
+ # :headers - hash of additional HTTP headers to include with the request
+ # :md5 - MD5 checksum against which to verify the retrieved object
+ #
+ # s3.retrieve_object(:bucket => "foobucket", :key => "foo")
+ # => {:verified_md5=>false,
+ # :headers=>{"last-modified"=>"Mon, 29 Sep 2008 18:58:56 GMT",
+ # "x-amz-id-2"=>"2Aj3TDz6HP5109qly//18uHZ2a1TNHGLns9hyAtq2ved7wmzEXDOPGRHOYEa3Qnp",
+ # "content-type"=>"",
+ # "etag"=>"\"a507841b1bc8115094b00bbe8c1b2954\"",
+ # "date"=>"Tue, 30 Sep 2008 00:52:44 GMT",
+ # "x-amz-request-id"=>"EE4855DE27A2688C",
+ # "server"=>"AmazonS3",
+ # "content-length"=>"10"},
+ # :object=>"polemonium"}
+ #
+ # s3.retrieve_object(:bucket => "foobucket", :key => "foo", :md5=>'a507841b1bc8115094b00bbe8c1b2954')
+ # => {:verified_md5=>true,
+ # :headers=>{"last-modified"=>"Mon, 29 Sep 2008 18:58:56 GMT",
+ # "x-amz-id-2"=>"mLWQcI+VuKVIdpTaPXEo84g0cz+vzmRLbj79TS8eFPfw19cGFOPxuLy4uGYVCvdH",
+ # "content-type"=>"", "etag"=>"\"a507841b1bc8115094b00bbe8c1b2954\"",
+ # "date"=>"Tue, 30 Sep 2008 00:53:08 GMT",
+ # "x-amz-request-id"=>"6E7F317356580599",
+ # "server"=>"AmazonS3",
+ # "content-length"=>"10"},
+ # :object=>"polemonium"}
+ def retrieve_object(params, &block)
+ AwsUtils.mandatory_arguments([:bucket, :key], params)
+ AwsUtils.allow_only([:bucket, :key, :headers, :md5], params)
+ params[:headers] = {} unless params[:headers]
+ req_hash = generate_rest_request('GET', params[:headers].merge(:url=>"#{params[:bucket]}/#{CGI::escape params[:key]}"))
+ resp = request_info(req_hash, S3HttpResponseBodyParser.new, &block)
+ resp[:verified_md5] = false
+ if(params[:md5] && (resp[:headers]['etag'].gsub(/\"/,'') == params[:md5]))
+ resp[:verified_md5] = true
+ end
+ resp
+ rescue
+ on_exception
+ end
+
+ # Identical in function to retrieve_object, but requires verification that the returned ETag is identical to the checksum passed in by the user as the 'md5' argument.
+ # If the check passes, returns the response metadata with the "verified_md5" field set true. Raises an exception if the checksums conflict.
+ # This call is implemented as a wrapper around retrieve_object and the user may gain different semantics by creating a custom wrapper.
+ def retrieve_object_and_verify(params, &block)
+ AwsUtils.mandatory_arguments([:md5], params)
+ resp = get_object(params, block)
+ return resp if resp[:verified_md5]
+ raise AwsError.new("Retrieved object failed MD5 checksum verification: #{resp.inspect}")
+ end
# Retrieves object metadata. Returns a +hash+ of http_response_headers.
#
@@ -932,6 +1106,28 @@ def tagend(name)
end
end
end
+
+ class S3LoggingParser < RightAWSParser # :nodoc:
+ def reset
+ @result = {:enabled => false, :targetbucket => '', :targetprefix => ''}
+ @current_grantee = {}
+ end
+ def tagend(name)
+ case name
+ # service info
+ when 'TargetBucket'
+ if @xmlpath == 'BucketLoggingStatus/LoggingEnabled'
+ @result[:targetbucket] = @text
+ @result[:enabled] = true
+ end
+ when 'TargetPrefix'
+ if @xmlpath == 'BucketLoggingStatus/LoggingEnabled'
+ @result[:targetprefix] = @text
+ @result[:enabled] = true
+ end
+ end
+ end
+ end
class S3CopyParser < RightAWSParser # :nodoc:
def reset
View
126 lib/sdb/active_sdb.rb
@@ -101,6 +101,15 @@ def connection
# Create a new handle to an Sdb account. All handles share the same per process or per thread
# HTTP connection to Amazon Sdb. Each handle is for a specific account.
# The +params+ are passed through as-is to RightAws::SdbInterface.new
+ # Params:
+ # { :server => 'sdb.amazonaws.com' # Amazon service host: 'sdb.amazonaws.com'(default)
+ # :port => 443 # Amazon service port: 80 or 443(default)
+ # :protocol => 'https' # Amazon service protocol: 'http' or 'https'(default)
+ # :signature_version => '0' # The signature version : '0' or '1'(default)
+ # :multi_thread => true|false # Multi-threaded (connection per each thread): true or false(default)
+ # :logger => Logger Object # Logger instance: logs to STDOUT if omitted
+ # :nil_representation => 'mynil'} # interpret Ruby nil as this string value; i.e. use this string in SDB to represent Ruby nils (default is the string 'nil')
+
def establish_connection(aws_access_key_id=nil, aws_secret_access_key=nil, params={})
@connection = RightAws::SdbInterface.new(aws_access_key_id, aws_secret_access_key, params)
end
@@ -259,9 +268,10 @@ def delete_domain
# Client.find_by_name('Matias Rust')
# Client.find_by_name_and_city('Putin','Moscow')
# Client.find_by_name_and_city_and_post('Medvedev','Moscow','president')
- #
+ #
# Client.find_all_by_author('G.Bush jr')
# Client.find_all_by_age_and_gender_and_ethnicity('34','male','russian')
+ # Client.find_all_by_gender_and_country('male', 'Russia', :auto_load => true, :order => 'name desc')
#
# Returned records have to be +reloaded+ to access their attributes.
#
@@ -276,37 +286,100 @@ def delete_domain
# Client.find(:all, :limit => 10, :next_token => Client.next_token)
# end while Client.next_token
#
+ # Sort oder:
+ # Client.find(:all, :order => 'gender')
+ # Client.find(:all, :order => 'name desc')
+ #
+ # Attributes auto load (be carefull - this may take lot of time for a huge bunch of records):
+ # Client.find(:first) #=> #<Client:0xb77d0d40 @attributes={"id"=>"2937601a-e45d-11dc-a75f-001bfc466dd7"}, @new_record=false>
+ # Client.find(:first, :auto_load => true) #=> #<Client:0xb77d0d40 @attributes={"id"=>"2937601a-e45d-11dc-a75f-001bfc466dd7", "name"=>["Cat"], "toys"=>["Jons socks", "clew", "mice"]}, @new_record=false>
+ #
def find(*args)
options = args.last.is_a?(Hash) ? args.pop : {}
case args.first
- when :all : find_every options
- when :first : find_initial options
- else find_from_ids args, options
+ when :all then find_every options
+ when :first then find_initial options
+ else find_from_ids args, options
end
end
- protected
-
- def query(query_expression=nil, max_number_of_items = nil, next_token = nil) # :nodoc:
- @next_token = next_token
+ protected
+
+ # Returns an array of query attributes.
+ # Query_expression must be a well formated SDB query string:
+ # query_attributes("['title' starts-with 'O\\'Reily'] intersection ['year' = '2007']") #=> ["title", "year"]
+ def query_attributes(query_expression)
+ attrs = []
+ array = query_expression.scan(/['"](.*?[^\\])['"]/).flatten
+ until array.empty? do
+ attrs << array.shift # skip it's value
+ array.shift #
+ end
+ attrs
+ end
+
+ # Returns an array of [attribute_name, 'asc'|'desc']
+ def sort_options(sort_string)
+ sort_string[/['"]?(\w+)['"]? *(asc|desc)?/i]
+ [$1, ($2 || 'asc')]
+ end
+
+ # Perform a query request.
+ #
+ # Options
+ # :query_expression nil | string | array
+ # :max_number_of_items nil | integer
+ # :next_token nil | string
+ # :sort_option nil | string "name desc|asc"
+ #
+ def query(options) # :nodoc:
+ @next_token = options[:next_token]
+ query_expression = options[:query_expression]
+ query_expression = connection.query_expression_from_array(query_expression) if query_expression.is_a?(Array)
+ # add sort_options to the query_expression
+ if options[:sort_option]
+ sort_by, sort_order = sort_options(options[:sort_option])
+ sort_query_expression = "['#{sort_by}' starts-with '']"
+ sort_by_expression = " sort '#{sort_by}' #{sort_order}"
+ # make query_expression to be a string (it may be null)
+ query_expression = query_expression.to_s
+ # quote from Amazon:
+ # The sort attribute must be present in at least one of the predicates of the query expression.
+ if query_expression.blank?
+ query_expression = sort_query_expression
+ elsif !query_attributes(query_expression).include?(sort_by)
+ query_expression += " intersection #{sort_query_expression}"
+ end
+ query_expression += sort_by_expression
+ end
# request items
- query_result = self.connection.query(domain, query_expression, max_number_of_items, @next_token)
+ query_result = self.connection.query(domain, query_expression, options[:max_number_of_items], @next_token)
@next_token = query_result[:next_token]
items = query_result[:items].map do |name|
new_item = self.new('id' => name)
new_item.mark_as_old
+ new_item.reload if options[:auto_load]
new_item
end
items
end
+ def reload_all_records(*list) # :nodoc:
+ list.flatten.each { |record| record.reload }
+ end
+
def find_every(options) # :nodoc:
- query(options[:conditions], options[:limit], options[:next_token])
+ records = query( :query_expression => options[:conditions],
+ :max_number_of_items => options[:limit],
+ :next_token => options[:next_token],
+ :sort_option => options[:sort] || options[:order] )
+ options[:auto_load] ? reload_all_records(records) : records
end
def find_initial(options) # :nodoc:
options[:limit] = 1
- find_every(options)[0]
+ record = find_every(options).first
+ options[:auto_load] ? reload_all_records(record).first : record
end
def find_from_ids(args, options) # :nodoc:
@@ -326,7 +399,7 @@ def find_from_ids(args, options) # :nodoc:
result = find_every(options)
# if one record was requested then return it
unless bunch_of_records_requested
- result.first
+ options[:auto_load] ? reload_all_records(result.first).first : result.first
else
# if a bunch of records was requested then return check that we found all of them
# and return as an array
@@ -334,35 +407,40 @@ def find_from_ids(args, options) # :nodoc:
id_list = args.map{|i| "'#{i}'"}.join(',')
raise ActiveSdbError.new("Couldn't find all #{name} with IDs (#{id_list}) (found #{result.size} results, but was looking for #{args.size})")
else
- result
+ options[:auto_load] ? reload_all_records(result) : result
end
end
end
# find_by helpers
- def find_all_by_(format_str, args, limit=nil) # :nodoc:
+ def find_all_by_(format_str, args, options) # :nodoc:
fields = format_str.to_s.sub(/^find_(all_)?by_/,'').split('_and_')
conditions = fields.map { |field| "['#{field}'=?]" }.join(' intersection ')
- find(:all, :conditions => [conditions, *args], :limit => limit)
+ options[:conditions] = [conditions, *args]
+ find(:all, options)
end
- def find_by_(format_str, args) # :nodoc:
- find_all_by_(format_str, args, 1)[0]
+ def find_by_(format_str, args, options) # :nodoc:
+ options[:limit] = 1
+ find_all_by_(format_str, args, options).first
end
def method_missing(method, *args) # :nodoc:
- if method.to_s[/^find_all_by_/] then return find_all_by_(method, args)
- elsif method.to_s[/^find_by_/] then return find_by_(method, args)
- else super(method, *args)
+ if method.to_s[/^(find_all_by_|find_by_)/]
+ options = args.last.is_a?(Hash) ? args.pop : {}
+ if method.to_s[/^find_all_by_/]
+ find_all_by_(method, args, options)
+ else
+ find_by_(method, args, options)
+ end
+ else
+ super(method, *args)
end
end
-
end
def self.generate_id # :nodoc:
- result = ''
- result = UUID.timestamp_create().to_s
- result
+ UUID.timestamp_create().to_s
end
public
View
46 lib/sdb/right_sdb_interface.rb
@@ -39,6 +39,8 @@ class SdbInterface < RightAwsBase
def self.bench_xml; @@bench.xml; end
def self.bench_sdb; @@bench.service; end
+ attr_reader :last_query_expression
+
# Creates new RightSdb instance.
#
# Params:
@@ -48,7 +50,7 @@ def self.bench_sdb; @@bench.service; end
# :signature_version => '0' # The signature version : '0' or '1'(default)
# :multi_thread => true|false # Multi-threaded (connection per each thread): true or false(default)
# :logger => Logger Object # Logger instance: logs to STDOUT if omitted
- # :nil_representation => 'mynil'} # interpret Ruby nil as this string value; i.e. use this string in SDB to represent nils
+ # :nil_representation => 'mynil'} # interpret Ruby nil as this string value; i.e. use this string in SDB to represent Ruby nils (default is the string 'nil')
#
# Example:
#
@@ -84,8 +86,8 @@ def generate_request(action, params={}) #:nodoc:
service_hash.update(params)
# prepare string to sight
string_to_sign = case signature_version
- when '0' : service_hash["Action"] + service_hash["Timestamp"]
- when '1' : service_hash.sort{|a,b| (a[0].to_s.downcase)<=>(b[0].to_s.downcase)}.to_s
+ when '0' then service_hash["Action"] + service_hash["Timestamp"]
+ when '1' then service_hash.sort{|a,b| (a[0].to_s.downcase)<=>(b[0].to_s.downcase)}.to_s
end
service_hash.update('Signature' => AwsUtils::sign(@aws_secret_access_key, string_to_sign))
service_string = service_hash.to_a.collect{|key,val| key + "=#{CGI::escape(val.to_s)}" }.join("&")
@@ -120,6 +122,7 @@ def pack_attributes(attributes, replace = false) #:nodoc:
result = {}
if attributes
idx = 0
+ skip_values = attributes.is_a?(Array)
attributes.each do |attribute, values|
# set replacement attribute
result["Attribute.#{idx}.Replace"] = 'true' if replace
@@ -127,12 +130,12 @@ def pack_attributes(attributes, replace = false) #:nodoc:
unless values.nil?
Array(values).each do |value|
result["Attribute.#{idx}.Name"] = attribute
- result["Attribute.#{idx}.Value"] = ruby_to_sdb(value)
+ result["Attribute.#{idx}.Value"] = ruby_to_sdb(value) unless skip_values
idx += 1
end
else
result["Attribute.#{idx}.Name"] = attribute
- result["Attribute.#{idx}.Value"] = ruby_to_sdb(nil)
+ result["Attribute.#{idx}.Value"] = ruby_to_sdb(nil) unless skip_values
idx += 1
end
end
@@ -406,10 +409,15 @@ def delete_attributes(domain_name, item_name, attributes = nil)
# query = [ "['cat'=?] union ['dog'=?]", "clew", "Jon's boot" ]
# sdb.query('family', query)
#
+ # query = [ "['cat'=?] union ['dog'=?] sort 'cat' desc", "clew", "Jon's boot" ]
+ # sdb.query('family', query)
+ #
# see: http://docs.amazonwebservices.com/AmazonSimpleDB/2007-11-07/DeveloperGuide/SDB_API_Query.html
+ # http://docs.amazonwebservices.com/AmazonSimpleDB/2007-11-07/DeveloperGuide/index.html?SortingData.html
#
def query(domain_name, query_expression = nil, max_number_of_items = nil, next_token = nil)
query_expression = query_expression_from_array(query_expression) if query_expression.is_a?(Array)
+ @last_query_expression = query_expression
#
request_params = { 'DomainName' => domain_name,
'QueryExpression' => query_expression,
@@ -441,10 +449,10 @@ def reset
end
def tagend(name)
case name
- when 'NextToken' : @result[:next_token] = @text
- when 'DomainName' : @result[:domains] << @text
- when 'BoxUsage' : @result[:box_usage] = @text
- when 'RequestId' : @result[:request_id] = @text
+ when 'NextToken' then @result[:next_token] = @text
+ when 'DomainName' then @result[:domains] << @text
+ when 'BoxUsage' then @result[:box_usage] = @text
+ when 'RequestId' then @result[:request_id] = @text
end
end
end
@@ -455,8 +463,8 @@ def reset
end
def tagend(name)
case name
- when 'BoxUsage' : @result[:box_usage] = @text
- when 'RequestId' : @result[:request_id] = @text
+ when 'BoxUsage' then @result[:box_usage] = @text
+ when 'RequestId' then @result[:request_id] = @text
end
end
end
@@ -468,10 +476,10 @@ def reset
end
def tagend(name)
case name
- when 'Name' : @last_attribute_name = @text
- when 'Value' : (@result[:attributes][@last_attribute_name] ||= []) << @text
- when 'BoxUsage' : @result[:box_usage] = @text
- when 'RequestId' : @result[:request_id] = @text
+ when 'Name' then @last_attribute_name = @text
+ when 'Value' then (@result[:attributes][@last_attribute_name] ||= []) << @text
+ when 'BoxUsage' then @result[:box_usage] = @text
+ when 'RequestId' then @result[:request_id] = @text
end
end
end
@@ -482,10 +490,10 @@ def reset
end
def tagend(name)
case name
- when 'ItemName' : @result[:items] << @text
- when 'BoxUsage' : @result[:box_usage] = @text
- when 'RequestId' : @result[:request_id] = @text
- when 'NextToken' : @result[:next_token] = @text
+ when 'ItemName' then @result[:items] << @text
+ when 'BoxUsage' then @result[:box_usage] = @text
+ when 'RequestId' then @result[:request_id] = @text
+ when 'NextToken' then @result[:next_token] = @text
end
end
end
View
8 lib/sqs/right_sqs.rb
@@ -54,6 +54,14 @@ module RightAws
# ...
# grantee2 = queue.grantees('another_cool_guy@email.address')
# grantee2.revoke('SENDMESSAGE')
+ #
+ # Params is a hash:
+ #
+ # {:server => 'queue.amazonaws.com' # Amazon service host: 'queue.amazonaws.com' (default)
+ # :port => 443 # Amazon service port: 80 or 443 (default)
+ # :multi_thread => true|false # Multi-threaded (connection per each thread): true or false (default)
+ # :signature_version => '0' # The signature version : '0' or '1'(default)
+ # :logger => Logger Object} # Logger instance: logs to STDOUT if omitted }
#
class Sqs
attr_reader :interface
View
8 lib/sqs/right_sqs_gen2.rb
@@ -55,6 +55,14 @@ module RightAws
# ...
#
# NB: Second-generation SQS has eliminated the entire access grant mechanism present in Gen 1.
+ #
+ # Params is a hash:
+ #
+ # {:server => 'queue.amazonaws.com' # Amazon service host: 'queue.amazonaws.com' (default)
+ # :port => 443 # Amazon service port: 80 or 443 (default)
+ # :multi_thread => true|false # Multi-threaded (connection per each thread): true or false (default)
+ # :signature_version => '0' # The signature version : '0' or '1'(default)
+ # :logger => Logger Object} # Logger instance: logs to STDOUT if omitted }
class SqsGen2
attr_reader :interface
View
49 test/s3/test_right_s3.rb
@@ -7,6 +7,7 @@ class TestS3 < Test::Unit::TestCase
def setup
@s3 = Rightscale::S3Interface.new(TestCredentials.aws_access_key_id, TestCredentials.aws_secret_access_key)
@bucket = 'right_s3_awesome_test_bucket_00001'
+ @bucket2 = 'right_s3_awesome_test_bucket_00002'
@key1 = 'test/woohoo1/'
@key2 = 'test1/key/woohoo2'
@key3 = 'test2/A%B@C_D&E?F+G=H"I'
@@ -72,9 +73,9 @@ def test_07_streaming_get
def test_08_keys
keys = @s3.list_bucket(@bucket).map{|b| b[:key]}
assert_equal keys.size, 3, "There should be 3 keys"
- assert(keys.include? @key1)
- assert(keys.include? @key2)
- assert(keys.include? @key3)
+ assert(keys.include?(@key1))
+ assert(keys.include?(@key2))
+ assert(keys.include?(@key3))
end
def test_09_copy_key
@@ -120,18 +121,26 @@ def test_11_rename_key
keys = @s3.list_bucket(@bucket).map{|b| b[:key]}
assert(!keys.include?(@key2))
end
-
- def test_12_delete_folder
+ def test_12_retrieve_object
+ assert_raise(Rightscale::AwsError) { @s3.retrieve_object(:bucket => @bucket, :key => 'undefined/key') }
+ data1 = @s3.retrieve_object(:bucket => @bucket, :key => @key1_new_name)
+ assert_equal RIGHT_OBJECT_TEXT, data1[:object], "Object text must be equal to '#{RIGHT_OBJECT_TEXT}'"
+ assert_equal 'Woohoo1!', data1[:headers]['x-amz-meta-family'], "x-amz-meta-family header must be equal to 'Woohoo1!'"
+ end
+ def test_13_delete_folder
assert_equal 1, @s3.delete_folder(@bucket, 'test').size, "Only one key(#{@key1}) must be deleted!"
end
-
- def test_13_delete_bucket
+
+ def test_14_delete_bucket
assert_raise(Rightscale::AwsError) { @s3.delete_bucket(@bucket) }
assert @s3.clear_bucket(@bucket), 'Clear_bucket fail'
assert_equal 0, @s3.list_bucket(@bucket).size, 'Bucket must be empty'
assert @s3.delete_bucket(@bucket)
assert !@s3.list_all_my_buckets.map{|bucket| bucket[:name]}.include?(@bucket), "#{@bucket} must not exist"
end
+
+
+
#---------------------------
# Rightscale::S3 classes
@@ -312,10 +321,10 @@ def test_32_grant_revoke_drop
assert grantee.apply
assert !grantee.exists?
# Check multiple perms assignment
- assert grantee.grant 'FULL_CONTROL', 'READ', 'WRITE'
+ assert grantee.grant('FULL_CONTROL', 'READ', 'WRITE')
assert_equal ['FULL_CONTROL','READ','WRITE'].sort, grantee.perms.sort
# Check multiple perms removal
- assert grantee.revoke 'FULL_CONTROL', 'WRITE'
+ assert grantee.revoke('FULL_CONTROL', 'WRITE')
assert_equal ['READ'], grantee.perms
# check 'Drop' method
assert grantee.drop
@@ -384,5 +393,27 @@ def test_36_set_amazon_problems
RightAws::S3Interface.amazon_problems= nil
assert_nil(RightAws::S3Interface.amazon_problems)
end
+
+ def test_37_access_logging
+ bucket = Rightscale::S3::Bucket.create(@s, @bucket, false)
+ targetbucket = Rightscale::S3::Bucket.create(@s, @bucket2, true)
+ # Take 'AllUsers' grantee
+ grantee = Rightscale::S3::Grantee.new(targetbucket,'http://acs.amazonaws.com/groups/s3/LogDelivery')
+
+ assert grantee.grant(['READ_ACP', 'WRITE'])
+
+ assert bucket.enable_logging(:targetbucket => targetbucket, :targetprefix => "loggylogs/")
+
+ assert_equal(bucket.logging_info, {:enabled => true, :targetbucket => @bucket2, :targetprefix => "loggylogs/"})
+
+ assert bucket.disable_logging
+
+ # check 'Drop' method
+ assert grantee.drop
+
+ # Delete bucket
+ bucket.delete(true)
+ targetbucket.delete(true)
+ end
end
View
11 test/sdb/test_active_sdb.rb
@@ -127,6 +127,11 @@ def test_06_find_all_by_helpers
assert_equal 2, Client.find_all_by_post_and_country('president','Russia').size
# find all women in USA that love flowers
assert_equal 2, Client.find_all_by_gender_and_country_and_hobby('female','Russia','flowers').size
+ # order and auto_load:
+ clients = Client.find_all_by_post('president', :order => 'name', :auto_load => true)
+ assert_equal [['Bush'], ['Medvedev'], ['Putin']], clients.map{|c| c['name']}
+ clients = Client.find_all_by_post('president', :order => 'name desc', :auto_load => true)
+ assert_equal [['Putin'], ['Medvedev'], ['Bush']], clients.map{|c| c['name']}
end
def test_07_find_by_helpers
@@ -135,7 +140,9 @@ def test_07_find_by_helpers
# find any russian president
assert Client.find_by_post_and_country('president','Russia')
# find Mary in Russia that loves flowers
- assert Client.find_by_gender_and_country_and_hobby('female','Russia','flowers')
+ # order and auto_load:
+ assert_equal ['Bush'], Client.find_by_post('president', :order => 'name', :auto_load => true)['name']
+ assert_equal ['Putin'], Client.find_by_post('president', :order => 'name desc', :auto_load => true)['name']
end
def test_08_reload
@@ -226,7 +233,7 @@ def test_11_delete
assert ['russian'], new_putin['language'].sort
# --- delete_attributes
putin.delete_attributes('language', 'hobby')
- wait SDB_DELAY, 'test 11: after put_attributes'
+ wait SDB_DELAY, 'test 11: after delete_attributes'
# trash hoddy and langs
new_putin = Client.find_by_name('Putin')
new_putin.reload
View
1  test/sdb/test_helper.rb
@@ -1,2 +1,3 @@
require 'test/unit'
require File.dirname(__FILE__) + '/../../lib/right_aws'
+require 'sdb/active_sdb'
Please sign in to comment.
Something went wrong with that request. Please try again.