Skip to content
Browse files

Merge branch 'master' of github.com:appoxy/simple_record

  • Loading branch information...
2 parents 6ebd3e8 + 7bf1552 commit cf34d3482a316a362f9b6c93f3f08e1a70fda7c1 @treeder treeder committed Jan 2, 2011
View
91 README.markdown
@@ -19,7 +19,7 @@ Brought to you by: [![Appoxy](http://www.simpledeployr.com/images/global/appoxy-
require 'simple_record'
class MyModel < SimpleRecord::Base
- has_attributes :name
+ has_strings :name
has_ints :age
end
@@ -45,25 +45,26 @@ More about ModelAttributes below.
puts 'got=' + mm2.name + ' and he/she is ' + mm.age.to_s + ' years old'
# Or more advanced queries? mms = MyModel?.find(:all, ["age=?", 32], :order=>"name", :limit=>10)
+That's literally all you need to do to get started. No database install, no other setup required.
## Attributes and modifiers for models
NOTE: All objects will automatically have :id, :created, :updated attributes.
-### has_attributes
+### has_strings
Add string attributes.
class MyModel < SimpleRecord::Base
- has_attributes :name
+ has_strings :name
end
### has_ints, has_dates and has_booleans
-Lets simple_record know that certain attributes defined in has_attributes should be treated as integers, dates or booleans. This is required because SimpleDB only has strings so SimpleRecord needs to know how to convert, pad, offset, etc.
+This is required because SimpleDB only has strings so SimpleRecord needs to know how to convert, pad, offset, etc.
class MyModel < SimpleRecord::Base
- has_attributes :name
+ has_strings :name
has_ints :age, :height
has_dates :birthday
has_booleans :is_nerd
@@ -75,7 +76,7 @@ Creates a many-to-one relationship. Can only have one per belongs_to call.
class MyModel < SimpleRecord::Base
belongs_to :school
- has_attributes :name
+ has_strings :name
has_ints :age, :height
has_dates :birthday
has_booleans :is_nerd
@@ -166,6 +167,30 @@ For rails, be sure to add this to your Application controller if using per_threa
SimpleRecord.close_connection
end
+### LOB and any other S3 Storage
+
+ :s3_bucket=>...
+
+* :old (default) will use the existing lob location of "#{aws_access_key}_lobs", but any new features will use the :new bucket.
+* :new will use the new and recommended s3 bucket location of "simple_record_#{aws_access_key}".
+* Any string value will use that value as the bucket name.
+
+
+NOTE: All projects should set this as we may make this default in a future major version (v3?). Existing projects should use
+:s3_bucket=>:
+
+### Created and Updated At Columns
+
+The default uses the columns "created" and "updated" which unfortunately are not the same as ActiveRecord, which
+uses created_at and updated_at. Although you can use created_at and updated_at methods, you may still want the columns in
+SimpleDB to be created_at and updated_at.
+
+ :created_col=>"created", :updated_col=>"updated"
+
+NOTE: All projects should set these as we may make the "_at" postfix default in a future major version (v3?). Existing
+projects should set :created_col=>"created", :updated_col=>"updated" (default) so if it changes in a future version,
+there won't be any issues.
+
## SimpleRecord on Rails
You don't really have to do anything except have your models extends SimpleRecord::Base instead of ActiveRecord::Base, but here are some tips you can use.
@@ -194,11 +219,21 @@ This is most helpful on windows so Rails doesn't need sqlite or mysql gems/drive
Typical databases support BLOB's and/or CLOB's, but SimpleDB has a 1024 character per attribute maximum so larger
values should be stored in S3. Fortunately SimpleRecord takes care of this for you by defining has_clobs for a large
-string value.
+string value. There is no support for blobs yet.
has_clobs :my_clob
-These clob values will be stored in s3 under a bucket named: "#{aws_access_key}_lobs"
+These clob values will be stored in s3 under a bucket named "#{aws_access_key}_lobs"
+OR "simple_record_#{aws_access_key}/lobs" if you set :s3_bucket=>:new in establish_connection (RECOMMENDED).
+
+If you have more than one clob on an object and if it makes sense for performance reasons, you can set a configuration option on the class to store all clobs
+as one item on s3 which means it will do a single put to s3 and a single get for all the clobs on the object.
+This would generally be good for somewhat small clob values or when you know you will always be accessing
+all the clobs on the object.
+
+ sr_config :single_clob=>true
+
+Setting this will automatically use :s3_bucket=>:new as well.
## Tips and Tricks and Things to Know
@@ -236,20 +271,36 @@ or
o.something_id = x
-### Batch Save
+Accessing the id can prevent a database call so if you only need the ID, then you
+should use this.
+
+## Batch Save
To do a batch save using SimpleDB's batch saving feature to improve performance, simply create your objects, add them to an array, then call:
MyClass.batch_save(object_list)
+## Batch Delete
+
+To do a batch delete using SimpleDB's batch delete feature to improve performance, simply create your objects, add them to an array, then call:
+
+ MyClass.batch_delete(object_list or list_of_ids)
+
+## Operations across a Query
+
+ MyClass.delete_all(find_options)
+ MyClass.destroy_all(find_options)
+
+find_options can include anything you'd add after a find(:all, find_options) including :conditions, :limit, etc.
+
## Caching
You can use any cache that supports the ActiveSupport::Cache::Store interface.
SimpleRecord::Base.cache_store = my_cache_store
-If you want a simple in memory cache store, try: http://gemcutter.org/gems/local_cache . It supports max cache size and
-timeouts. You can also use memcached or http://www.quetzall.com/cloudcache.
+If you want a simple in memory cache store that supports max cache size and expiration, try: <http://gemcutter.org/gems/local_cache>.
+You can also use memcached or http://www.quetzall.com/cloudcache.
## Encryption
@@ -275,6 +326,24 @@ ob2.password == "mypassword"
This will actually be compared by hashing "mypassword" first.
+## Sharding
+
+Sharding allows you to partition your data for a single class across multiple domains allowing increased write throughput,
+faster queries and more space (multiply your 10GB per domain limit). And it's very easy to implement with SimpleRecord.
+
+ shard :shards=>:my_shards_function, :map=>:my_mapping_function
+
+The :shards function should return a list of shard names, for example: ['CA', 'FL', 'HI', ...] or [1,2,3,4,...]
+
+The :map function should return which shard name the object should be stored to.
+
+When executing a find() operation, you can explicitly specify the shard(s) you'd like to find on. This is
+particularly useful if you know in advance which shard the data will be in.
+
+ MyClass.find(:all, :conditions=>....., :shard=>["CA", "FL"])
+
+You can see some [example classes here](https://github.com/appoxy/simple_record/blob/master/test/my_sharded_model.rb).
+
## Kudos
Special thanks to Garrett Cox for creating Activerecord2sdb which SimpleRecord is based on:
View
6 VERSION.yml
@@ -1,5 +1,5 @@
---
-:major: 1
-:minor: 5
-:patch: 8
+:major: 2
+:minor: 0
+:patch: 4
:build:
View
1,726 lib/simple_record.rb
@@ -46,1047 +46,1093 @@
module SimpleRecord
- @@options = {}
- @@stats = SimpleRecord::Stats.new
- @@logging = false
- @@s3 = nil
- @@auto_close_s3 = false
- @@logger = Logger.new(STDOUT)
- @@logger.level = Logger::INFO
+ @@options = {}
+ @@stats = SimpleRecord::Stats.new
+ @@logging = false
+ @@s3 = nil
+ @@auto_close_s3 = false
+ @@logger = Logger.new(STDOUT)
+ @@logger.level = Logger::INFO
+
+ class << self;
+ attr_accessor :aws_access_key, :aws_secret_key
+
+ # Deprecated
+ def enable_logging
+ @@logging = true
+ @@logger.level = Logger::DEBUG
+ end
- class << self;
- attr_accessor :aws_access_key, :aws_secret_key
+ # Deprecated
+ def disable_logging
+ @@logging = false
+ end
- # Deprecated
- def enable_logging
- @@logging = true
- @@logger.level = Logger::DEBUG
- end
+ # Deprecated
+ def logging?
+ @@logging
+ end
- # Deprecated
- def disable_logging
- @@logging = false
- end
+ def logger
+ @@logger
+ end
- # Deprecated
- def logging?
- @@logging
- end
+ # This can be used to log queries and what not to a file.
+ # Params:
+ # :select=>{:filename=>"file_to_write_to", :format=>"csv"}
+ def log_usage(types={})
+ @usage_logging_options = {} unless @usage_logging_options
+ return if types.nil?
+ types.each_pair do |type, options|
+ options[:lines_between_flushes] = 100 unless options[:lines_between_flushes]
+ @usage_logging_options[type] = options
+ end
+ #puts 'SimpleRecord.usage_logging_options=' + SimpleRecord.usage_logging_options.inspect
+ end
- def logger
- @@logger
- end
+ def close_usage_log(type)
+ return unless @usage_logging_options[type]
+ @usage_logging_options[type][:file].close if @usage_logging_options[type][:file]
+ end
- # This can be used to log queries and what not to a file.
- # Params:
- # :select=>{:filename=>"file_to_write_to", :format=>"csv"}
- def log_usage(types={})
- @usage_logging_options = {} unless @usage_logging_options
- return if types.nil?
- types.each_pair do |type, options|
- options[:lines_between_flushes] = 100 unless options[:lines_between_flushes]
- @usage_logging_options[type] = options
- end
- #puts 'SimpleRecord.usage_logging_options=' + SimpleRecord.usage_logging_options.inspect
- end
+ def usage_logging_options
+ @usage_logging_options
+ end
- def close_usage_log(type)
- return unless @usage_logging_options[type]
- @usage_logging_options[type][:file].close if @usage_logging_options[type][:file]
- end
+ def stats
+ @@stats
+ end
- def usage_logging_options
- @usage_logging_options
- end
- def stats
- @@stats
- end
+ # Create a new handle to an Sdb account. All handles share the same per process or per thread
+ # HTTP connection to Amazon Sdb. Each handle is for a specific account.
+ # The +params+ are passed through as-is to Aws::SdbInterface.new
+ # Params:
+ # { :server => 'sdb.amazonaws.com' # Amazon service host: 'sdb.amazonaws.com'(default)
+ # :port => 443 # Amazon service port: 80(default) or 443
+ # :protocol => 'https' # Amazon service protocol: 'http'(default) or 'https'
+ # :signature_version => '0' # The signature version : '0' or '1'(default)
+ # :connection_mode => :default # options are
+ # :default (will use best known safe (as in won't need explicit close) option, may change in the future)
+ # :per_request (opens and closes a connection on every request to SDB)
+ # :single (one thread across entire app)
+ # :per_thread (one connection per thread)
+ # :pool (uses a connection pool with a maximum number of connections - NOT IMPLEMENTED YET)
+ # :logger => Logger Object # Logger instance: logs to STDOUT if omitted
+ def establish_connection(aws_access_key=nil, aws_secret_key=nil, options={})
+ @aws_access_key = aws_access_key
+ @aws_secret_key = aws_secret_key
+ @@options.merge!(options)
+ #puts 'SimpleRecord.establish_connection with options: ' + @@options.inspect
+ SimpleRecord::ActiveSdb.establish_connection(aws_access_key, aws_secret_key, @@options)
+ if options[:connection_mode] == :per_thread
+ @@auto_close_s3 = true
+ # todo: should we init this only when needed?
+ end
+ s3_ops = {:connection_mode=>options[:connection_mode] || :default}
+ @@s3 = Aws::S3.new(SimpleRecord.aws_access_key, SimpleRecord.aws_secret_key, s3_ops)
+
+ if options[:created_col]
+ SimpleRecord::Base.has_dates options[:created_col]
+ end
+ if options[:updated_col]
+ SimpleRecord::Base.has_dates options[:updated_col]
+ end
- # Create a new handle to an Sdb account. All handles share the same per process or per thread
- # HTTP connection to Amazon Sdb. Each handle is for a specific account.
- # The +params+ are passed through as-is to Aws::SdbInterface.new
- # Params:
- # { :server => 'sdb.amazonaws.com' # Amazon service host: 'sdb.amazonaws.com'(default)
- # :port => 443 # Amazon service port: 80(default) or 443
- # :protocol => 'https' # Amazon service protocol: 'http'(default) or 'https'
- # :signature_version => '0' # The signature version : '0' or '1'(default)
- # :connection_mode => :default # options are
- # :default (will use best known safe (as in won't need explicit close) option, may change in the future)
- # :per_request (opens and closes a connection on every request to SDB)
- # :single (one thread across entire app)
- # :per_thread (one connection per thread)
- # :pool (uses a connection pool with a maximum number of connections - NOT IMPLEMENTED YET)
- # :logger => Logger Object # Logger instance: logs to STDOUT if omitted
- def establish_connection(aws_access_key=nil, aws_secret_key=nil, options={})
- @aws_access_key = aws_access_key
- @aws_secret_key = aws_secret_key
- @@options.merge!(options)
- #puts 'SimpleRecord.establish_connection with options: ' + @@options.inspect
- SimpleRecord::ActiveSdb.establish_connection(aws_access_key, aws_secret_key, @@options)
- if options[:connection_mode] == :per_thread
- @@auto_close_s3 = true
- # todo: should we init this only when needed?
- @@s3 = Aws::S3.new(SimpleRecord.aws_access_key, SimpleRecord.aws_secret_key, {:connection_mode=>:per_thread})
- end
- end
-
- # Call this to close the connection to SimpleDB.
- # If you're using this in Rails with per_thread connection mode, you should do this in
- # an after_filter for each request.
- def close_connection()
- SimpleRecord::ActiveSdb.close_connection
- @@s3.close_connection if @@auto_close_s3
- end
+ end
- # If you'd like to specify the s3 connection to use for LOBs, you can pass it in here.
- # We recommend that this connection matches the type of connection you're using for SimpleDB,
- # at least if you're using per_thread connection mode.
- def s3=(s3)
- @@s3 = s3
- end
+ # Call this to close the connection to SimpleDB.
+ # If you're using this in Rails with per_thread connection mode, you should do this in
+ # an after_filter for each request.
+ def close_connection()
+ SimpleRecord::ActiveSdb.close_connection
+ @@s3.close_connection if @@auto_close_s3
+ end
- def s3
- @@s3
- end
+ # If you'd like to specify the s3 connection to use for LOBs, you can pass it in here.
+ # We recommend that this connection matches the type of connection you're using for SimpleDB,
+ # at least if you're using per_thread connection mode.
+ def s3=(s3)
+ @@s3 = s3
+ end
- def options
- @@options
- end
+ def s3
+ @@s3
+ end
+ def options
+ @@options
end
- class Base < SimpleRecord::ActiveSdb::Base
+ end
+
+ class Base < SimpleRecord::ActiveSdb::Base
# puts 'Is ActiveModel defined? ' + defined?(ActiveModel).inspect
- if defined?(ActiveModel)
- extend ActiveModel::Naming
- include ActiveModel::Conversion
- include ActiveModel::Validations
- else
- attr_accessor :errors
- include SimpleRecord::Rails2
- end
+ if defined?(ActiveModel)
+ extend ActiveModel::Naming
+ include ActiveModel::Conversion
+ include ActiveModel::Validations
+ else
+ attr_accessor :errors
+ include SimpleRecord::Rails2
+ end
- include SimpleRecord::Translations
+ include SimpleRecord::Translations
# include SimpleRecord::Attributes
- extend SimpleRecord::Attributes::ClassMethods
- include SimpleRecord::Attributes
- extend SimpleRecord::Sharding::ClassMethods
- include SimpleRecord::Sharding
- include SimpleRecord::Callbacks
- include SimpleRecord::Json
- include SimpleRecord::Logging
- extend SimpleRecord::Logging::ClassMethods
+ extend SimpleRecord::Attributes::ClassMethods
+ include SimpleRecord::Attributes
+ extend SimpleRecord::Sharding::ClassMethods
+ include SimpleRecord::Sharding
+ include SimpleRecord::Callbacks
+ include SimpleRecord::Json
+ include SimpleRecord::Logging
+ extend SimpleRecord::Logging::ClassMethods
- def self.extended(base)
+ def self.extended(base)
- end
+ end
- def initialize(attrs={})
- # todo: Need to deal with objects passed in. iterate through belongs_to perhaps and if in attrs, set the objects id rather than the object itself
+ def initialize(attrs={})
+ # todo: Need to deal with objects passed in. iterate through belongs_to perhaps and if in attrs, set the objects id rather than the object itself
- initialize_base(attrs)
+ initialize_base(attrs)
- # Convert attributes to sdb values
- attrs.each_pair do |name, value|
- set(name, value, true)
- end
- end
+ # Convert attributes to sdb values
+ attrs.each_pair do |name, value|
+ set(name, value, true)
+ end
+ end
- def initialize_base(attrs={})
+ def initialize_base(attrs={})
- #we have to handle the virtuals.
- Attributes.handle_virtuals(attrs)
+ #we have to handle the virtuals.
+ Attributes.handle_virtuals(attrs)
- @errors=SimpleRecord_errors.new if not (defined?(ActiveModel))
- @dirty = {}
+ clear_errors
- @attributes = {} # sdb values
- @attributes_rb = {} # ruby values
- @lobs = {}
- @new_record = true
+ @dirty = {}
- end
+ @attributes = {} # sdb values
+ @attributes_rb = {} # ruby values
+ @lobs = {}
+ @new_record = true
- def initialize_from_db(attrs={})
- initialize_base(attrs)
- attrs.each_pair do |k, v|
- @attributes[k.to_s] = v
- end
- end
+ end
+
+ def initialize_from_db(attrs={})
+ initialize_base(attrs)
+ attrs.each_pair do |k, v|
+ @attributes[k.to_s] = v
+ end
+ end
- def self.inherited(base)
- #puts 'SimpleRecord::Base is inherited by ' + base.inspect
- Callbacks.setup_callbacks(base)
+ def self.inherited(base)
+# puts 'SimpleRecord::Base is inherited by ' + base.inspect
+ Callbacks.setup_callbacks(base)
# base.has_strings :id
- base.has_dates :created, :updated
- base.before_create :set_created, :set_updated
- base.before_update :set_updated
+ base.has_dates :created, :updated
+ base.before_create :set_created, :set_updated
+ base.before_update :set_updated
- end
+ end
- def persisted?
- true
- end
+ def persisted?
+ !@new_record && !destroyed?
+ end
+ def destroyed?
+ @deleted
+ end
- def defined_attributes_local
- # todo: store this somewhere so it doesn't keep going through this
- ret = self.class.defined_attributes
- ret.merge!(self.class.superclass.defined_attributes) if self.class.superclass.respond_to?(:defined_attributes)
- end
+ def defined_attributes_local
+ # todo: store this somewhere so it doesn't keep going through this
+ ret = self.class.defined_attributes
+ ret.merge!(self.class.superclass.defined_attributes) if self.class.superclass.respond_to?(:defined_attributes)
+ end
- class << self;
- attr_accessor :domain_prefix
- end
+ class << self;
+ attr_accessor :domain_prefix
+ end
- #@domain_name_for_class = nil
+ #@domain_name_for_class = nil
- @@cache_store = nil
- # Set the cache to use
- def self.cache_store=(cache)
- @@cache_store = cache
- end
+ @@cache_store = nil
+ # Set the cache to use
+ def self.cache_store=(cache)
+ @@cache_store = cache
+ end
- def self.cache_store
- return @@cache_store
- end
+ def self.cache_store
+ return @@cache_store
+ end
- # If you want a domain prefix for all your models, set it here.
- def self.set_domain_prefix(prefix)
- #puts 'set_domain_prefix=' + prefix
- self.domain_prefix = prefix
- end
+ # If you want a domain prefix for all your models, set it here.
+ def self.set_domain_prefix(prefix)
+ #puts 'set_domain_prefix=' + prefix
+ self.domain_prefix = prefix
+ end
- # Same as set_table_name
- def self.set_table_name(table_name)
- set_domain_name table_name
- end
+ # Same as set_table_name
+ def self.set_table_name(table_name)
+ set_domain_name table_name
+ end
- # Sets the domain name for this class
- def self.set_domain_name(table_name)
- super
- end
+ # Sets the domain name for this class
+ def self.set_domain_name(table_name)
+ super
+ end
- def domain
- self.class.domain
- end
+ def domain
+ self.class.domain
+ end
- def self.domain
- unless @domain
- # This strips off the module if there is one.
- n2 = name.split('::').last || name
+ def self.domain
+ unless @domain
+ # This strips off the module if there is one.
+ n2 = name.split('::').last || name
# puts 'n2=' + n2
- if n2.respond_to?(:tableize)
- @domain = n2.tableize
- else
- @domain = n2.downcase
- end
- set_domain_name @domain
- end
- domain_name_for_class = (SimpleRecord::Base.domain_prefix || "") + @domain.to_s
- domain_name_for_class
- end
-
- def get_attribute_sdb(name)
- name = name.to_sym
- ret = strip_array(@attributes[sdb_att_name(name)])
- return ret
+ if n2.respond_to?(:tableize)
+ @domain = n2.tableize
+ else
+ @domain = n2.downcase
end
+ set_domain_name @domain
+ end
+ domain_name_for_class = (SimpleRecord::Base.domain_prefix || "") + @domain.to_s
+ domain_name_for_class
+ end
- def has_id_on_end(name_s)
- name_s = name_s.to_s
- name_s.length > 3 && name_s[-3..-1] == "_id"
- end
+ def has_id_on_end(name_s)
+ name_s = name_s.to_s
+ name_s.length > 3 && name_s[-3..-1] == "_id"
+ end
- def get_att_meta(name)
- name_s = name.to_s
- att_meta = defined_attributes_local[name.to_sym]
- if att_meta.nil? && has_id_on_end(name_s)
- att_meta = defined_attributes_local[name_s[0..-4].to_sym]
- end
- return att_meta
- end
+ def get_att_meta(name)
+ name_s = name.to_s
+ att_meta = defined_attributes_local[name.to_sym]
+ if att_meta.nil? && has_id_on_end(name_s)
+ att_meta = defined_attributes_local[name_s[0..-4].to_sym]
+ end
+ return att_meta
+ end
- def sdb_att_name(name)
- att_meta = get_att_meta(name)
- if att_meta.type == :belongs_to && !has_id_on_end(name.to_s)
- return "#{name}_id"
- end
- name.to_s
- end
+ def sdb_att_name(name)
+ att_meta = get_att_meta(name)
+ if att_meta.type == :belongs_to && !has_id_on_end(name.to_s)
+ return "#{name}_id"
+ end
+ name.to_s
+ end
- def strip_array(arg)
- if arg.class==Array
- if arg.length==1
- ret = arg[0]
- else
- ret = arg
- end
- else
- ret = arg
- end
- return ret
+ def strip_array(arg)
+ if arg.is_a? Array
+ if arg.length==1
+ ret = arg[0]
+ else
+ ret = arg
end
+ else
+ ret = arg
+ end
+ return ret
+ end
- def make_dirty(arg, value)
- sdb_att_name = sdb_att_name(arg)
- arg = arg.to_s
+ def make_dirty(arg, value)
+ sdb_att_name = sdb_att_name(arg)
+ arg = arg.to_s
# puts "Marking #{arg} dirty with #{value}" if SimpleRecord.logging?
- if @dirty.include?(sdb_att_name)
- old = @dirty[sdb_att_name]
+ if @dirty.include?(sdb_att_name)
+ old = @dirty[sdb_att_name]
# puts "#{sdb_att_name} was already dirty #{old}"
- @dirty.delete(sdb_att_name) if value == old
- else
- old = get_attribute(arg)
+ @dirty.delete(sdb_att_name) if value == old
+ else
+ old = get_attribute(arg)
# puts "dirtifying #{sdb_att_name} old=#{old.inspect} to new=#{value.inspect}" if SimpleRecord.logging?
- @dirty[sdb_att_name] = old if value != old
- end
- end
+ @dirty[sdb_att_name] = old if value != old
+ end
+ end
- def clear_errors
- @errors=SimpleRecord_errors.new
- end
+ def clear_errors
+# @errors=SimpleRecord_errors.new
+ if not (defined?(ActiveModel))
+ @errors=SimpleRecord_errors.new
+ else
+ @errors = ActiveModel::Errors.new(self)
+ end
+ end
- def []=(attribute, values)
- make_dirty(attribute, values)
- super
- end
+ def []=(attribute, values)
+ make_dirty(attribute, values)
+ super
+ end
- def [](attribute)
- super
- end
+ def [](attribute)
+ super
+ end
- def set_created
- set(:created, Time.now)
- end
+ def set_created
+ set(SimpleRecord.options[:created_col] || :created, Time.now)
+ end
- def set_updated
- set(:updated, Time.now)
- end
+ def set_updated
+ set(SimpleRecord.options[:updated_col] || :updated, Time.now)
+ end
- # an aliased method since many people use created_at/updated_at naming convention
- def created_at
- self.created
- end
+ # an aliased method since many people use created_at/updated_at naming convention
+ def created_at
+ self.created
+ end
- # an aliased method since many people use created_at/updated_at naming convention
- def updated_at
- self.updated
- end
+ # an aliased method since many people use created_at/updated_at naming convention
+ def updated_at
+ self.updated
+ end
- def cache_store
- @@cache_store
- end
+ def read_attribute_for_validation(key)
+ @attributes[key.to_s]
+ end
- def domain_ok(ex, options={})
- if (ex.message().index("NoSuchDomain") != nil)
- dom = options[:domain] || domain
- self.class.create_domain(dom)
- return true
- end
- return false
- end
+ def cache_store
+ @@cache_store
+ end
+ def domain_ok(ex, options={})
+ if (ex.message().index("NoSuchDomain") != nil)
+ dom = options[:domain] || domain
+ self.class.create_domain(dom)
+ return true
+ end
+ return false
+ end
- def new_record?
- # todo: new_record in activesdb should align with how we're defining a new record here, ie: if id is nil
- super
- end
+ def new_record?
+ # todo: new_record in activesdb should align with how we're defining a new record here, ie: if id is nil
+ super
+ end
- @create_domain_called = false
- # Options:
- # - :except => Array of attributes to NOT save
- # - :dirty => true - Will only store attributes that were modified. To make it save regardless and have it update the :updated value, include this and set it to false.
- # - :domain => Explicitly define domain to use.
- #
- def save(options={})
+ @create_domain_called = false
+
+ # Options:
+ # - :except => Array of attributes to NOT save
+ # - :dirty => true - Will only store attributes that were modified. To make it save regardless and have it update the :updated value, include this and set it to false.
+ # - :domain => Explicitly define domain to use.
+ #
+ def save(options={})
# puts 'SAVING: ' + self.inspect if SimpleRecord.logging?
- # todo: Clean out undefined values in @attributes (in case someone set the attributes hash with values that they hadn't defined)
- clear_errors
- # todo: decide whether this should go before pre_save or after pre_save? pre_save dirties "updated" and perhaps other items due to callbacks
- if options[:dirty]
+ # todo: Clean out undefined values in @attributes (in case someone set the attributes hash with values that they hadn't defined)
+ clear_errors
+ # todo: decide whether this should go before pre_save or after pre_save? pre_save dirties "updated" and perhaps other items due to callbacks
+ if options[:dirty]
# puts '@dirtyA=' + @dirty.inspect
- return true if @dirty.size == 0 # Nothing to save so skip it
- end
- is_create = self[:id].nil?
- ok = pre_save(options) # Validates and sets ID
- if ok
- begin
- dirty = @dirty
+ return true if @dirty.size == 0 # Nothing to save so skip it
+ end
+ is_create = self[:id].nil?
+ ok = pre_save(options) # Validates and sets ID
+ if ok
+ begin
+ dirty = @dirty
# puts 'dirty before=' + @dirty.inspect
- if options[:dirty]
+ if options[:dirty]
# puts '@dirty=' + @dirty.inspect
- return true if @dirty.size == 0 # This should probably never happen because after pre_save, created/updated dates are changed
- options[:dirty_atts] = @dirty
- end
- to_delete = get_atts_to_delete
- SimpleRecord.stats.saves += 1
-
- if self.class.is_sharded?
- options[:domain] = sharded_domain
- end
-
- if super(options)
- self.class.cache_results(self)
- delete_niled(to_delete)
- save_lobs(dirty)
- after_save_cleanup
- if (is_create ? run_after_create : run_after_update) && run_after_save
+ return true if @dirty.size == 0 # This should probably never happen because after pre_save, created/updated dates are changed
+ options[:dirty_atts] = @dirty
+ end
+ to_delete = get_atts_to_delete
+ SimpleRecord.stats.saves += 1
+
+ if self.class.is_sharded?
+ options[:domain] = sharded_domain
+ end
+
+ if super(options)
+ self.class.cache_results(self)
+ delete_niled(to_delete)
+ save_lobs(dirty)
+ after_save_cleanup
+ if (is_create ? run_after_create : run_after_update) && run_after_save
# puts 'all good?'
- return true
- else
- return false
- end
- else
- return false
- end
- rescue Aws::AwsError => ex
- # puts "RESCUED in save: " + $!
- if (domain_ok(ex, options))
- if !@create_domain_called
- @create_domain_called = true
- save(options)
- else
- raise $!
- end
- else
- raise $!
- end
- end
+ return true
else
- #@debug = "not saved"
- return false
+ return false
end
- end
+ else
+ return false
+ end
+ rescue Aws::AwsError => ex
+ # puts "RESCUED in save: " + $!
+ # Domain is created in aws lib now using :create_domain=>true
+# if (domain_ok(ex, options))
+# if !@create_domain_called
+# @create_domain_called = true
+# save(options)
+# else
+# raise $!
+# end
+# else
+# raise $!
+# end
+ raise ex
+ end
+ else
+ #@debug = "not saved"
+ return false
+ end
+ end
- def save_lobs(dirty=nil)
+ def save_lobs(dirty=nil)
# puts 'dirty.inspect=' + dirty.inspect
- dirty = @dirty if dirty.nil?
- defined_attributes_local.each_pair do |k, v|
- if v.type == :clob
-# puts 'storing clob '
- if dirty.include?(k.to_s)
- begin
- val = @lobs[k]
-# puts 'val=' + val.inspect
- s3_bucket.put(s3_lob_id(k), val)
- rescue Aws::AwsError => ex
- if ex.include? /NoSuchBucket/
- s3_bucket(true).put(s3_lob_id(k), val)
- else
- raise ex
- end
- end
- SimpleRecord.stats.s3_puts += 1
- else
+ dirty = @dirty if dirty.nil?
+ all_clobs = {}
+ dirty_clobs = {}
+ defined_attributes_local.each_pair do |k, v|
+ # collect up the clobs in case it's a single put
+ if v.type == :clob
+ val = @lobs[k]
+ all_clobs[k] = val
+ if dirty.include?(k.to_s)
+ dirty_clobs[k] = val
+ else
# puts 'NOT DIRTY'
- end
+ end
- end
- end
end
+ end
+ if dirty_clobs.size > 0
+ if self.class.get_sr_config[:single_clob]
+ # all clobs in one chunk
+ # using json for now, could change later
+ val = all_clobs.to_json
+ puts 'val=' + val.inspect
+ put_lob(single_clob_id, val, :s3_bucket=>:new)
+ else
+ dirty_clobs.each_pair do |k, val|
+ put_lob(s3_lob_id(k), val)
+ end
+ end
+ end
+ end
+
+ def delete_lobs
+ defined_attributes_local.each_pair do |k, v|
+ if v.type == :clob
+ if self.class.get_sr_config[:single_clob]
+ s3_bucket(false, :s3_bucket=>:new).delete_key(single_clob_id)
+ SimpleRecord.stats.s3_deletes += 1
+ return
+ else
+ s3_bucket.delete_key(s3_lob_id(k))
+ SimpleRecord.stats.s3_deletes += 1
+ end
+ end
+ end
+ end
+
+
+ def put_lob(k, val, options={})
+ begin
+ s3_bucket(false, options).put(k, val)
+ rescue Aws::AwsError => ex
+ if ex.include? /NoSuchBucket/
+ s3_bucket(true, options).put(k, val)
+ else
+ raise ex
+ end
+ end
+ SimpleRecord.stats.s3_puts += 1
+ end
- def is_dirty?(name)
- # todo: should change all the dirty stuff to symbols?
+
+ def is_dirty?(name)
+ # todo: should change all the dirty stuff to symbols?
# puts '@dirty=' + @dirty.inspect
# puts 'name=' +name.to_s
- @dirty.include? name.to_s
- end
+ @dirty.include? name.to_s
+ end
- def s3
+ def s3
- return SimpleRecord.s3 if SimpleRecord.s3
- # todo: should optimize this somehow, like use the same connection_mode as used in SR
- # or keep open while looping in ResultsArray.
- Aws::S3.new(SimpleRecord.aws_access_key, SimpleRecord.aws_secret_key)
- end
+ return SimpleRecord.s3 if SimpleRecord.s3
+ # todo: should optimize this somehow, like use the same connection_mode as used in SR
+ # or keep open while looping in ResultsArray.
+ Aws::S3.new(SimpleRecord.aws_access_key, SimpleRecord.aws_secret_key)
+ end
- def s3_bucket(create=false)
- s3.bucket(s3_bucket_name, create)
- end
+ # options:
+ # :s3_bucket => :old/:new/"#{any_bucket_name}". :new if want to use new bucket. Defaults to :old for backwards compatability.
+ def s3_bucket(create=false, options={})
+ s3.bucket(s3_bucket_name(options[:s3_bucket]), create)
+ end
- def s3_bucket_name
- SimpleRecord.aws_access_key + "_lobs"
- end
+ def s3_bucket_name(s3_bucket_option=:old)
+ if s3_bucket_option == :new || SimpleRecord.options[:s3_bucket] == :new
+ # this is the bucket that will be used going forward for anything related to s3
+ ret = "simple_record_#{SimpleRecord.aws_access_key}"
+ elsif !SimpleRecord.options[:s3_bucket].nil? && SimpleRecord.options[:s3_bucket] != :old
+ ret = SimpleRecord.options[:s3_bucket]
+ else
+ ret = SimpleRecord.aws_access_key + "_lobs"
+ end
+ ret
+ end
- def s3_lob_id(name)
- self.id + "_" + name.to_s
- end
+ def s3_lob_id(name)
+ # if s3_bucket is not nil and not :old, then we use the new key.
+ if !SimpleRecord.options[:s3_bucket].nil? && SimpleRecord.options[:s3_bucket] != :old
+ "lobs/#{self.id}_#{name}"
+ else
+ self.id + "_" + name.to_s
+ end
+ end
- def save!(options={})
- save(options) || raise(RecordNotSaved)
- end
+ def single_clob_id
+ "lobs/#{self.id}_single_clob"
+ end
- def save_with_validation!(options={})
- if valid?
- save
- else
- raise RecordInvalid.new(self)
- end
- end
+ def save!(options={})
+ save(options) || raise(RecordNotSaved)
+ end
+
+ def self.create(attributes={})
+# puts "About to create in domain #{domain}"
+ super
+ end
+ def self.create!(attributes={})
+ item = self.new(attributes)
+ item.save!
+ item
+ end
+
+ def save_with_validation!(options={})
+ if valid?
+ save
+ else
+ raise RecordInvalid.new(self)
+ end
+ end
- def self.get_encryption_key()
- key = SimpleRecord.options[:encryption_key]
+
+ def self.get_encryption_key()
+ key = SimpleRecord.options[:encryption_key]
# if key.nil?
# puts 'WARNING: Encrypting attributes with your AWS Access Key. You should use your own :encryption_key so it doesn\'t change'
# key = connection.aws_access_key_id # default to aws access key. NOT recommended in case you start using a new key
# end
- return key
- end
-
-
- def validate
- true
- end
-
- def validate_on_create
- true
- end
-
- def validate_on_update
- true
- end
+ return key
+ end
+ def validate
+ true
+ end
- def pre_save(options)
+ def validate_on_create
+ true
+ end
- is_create = self[:id].nil?
- ok = run_before_validation && (is_create ? run_before_validation_on_create : run_before_validation_on_update)
- return false unless ok
+ def validate_on_update
+ true
+ end
- validate()
- is_create ? validate_on_create : validate_on_update
-# puts 'AFTER VALIDATIONS, ERRORS=' + errors.inspect
- if (!errors.nil? && errors.size > 0)
-# puts 'THERE ARE ERRORS, returning false'
- return false
- end
+ def pre_save(options)
- ok = run_after_validation && (is_create ? run_after_validation_on_create : run_after_validation_on_update)
- return false unless ok
+ is_create = self[:id].nil?
+ ok = run_before_validation && (is_create ? run_before_validation_on_create : run_before_validation_on_update)
+ return false unless ok
- ok = respond_to?('before_save') ? before_save : true
- if ok
- if is_create && respond_to?('before_create')
- ok = before_create
- elsif !is_create && respond_to?('before_update')
- ok = before_update
- end
- end
- if ok
- ok = run_before_save && (is_create ? run_before_create : run_before_update)
- end
- if ok
- # Now translate all fields into SimpleDB friendly strings
+# validate()
+# is_create ? validate_on_create : validate_on_update
+ if !valid?
+ return false
+ end
+#
+## puts 'AFTER VALIDATIONS, ERRORS=' + errors.inspect
+# if (!errors.nil? && errors.size > 0)
+## puts 'THERE ARE ERRORS, returning false'
+# return false
+# end
+
+ ok = run_after_validation && (is_create ? run_after_validation_on_create : run_after_validation_on_update)
+ return false unless ok
+
+ ok = respond_to?('before_save') ? before_save : true
+ if ok
+ if is_create && respond_to?('before_create')
+ ok = before_create
+ elsif !is_create && respond_to?('before_update')
+ ok = before_update
+ end
+ end
+ if ok
+ ok = run_before_save && (is_create ? run_before_create : run_before_update)
+ end
+ if ok
+ # Now translate all fields into SimpleDB friendly strings
# convert_all_atts_to_sdb()
- end
- prepare_for_update
- ok
- end
+ end
+ prepare_for_update
+ ok
+ end
- def get_atts_to_delete
- to_delete = []
- changes.each_pair do |key, v|
- if v[1].nil?
- to_delete << key
- @attributes.delete(key)
- end
- end
+ def get_atts_to_delete
+ to_delete = []
+ changes.each_pair do |key, v|
+ if v[1].nil?
+ to_delete << key
+ @attributes.delete(key)
+ end
+ end
# @attributes.each do |key, value|
## puts 'key=' + key.inspect + ' value=' + value.inspect
# if value.nil? || (value.is_a?(Array) && value.size == 0) || (value.is_a?(Array) && value.size == 1 && value[0] == nil)
# to_delete << key
# @attributes.delete(key)
# end
# end
- return to_delete
- end
-
- # Run pre_save on each object, then runs batch_put_attributes
- # Returns
- def self.batch_save(objects, options={})
- results = []
- to_save = []
- if objects && objects.size > 0
- objects.each do |o|
- ok = o.pre_save(options)
- raise "Pre save failed on object [" + o.inspect + "]" if !ok
- results << ok
- next if !ok # todo: this shouldn't be here should it? raises above
- o.pre_save2
- to_save << Aws::SdbInterface::Item.new(o.id, o.attributes, true)
- if to_save.size == 25 # Max amount SDB will accept
- connection.batch_put_attributes(domain, to_save)
- to_save.clear
- end
- end
- end
- connection.batch_put_attributes(domain, to_save) if to_save.size > 0
- objects.each do |o|
- o.save_lobs(nil)
- end
- results
- end
+ return to_delete
+ end
- #
- # Usage: ClassName.delete id
- #
- def self.delete(id)
- connection.delete_attributes(domain, id)
- end
+ # Run pre_save on each object, then runs batch_put_attributes
+ # Returns
+ def self.batch_save(objects, options={})
+ options[:create_domain] = true if options[:create_domain].nil?
+ results = []
+ to_save = []
+ if objects && objects.size > 0
+ objects.each do |o|
+ ok = o.pre_save(options)
+ raise "Pre save failed on object [" + o.inspect + "]" if !ok
+ results << ok
+ next if !ok # todo: this shouldn't be here should it? raises above
+ o.pre_save2
+ to_save << Aws::SdbInterface::Item.new(o.id, o.attributes, true)
+ if to_save.size == 25 # Max amount SDB will accept
+ connection.batch_put_attributes(domain, to_save, options)
+ to_save.clear
+ end
+ end
+ end
+ connection.batch_put_attributes(domain, to_save, options) if to_save.size > 0
+ objects.each do |o|
+ o.save_lobs(nil)
+ end
+ results
+ end
- def self.delete_all(*params)
- # could make this quicker by just getting item_names and deleting attributes rather than creating objects
- obs = self.find(params)
- i = 0
- obs.each do |a|
- a.delete
- i+=1
- end
- return i
- end
+ # Pass in an array of objects
+ def self.batch_delete(objects, options={})
+ if objects
+ # 25 item limit, we should maybe handle this limit in here.
+ connection.batch_delete_attributes @domain, objects.collect { |x| x.id }
+ end
+ end
- def self.destroy_all(*params)
- obs = self.find(params)
- i = 0
- obs.each do |a|
- a.destroy
- i+=1
- end
- return i
- end
+ #
+ # Usage: ClassName.delete id
+ #
+ def self.delete(id)
+ connection.delete_attributes(domain, id)
+ @deleted = true
+ end
- def delete()
- # TODO: DELETE CLOBS, etc from s3
- options = {}
- if self.class.is_sharded?
- options[:domain] = sharded_domain
- end
- super(options)
- end
+ # Pass in the same OPTIONS you'd pass into a find(:all, OPTIONS)
+ def self.delete_all(options)
+ # could make this quicker by just getting item_names and deleting attributes rather than creating objects
+ obs = self.find(:all, options)
+ i = 0
+ obs.each do |a|
+ a.delete
+ i+=1
+ end
+ return i
+ end
- def destroy
- return run_before_destroy && delete && run_after_destroy
- end
+ # Pass in the same OPTIONS you'd pass into a find(:all, OPTIONS)
+ def self.destroy_all(options)
+ obs = self.find(:all, options)
+ i = 0
+ obs.each do |a|
+ a.destroy
+ i+=1
+ end
+ return i
+ end
+ def delete(options={})
+ if self.class.is_sharded?
+ options[:domain] = sharded_domain
+ end
+ super(options)
- # Since SimpleDB supports multiple attributes per value, the values are an array.
- # This method will return the value unwrapped if it's the only, otherwise it will return the array.
- def get_attribute(name)
-# puts "GET #{arg}"
- # Check if this arg is already converted
- name_s = name.to_s
- name = name.to_sym
- att_meta = get_att_meta(name)
-# puts "att_meta for #{name}: " + att_meta.inspect
- if att_meta && att_meta.type == :clob
- ret = @lobs[name]
-# puts 'get_attribute clob ' + ret.inspect
- if ret
- if ret.is_a? RemoteNil
- return nil
- else
- return ret
- end
- end
- # get it from s3
- unless new_record?
- begin
- ret = s3_bucket.get(s3_lob_id(name))
-# puts 'got from s3 ' + ret.inspect
- SimpleRecord.stats.s3_gets += 1
- rescue Aws::AwsError => ex
- if ex.include? /NoSuchKey/
- ret = nil
- else
- raise ex
- end
- end
-
- if ret.nil?
- ret = RemoteNil.new
- end
- end
- @lobs[name] = ret
- return nil if ret.is_a? RemoteNil
- return ret
- else
- @attributes_rb = {} unless @attributes_rb # was getting errors after upgrade.
- ret = @attributes_rb[name_s] # instance_variable_get(instance_var)
- return ret unless ret.nil?
- return nil if ret.is_a? RemoteNil
- ret = get_attribute_sdb(name)
- ret = sdb_to_ruby(name, ret)
- @attributes_rb[name_s] = ret
- return ret
- end
+ # delete lobs now too
+ delete_lobs
+ end
- end
+ def destroy
+ return run_before_destroy && delete && run_after_destroy
+ end
- def delete_niled(to_delete)
+ def delete_niled(to_delete)
# puts 'to_delete=' + to_delete.inspect
- if to_delete.size > 0
+ if to_delete.size > 0
# puts 'Deleting attributes=' + to_delete.inspect
- SimpleRecord.stats.deletes += 1
- delete_attributes to_delete
- to_delete.each do |att|
- att_meta = get_att_meta(att)
- if att_meta.type == :clob
- s3_bucket.key(s3_lob_id(att)).delete
- end
- end
- end
- end
+ SimpleRecord.stats.deletes += 1
+ delete_attributes to_delete
+ to_delete.each do |att|
+ att_meta = get_att_meta(att)
+ if att_meta.type == :clob
+ s3_bucket.key(s3_lob_id(att)).delete
+ end
+ end
+ end
+ end
- def reload
- super()
- end
+ def reload
+ super()
+ end
- def update_attributes(atts)
- set_attributes(atts)
- save
- end
+ def update_attributes(atts)
+ set_attributes(atts)
+ save
+ end
- def update_attributes!(atts)
- set_attributes(atts)
- save!
- end
+ def update_attributes!(atts)
+ set_attributes(atts)
+ save!
+ end
- def self.quote_regexp(a, re)
- a =~ re
- #was there a match?
- if $&
- before=$`
- middle=$&
- after =$'
-
- before =~ /'$/ #is there already a quote immediately before the match?
- unless $&
- return "#{before}'#{middle}'#{quote_regexp(after, re)}" #if not, put quotes around the match
- else
- return "#{before}#{middle}#{quote_regexp(after, re)}" #if so, assume it is quoted already and move on
- end
- else
- #no match, just return the string
- return a
- end
- end
+ def self.quote_regexp(a, re)
+ a =~ re
+ #was there a match?
+ if $&
+ before=$`
+ middle=$&
+ after =$'
- def self.create(attributes={})
-# puts "About to create in domain #{domain}"
- super
+ before =~ /'$/ #is there already a quote immediately before the match?
+ unless $&
+ return "#{before}'#{middle}'#{quote_regexp(after, re)}" #if not, put quotes around the match
+ else
+ return "#{before}#{middle}#{quote_regexp(after, re)}" #if so, assume it is quoted already and move on
end
+ else
+ #no match, just return the string
+ return a
+ end
+ end
- @@regex_no_id = /.*Couldn't find.*with ID.*/
-
- #
- # Usage:
- # Find by ID:
- # MyModel.find(ID)
- #
- # Query example:
- # MyModel.find(:all, :conditions=>["name = ?", name], :order=>"created desc", :limit=>10)
- #
- # Extra options:
- # :per_token => the number of results to return per next_token, max is 2500.
- # :consistent_read => true/false -- as per http://developer.amazonwebservices.com/connect/entry.jspa?externalID=3572
- def self.find(*params)
- #puts 'params=' + params.inspect
-
- q_type = :all
- select_attributes=[]
- if params.size > 0
- q_type = params[0]
- end
- options = {}
- if params.size > 1
- options = params[1]
- end
-
- if !options[:shard_find] && is_sharded?
- # then break off and get results across all shards
- return find_sharded(*params)
- end
-
- # Pad and Offset number attributes
- params_dup = params.dup
- if params.size > 1
- options = params[1]
- #puts 'options=' + options.inspect
- #puts 'after collect=' + options.inspect
- convert_condition_params(options)
- per_token = options[:per_token]
- consistent_read = options[:consistent_read]
- if per_token || consistent_read then
- op_dup = options.dup
- op_dup[:limit] = per_token # simpledb uses Limit as a paging thing, not what is normal
- op_dup[:consistent_read] = consistent_read
- params_dup[1] = op_dup
- end
- end
+ @@regex_no_id = /.*Couldn't find.*with ID.*/
+
+ #
+ # Usage:
+ # Find by ID:
+ # MyModel.find(ID)
+ #
+ # Query example:
+ # MyModel.find(:all, :conditions=>["name = ?", name], :order=>"created desc", :limit=>10)
+ #
+ # Extra options:
+ # :per_token => the number of results to return per next_token, max is 2500.
+ # :consistent_read => true/false -- as per http://developer.amazonwebservices.com/connect/entry.jspa?externalID=3572
+ # :retries => maximum number of times to retry this query on an error response.
+ # :shard => shard name or array of shard names to use on this query.
+ def self.find(*params)
+ #puts 'params=' + params.inspect
+
+ q_type = :all
+ select_attributes=[]
+ if params.size > 0
+ q_type = params[0]
+ end
+ options = {}
+ if params.size > 1
+ options = params[1]
+ end
+
+ if !options[:shard_find] && is_sharded?
+ # then break off and get results across all shards
+ return find_sharded(*params)
+ end
+
+ # Pad and Offset number attributes
+ params_dup = params.dup
+ if params.size > 1
+ options = params[1]
+ #puts 'options=' + options.inspect
+ #puts 'after collect=' + options.inspect
+ convert_condition_params(options)
+ per_token = options[:per_token]
+ consistent_read = options[:consistent_read]
+ if per_token || consistent_read then
+ op_dup = options.dup
+ op_dup[:limit] = per_token # simpledb uses Limit as a paging thing, not what is normal
+ op_dup[:consistent_read] = consistent_read
+ params_dup[1] = op_dup
+ end
+ end
# puts 'params2=' + params.inspect
- ret = q_type == :all ? [] : nil
- begin
-
- results=find_with_metadata(*params_dup)
+ ret = q_type == :all ? [] : nil
+ begin
+ results=find_with_metadata(*params_dup)
# puts "RESULT=" + results.inspect
- write_usage(:select, domain, q_type, options, results)
- #puts 'params3=' + params.inspect
- SimpleRecord.stats.selects += 1
- if q_type == :count
- ret = results[:count]
- elsif q_type == :first
- ret = results[:items].first
- # todo: we should store request_id and box_usage with the object maybe?
- cache_results(ret)
- elsif results[:single]
- ret = results[:single]
- cache_results(ret)
- else
- if results[:items] #.is_a?(Array)
- cache_results(results[:items])
- ret = SimpleRecord::ResultsArray.new(self, params, results, next_token)
- end
- end
- rescue Aws::AwsError, SimpleRecord::ActiveSdb::ActiveSdbError => ex
+ write_usage(:select, domain, q_type, options, results)
+ #puts 'params3=' + params.inspect
+ SimpleRecord.stats.selects += 1
+ if q_type == :count
+ ret = results[:count]
+ elsif q_type == :first
+ ret = results[:items].first
+ # todo: we should store request_id and box_usage with the object maybe?
+ cache_results(ret)
+ elsif results[:single]
+ ret = results[:single]
+ cache_results(ret)
+ else
+ if results[:items] #.is_a?(Array)
+ cache_results(results[:items])
+ ret = SimpleRecord::ResultsArray.new(self, params, results, next_token)
+ end
+ end
+ rescue Aws::AwsError, SimpleRecord::ActiveSdb::ActiveSdbError => ex
# puts "RESCUED: " + ex.message
- if (ex.message().index("NoSuchDomain") != nil)
- # this is ok
- elsif (ex.message() =~ @@regex_no_id)
- ret = nil
- else
- raise ex
- end
- end
-# puts 'single2=' + ret.inspect
- return ret
+ if (ex.message().index("NoSuchDomain") != nil)
+ # this is ok
+ elsif (ex.message() =~ @@regex_no_id)
+ ret = nil
+ else
+ raise ex
end
+ end
+# puts 'single2=' + ret.inspect
+ return ret
+ end
- def self.select(*params)
- return find(*params)
- end
+ def self.select(*params)
+ return find(*params)
+ end
- def self.all(*args)
- find(:all, *args)
- end
+ def self.all(*args)
+ find(:all, *args)
+ end
- def self.first(*args)
- find(:first, *args)
- end
+ def self.first(*args)
+ find(:first, *args)
+ end
- def self.count(*args)
- find(:count, *args)
- end
+ def self.count(*args)
+ find(:count, *args)
+ end
- # This gets less and less efficient the higher the page since SimpleDB has no way
- # to start at a specific row. So it will iterate from the first record and pull out the specific pages.
- def self.paginate(options={})
+ # This gets less and less efficient the higher the page since SimpleDB has no way
+ # to start at a specific row. So it will iterate from the first record and pull out the specific pages.
+ def self.paginate(options={})
# options = args.pop
# puts 'paginate options=' + options.inspect if SimpleRecord.logging?
- page = options[:page] || 1
- per_page = options[:per_page] || 50
+ page = options[:page] || 1
+ per_page = options[:per_page] || 50
# total = options[:total_entries].to_i
- options[:page] = page.to_i # makes sure it's to_i
- options[:per_page] = per_page.to_i
- options[:limit] = options[:page] * options[:per_page]
+ options[:page] = page.to_i # makes sure it's to_i
+ options[:per_page] = per_page.to_i
+ options[:limit] = options[:page] * options[:per_page]
# puts 'paging options=' + options.inspect
- fr = find(:all, options)
- return fr
+ fr = find(:all, options)
+ return fr
- end
+ end
- def self.convert_condition_params(options)
- return if options.nil?
- conditions = options[:conditions]
- if !conditions.nil? && conditions.size > 1
- # all after first are values
- conditions.collect! { |x|
- Translations.pad_and_offset(x)
- }
- end
+ def self.convert_condition_params(options)
+ return if options.nil?
+ conditions = options[:conditions]
+ if !conditions.nil? && conditions.size > 1
+ # all after first are values
+ conditions.collect! { |x|
+ Translations.pad_and_offset(x)
+ }
+ end
- end
+ end
- def self.cache_results(results)
- if !cache_store.nil? && !results.nil?
- if results.is_a?(Array)
- # todo: cache each result
- results.each do |item|
- class_name = item.class.name
- id = item.id
- cache_key = self.cache_key(class_name, id)
- #puts 'caching result at ' + cache_key + ': ' + results.inspect
- cache_store.write(cache_key, item, :expires_in =>30)
- end
- else
- class_name = results.class.name
- id = results.id
- cache_key = self.cache_key(class_name, id)
- #puts 'caching result at ' + cache_key + ': ' + results.inspect
- cache_store.write(cache_key, results, :expires_in =>30)
- end
- end
+ def self.cache_results(results)
+ if !cache_store.nil? && !results.nil?
+ if results.is_a?(Array)
+ # todo: cache each result
+ results.each do |item|
+ class_name = item.class.name
+ id = item.id
+ cache_key = self.cache_key(class_name, id)
+ #puts 'caching result at ' + cache_key + ': ' + results.inspect
+ cache_store.write(cache_key, item, :expires_in =>30)
+ end
+ else
+ class_name = results.class.name
+ id = results.id
+ cache_key = self.cache_key(class_name, id)
+ #puts 'caching result at ' + cache_key + ': ' + results.inspect
+ cache_store.write(cache_key, results, :expires_in =>30)
end
+ end
+ end
- def self.cache_key(class_name, id)
- return class_name + "/" + id.to_s
- end
+ def self.cache_key(class_name, id)
+ return class_name + "/" + id.to_s
+ end
- @@debug=""
+ @@debug=""
- def self.debug
- @@debug
- end
+ def self.debug
+ @@debug
+ end
- def self.sanitize_sql(*params)
- return ActiveRecord::Base.sanitize_sql(*params)
- end
+ def self.sanitize_sql(*params)
+ return ActiveRecord::Base.sanitize_sql(*params)
+ end
- def self.table_name
- return domain
- end
+ def self.table_name
+ return domain
+ end
- def changed
- return @dirty.keys
- end
+ def changed
+ return @dirty.keys
+ end
- def changed?
- return @dirty.size > 0
- end
+ def changed?
+ return @dirty.size > 0
+ end
- def changes
- ret = {}
- #puts 'in CHANGES=' + @dirty.inspect
- @dirty.each_pair { |key, value| ret[key] = [value, get_attribute(key)] }
- return ret
- end
+ def changes
+ ret = {}
+ #puts 'in CHANGES=' + @dirty.inspect
+ @dirty.each_pair { |key, value| ret[key] = [value, get_attribute(key)] }
+ return ret
+ end
- def after_save_cleanup
- @dirty = {}
- end
+ def after_save_cleanup
+ @dirty = {}
+ end
- def hash
- # same as ActiveRecord
- id.hash
- end
+ def hash
+ # same as ActiveRecord
+ id.hash
+ end
- end
+ end
- class Activerecordtosdb_subrecord_array
- def initialize(subname, referencename, referencevalue)
- @subname =subname.classify
- @referencename =referencename.tableize.singularize + "_id"
- @referencevalue=referencevalue
- end
+ class Activerecordtosdb_subrecord_array
+ def initialize(subname, referencename, referencevalue)
+ @subname =subname.classify
+ @referencename =referencename.tableize.singularize + "_id"
+ @referencevalue=referencevalue
+ end
- # Performance optimization if you know the array should be empty
+ # Performance optimization if you know the array should be empty
- def init_empty
- @records = []
- end
+ def init_empty
+ @records = []
+ end
- def load
- if @records.nil?
- @records = find_all
- end
- return @records
- end
+ def load
+ if @records.nil?
+ @records = find_all
+ end
+ return @records
+ end
- def [](key)
- return load[key]
- end
+ def [](key)
+ return load[key]
+ end
- def <<(ob)
- return load << ob
- end
+ def first
+ load[0]
+ end
- def count
- return load.count
- end
+ def <<(ob)
+ return load << ob
+ end
- def size
- return count
- end
+ def count
+ return load.count
+ end
- def each(*params, &block)
- return load.each(*params) { |record| block.call(record) }
- end
+ def size
+ return count
+ end
- def find_all(*params)
- find(:all, *params)
- end
+ def each(*params, &block)
+ return load.each(*params) { |record| block.call(record) }
+ end
- def empty?
- return load.empty?
- end
+ def find_all(*params)
+ find(:all, *params)
+ end
- def build(*params)
- params[0][@referencename]=@referencevalue
- eval(@subname).new(*params)
- end
+ def empty?
+ return load.empty?
+ end
- def create(*params)
- params[0][@referencename]=@referencevalue
- record = eval(@subname).new(*params)
- record.save
- end
+ def build(*params)
+ params[0][@referencename]=@referencevalue
+ eval(@subname).new(*params)
+ end
- def find(*params)
- query=[:first, {}]
- #{:conditions=>"id=>1"}
- if params[0]
- if params[0]==:all
- query[0]=:all
- end
- end
+ def create(*params)
+ params[0][@referencename]=@referencevalue
+ record = eval(@subname).new(*params)
+ record.save
+ end
- if params[1]
- query[1]=params[1]
- if query[1][:conditions]
- query[1][:conditions]=SimpleRecord::Base.sanitize_sql(query[1][:conditions])+" AND "+ SimpleRecord::Base.sanitize_sql(["#{@referencename} = ?", @referencevalue])
- #query[1][:conditions]=Activerecordtosdb.sanitize_sql(query[1][:conditions])+" AND id='#{@id}'"
- else
- query[1][:conditions]=["#{@referencename} = ?", @referencevalue]
- #query[1][:conditions]="id='#{@id}'"
- end
- else
- query[1][:conditions]=["#{@referencename} = ?", @referencevalue]
- #query[1][:conditions]="id='#{@id}'"
- end
+ def find(*params)
+ query=[:first, {}]
+ #{:conditions=>"id=>1"}
+ if params[0]
+ if params[0]==:all
+ query[0]=:all
+ end
+ end
- return eval(@subname).find(*query)
+ if params[1]
+ query[1]=params[1]
+ if query[1][:conditions]
+ query[1][:conditions]=SimpleRecord::Base.sanitize_sql(query[1][:conditions])+" AND "+ SimpleRecord::Base.sanitize_sql(["#{@referencename} = ?", @referencevalue])
+ #query[1][:conditions]=Activerecordtosdb.sanitize_sql(query[1][:conditions])+" AND id='#{@id}'"
+ else
+ query[1][:conditions]=["#{@referencename} = ?", @referencevalue]
+ #query[1][:conditions]="id='#{@id}'"
end
+ else
+ query[1][:conditions]=["#{@referencename} = ?", @referencevalue]
+ #query[1][:conditions]="id='#{@id}'"
+ end
+ return eval(@subname).find(*query)
end
- # This is simply a place holder so we don't keep doing gets to s3 or simpledb if already checked.
- class RemoteNil
+ end
- end
+ # This is simply a place holder so we don't keep doing gets to s3 or simpledb if already checked.
+ class RemoteNil
+
+ end
end
View
12 lib/simple_record/active_sdb.rb
@@ -631,10 +631,10 @@ def build_conditions(conditions) # :nodoc:
# This will currently return and's, or's and betweens. Doesn't hurt anything, but could remove.
def parse_condition_fields(conditions)
return nil unless conditions && conditions.present?
- rx = /\b(\w*)[\s|>=|<=|!=|=|>|<|like]/
+ rx = /\b(\w*)[\s|>=|<=|!=|=|>|<|like|between]/
fields = conditions[0].scan(rx)
# puts 'condition_fields = ' + fields.inspect
- fields[0]
+ fields.flatten
end
end
@@ -857,6 +857,7 @@ def put_attributes(attrs)
#
# compare to +put+ method
def save(options={})
+ options[:create_domain] = true if options[:create_domain].nil?
pre_save2
atts_to_save = @attributes.dup
#puts 'atts_to_save=' + atts_to_save.inspect
@@ -873,7 +874,7 @@ def save(options={})
end
dom = options[:domain] || domain
#puts 'atts_to_save2=' + atts_to_save.inspect
- connection.put_attributes(dom, id, atts_to_save, :replace)
+ connection.put_attributes(dom, id, atts_to_save, :replace, options)
apres_save2
@attributes
end
@@ -971,11 +972,6 @@ def delete(options={})
connection.delete_attributes(options[:domain] || domain, id)
end
- # Item ID
-# def to_s
-# @id
-# end
-
# Returns true if this object hasn�t been saved yet.
def new_record?
@new_record
View
129 lib/simple_record/attributes.rb
@@ -22,6 +22,17 @@ def self.defined_attributes
module ClassMethods
+ # Add configuration to this particular class.
+ # :single_clob=> true/false. If true will store all clobs as a single object in s3. Default is false.
+ def sr_config(options={})
+ get_sr_config
+ @sr_config.merge!(options)
+ end
+
+ def get_sr_config
+ @sr_config ||= {}
+ end
+
def defined_attributes
@attributes ||= {}
@attributes
@@ -40,6 +51,8 @@ def has_attributes2(args, options_for_all={})
# then attribute may have extra options
arg_options = arg
arg = arg_options[:name].to_sym
+ else
+ arg = arg.to_sym
end
type = options_for_all[:type] || :string
attr = Attribute.new(type, arg_options)
@@ -108,26 +121,26 @@ def has_booleans(*args)
def are_ints(*args)
# puts 'calling are_ints: ' + args.inspect
args.each do |arg|
- defined_attributes[arg].type = :int
+ defined_attributes[arg.to_sym].type = :int
end
end
def are_floats(*args)
# puts 'calling are_ints: ' + args.inspect
args.each do |arg|
- defined_attributes[arg].type = :float
+ defined_attributes[arg.to_sym].type = :float
end
end
def are_dates(*args)
args.each do |arg|
- defined_attributes[arg].type = :date
+ defined_attributes[arg.to_sym].type = :date
end
end
def are_booleans(*args)
args.each do |arg|
- defined_attributes[arg].type = :boolean
+ defined_attributes[arg.to_sym].type = :boolean
end
end
@@ -136,8 +149,6 @@ def has_clobs(*args)
end
- @@virtuals=[]
-
def has_virtuals(*args)
@@virtuals = args
args.each do |arg|
@@ -224,15 +235,17 @@ def has_one(*args)
end
- def self.handle_virtuals(attrs)
- @@virtuals.each do |virtual|
- #we first copy the information for the virtual to an instance variable of the same name
- eval("@#{virtual}=attrs['#{virtual}']")
- #and then remove the parameter before it is passed to initialize, so that it is NOT sent to SimpleDB
- eval("attrs.delete('#{virtual}')")
- end
- end
+ end
+
+ @@virtuals=[]
+ def self.handle_virtuals(attrs)
+ @@virtuals.each do |virtual|
+ #we first copy the information for the virtual to an instance variable of the same name
+ eval("@#{virtual}=attrs['#{virtual}']")
+ #and then remove the parameter before it is passed to initialize, so that it is NOT sent to SimpleDB
+ eval("attrs.delete('#{virtual}')")
+ end
end
@@ -301,6 +314,88 @@ def set_attribute_sdb(name, val)
@attributes[sdb_att_name(name)] = val
end
+
+ def get_attribute_sdb(name)
+ name = name.to_sym
+ ret = strip_array(@attributes[sdb_att_name(name)])
+ return ret
+ end
+
+ # Since SimpleDB supports multiple attributes per value, the values are an array.
+ # This method will return the value unwrapped if it's the only, otherwise it will return the array.
+ def get_attribute(name)
+# puts "get_attribute #{name}"
+ # Check if this arg is already converted
+ name_s = name.to_s
+ name = name.to_sym
+ att_meta = get_att_meta(name)
+# puts "att_meta for #{name}: " + att_meta.inspect
+ if att_meta && att_meta.type == :clob
+ ret = @lobs[name]
+# puts 'get_attribute clob ' + ret.inspect
+ if ret
+ if ret.is_a? RemoteNil
+ return nil
+ else
+ return ret
+ end
+ end
+ # get it from s3
+ unless new_record?
+ if self.class.get_sr_config[:single_clob]
+ begin
+ single_clob = s3_bucket(false, :s3_bucket=>:new).get(single_clob_id)
+ single_clob = JSON.parse(single_clob)
+ puts "single_clob=" + single_clob.inspect
+ single_clob.each_pair do |name2,val|
+ @lobs[name2.to_sym] = val
+ end
+ ret = @lobs[name]
+ SimpleRecord.stats.s3_gets += 1
+ rescue Aws::AwsError => ex
+ if ex.include? /NoSuchKey/
+ ret = nil
+ else
+ raise ex
+ end
+ end
+ else
+ begin
+ ret = s3_bucket.get(s3_lob_id(name))
+ # puts 'got from s3 ' + ret.inspect
+ SimpleRecord.stats.s3_gets += 1
+ rescue Aws::AwsError => ex
+ if ex.include? /NoSuchKey/
+ ret = nil
+ else
+ raise ex
+ end
+ end
+ end
+
+ if ret.nil?
+ ret = RemoteNil.new
+ end
+ end
+ @lobs[name] = ret
+ return nil if ret.is_a? RemoteNil
+ return ret