Skip to content

Commit

Permalink
Allow the :eager_loader association proc to accept a single hash inst…
Browse files Browse the repository at this point in the history
…ead of 3 arguments

The previously supported API of using 3 separate arguments is still
supported.  However, the new API is more flexible, and is required
to get access to the dataset that is doing the eager loading.  That
will be used in the future to make sure the sharding plugin works
for eager loading.

With the new API, the hash passed to the eager_loader proc is
populated with the keys :key_hash, :rows, :associations, and
:self, with the first three keys corresponding to the old
argument order, and the new :self key being the dataset doing
the eager loading.

The Model.eager_loading_dataset method now also accepts an
options hash as an optional fifth argument, which should be
backwards compatible.
  • Loading branch information
jeremyevans committed May 25, 2010
1 parent 3670ddc commit 14e9bcd
Show file tree
Hide file tree
Showing 9 changed files with 119 additions and 96 deletions.
2 changes: 2 additions & 0 deletions CHANGELOG
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
=== HEAD

* Allow the :eager_loader association proc to accept a single hash instead of 3 arguments (jeremyevans)

* Add a Dataset#order_append alias for order_more, for consistency with order_prepend (jeremyevans)

* Add a Dataset#order_prepend method that adds to the end of an existing order (jeremyevans)
Expand Down
2 changes: 1 addition & 1 deletion doc/active_record.rdoc
Original file line number Diff line number Diff line change
Expand Up @@ -337,7 +337,7 @@ With either way of eager loading, you must call +all+ to retrieve all records at

Like ActiveRecord, Sequel supports cascading of eager loading for both methods of eager loading.

Unlike ActiveRecord, Sequel allows you to eager load custom associations using the <tt>:eager_loader</tt> and <tt>:eager_grapher</tt> association options.
Unlike ActiveRecord, Sequel allows you to eager load custom associations using the <tt>:eager_loader</tt> and <tt>:eager_grapher</tt> association options. See the {Advanced Associations guide}[link:files/doc/advanced_associations_rdoc.html] for more details.

Table aliasing when eager loading via +eager_graph+ is different in Sequel than ActiveRecord. Sequel will always attempt to use the association name, not the table name, for any associations. If the association name has already been used, Sequel will append _N to it, where N starts at 0 and increases by 1. For example, for a self referential association:

Expand Down
77 changes: 46 additions & 31 deletions doc/advanced_associations.rdoc
Original file line number Diff line number Diff line change
Expand Up @@ -77,9 +77,21 @@ option. Though it can often be verbose (compared to other things in Sequel),
it allows you complete control over how to eagerly load associations for a
group of objects.

:eager_loader should be a proc that takes 3 arguments, a key_hash,
an array of records, and a hash of dependent associations. Since you
are given all of the records, you can do things like filter on
:eager_loader should be a proc that takes 1 or 3 arguments. If the proc
takes one argument, it will be given a hash with the following keys:

:key_hash :: A key_hash, described below
:rows :: An array of model objects
:associations :: A hash of dependent associations to eagerly load
:self :: The dataset that is doing the eager loading

If the proc takes three arguments, it gets passed the :key_hash, :rows,
and :associations values. The only way to get the :self value is to
accept one argument. The 3 argument procs are allowed for backwards
compatibility, and it is recommended to use the 1 argument proc format
for new code.

Since you are given all of the records, you can do things like filter on
associations that are specified by multiple keys, or do multiple
queries depending on the content of the records (which would be
necessary for polymorphic associations). Inside the :eager_loader
Expand Down Expand Up @@ -277,9 +289,9 @@ Sequel::Model:
klass = attachable_type.constantize
klass.filter(klass.primary_key=>attachable_id)
end), \
:eager_loader=>(proc do |key_hash, assets, associations|
:eager_loader=>(proc do |eo|
id_map = {}
assets.each do |asset|
eo[:rows].each do |asset|
asset.associations[:attachable] = nil
((id_map[asset.attachable_type] ||= {})[asset.attachable_id] ||= []) << asset
end
Expand Down Expand Up @@ -370,9 +382,9 @@ design, but sometimes you have to play with the cards you are dealt).
class Client < Sequel::Model
one_to_many :invoices, :reciprocal=>:client, \
:dataset=>proc{Invoice.filter(:client_name=>name)}, \
:eager_loader=>(proc do |key_hash, clients, associations|
:eager_loader=>(proc do |eo|
id_map = {}
clients.each do |client|
eo[:rows].each do |client|
id_map[client.name] = client
client.associations[:invoices] = []
end
Expand Down Expand Up @@ -400,9 +412,9 @@ design, but sometimes you have to play with the cards you are dealt).
class Invoice < Sequel::Model
many_to_one :client, :key=>:client_name, \
:dataset=>proc{Client.filter(:name=>client_name)}, \
:eager_loader=>(proc do |key_hash, invoices, associations|
id_map = key_hash[:client_name]
invoices.each{|inv| inv.associations[:client] = nil}
:eager_loader=>(proc do |eo|
id_map = eo[:key_hash][:client_name]
eo[:rows].each{|inv| inv.associations[:client] = nil}
Client.filter(:name=>id_map.keys).all do |client|
id_map[client.name].each{|inv| inv.associations[:client] = client}
end
Expand Down Expand Up @@ -439,9 +451,9 @@ Here's the old way to do it via custom associations:
:dataset=>(proc do
FavoriteTrack.filter(:disc_number=>disc_number, :number=>number, :album_id=>album_id)
end), \
:eager_loader=>(proc do |key_hash, tracks, associations|
:eager_loader=>(proc do |eo|
id_map = {}
tracks.each do |t|
eo[:rows].each do |t|
t.associations[:favorite_track] = nil
id_map[[t[:album_id], t[:disc_number], t[:number]]] = t
end
Expand All @@ -458,9 +470,9 @@ Here's the old way to do it via custom associations:
:dataset=>(proc do
Track.filter(:disc_number=>disc_number, :number=>number, :album_id=>album_id)
end), \
:eager_loader=>(proc do |key_hash, ftracks, associations|
:eager_loader=>(proc do |eo|
id_map = {}
ftracks.each{|ft| id_map[[ft[:album_id], ft[:disc_number], ft[:number]]] = ft}
eo[:rows].each{|ft| id_map[[ft[:album_id], ft[:disc_number], ft[:number]]] = ft}
Track.filter([:album_id, :disc_number, :number]=>id_map.keys).all do |t|
id_map[[t[:album_id], t[:disc_number], t[:number]]].associations[:track] = t
end
Expand Down Expand Up @@ -490,10 +502,10 @@ without knowing the depth of the tree?

class Node < Sequel::Model
many_to_one :ancestors, :class=>self,
:eager_loader=>(proc do |key_hash, nodes, associations|
:eager_loader=>(proc do |eo|
# Handle cases where the root node has the same parent_id as primary_key
# and also when it is NULL
non_root_nodes = nodes.reject do |n|
non_root_nodes = eo[:rows].reject do |n|
if [nil, n.pk].include?(n.parent_id)
# Make sure root nodes have their parent association set to nil
n.associations[:parent] = nil
Expand All @@ -514,9 +526,9 @@ without knowing the depth of the tree?
end
end
end)
many_to_one :descendants, :eager_loader=>(proc do |key_hash, nodes, associations|
many_to_one :descendants, :eager_loader=>(proc do |eo|
id_map = {}
nodes.each do |n|
eo[:rows].each do |n|
# Initialize an empty array of child associations for each parent node
n.associations[:children] = []
# Populate identity map of nodes
Expand Down Expand Up @@ -549,9 +561,9 @@ supports recursive common table expressions):
Node.join(:t, :id=>:parent_id).
select(:nodes.*))
end),
:eager_loader=>(proc do |key_hash, nodes, associations|
id_map = key_hash[:id]
nodes.each{|n| n.associations[:descendants] = []}
:eager_loader=>(proc do |eo|
id_map = eo[:key_hash][:id]
eo[:rows].each{|n| n.associations[:descendants] = []}
Node.from(:t).
with_recursive(:t, Node.filter(:parent_id=>id_map.keys).
select(:parent_id___root, :id, :parent_id),
Expand All @@ -565,8 +577,11 @@ supports recursive common table expressions):
end)
end

You could modify the code to also store direct children relationships at the same time,
for functionality similar to the non-common table expression case.
Sequel ships with an +rcte_tree+ plugin that allows simple creation
of ancestors and descendants relationships that use recursive common
table expressions:

Node.plugin :rcte_tree

=== Joining multiple keys to a single key, through a third table

Expand All @@ -584,10 +599,10 @@ name, with no duplicates?
class Artist < Sequel::Model
one_to_many :songs, :order=>:songs__name, \
:dataset=>proc{Song.select(:songs.*).join(Lyric, :id=>:lyric_id, id=>[:composer_id, :arranger_id, :vocalist_id, :lyricist_id])}, \
:eager_loader=>(proc do |key_hash, records, associations|
h = key_hash[:id]
:eager_loader=>(proc do |eo|
h = eo[:key_hash][:id]
ids = h.keys
records.each{|r| r.associations[:songs] = []}
eo[:rows].each{|r| r.associations[:songs] = []}
Song.select(:songs.*, :lyrics__composer_id, :lyrics__arranger_id, :lyrics__vocalist_id, :lyrics__lyricist_id)\
.join(Lyric, :id=>:lyric_id){{:composer_id=>ids, :arranger_id=>ids, :vocalist_id=>ids, :lyricist_id=>ids}.sql_or}\
.order(:songs__name).all do |song|
Expand All @@ -596,7 +611,7 @@ name, with no duplicates?
recs.each{|r| r.associations[:songs] << song} if recs
end
end
records.each{|r| r.associations[:songs].uniq!}
eo[:rows].each{|r| r.associations[:songs].uniq!}
end)
end

Expand All @@ -614,13 +629,13 @@ associated tickets.
one_to_many :tickets
many_to_one :ticket_hours, :read_only=>true, :key=>:id,
:dataset=>proc{Ticket.filter(:project_id=>id).select{sum(hours).as(hours)}},
:eager_loader=>(proc do |kh, projects, a|
projects.each{|p| p.associations[:ticket_hours] = nil}
Ticket.filter(:project_id=>kh[:id].keys).
:eager_loader=>(proc do |eo|
eo[:rows].each{|p| p.associations[:ticket_hours] = nil}
Ticket.filter(:project_id=>eo[:key_hash][:id].keys).
group(:project_id).
select{[project_id, sum(hours).as(hours)]}.
all do |t|
p = kh[:id][t.values.delete(:project_id)].first
p = eo[:key_hash][:id][t.values.delete(:project_id)].first
p.associations[:ticket_hours] = t
end
end)
Expand Down
11 changes: 2 additions & 9 deletions doc/association_basics.rdoc
Original file line number Diff line number Diff line change
Expand Up @@ -1093,16 +1093,9 @@ to eagerly load:

==== :eager_loader

A custom loader to use when eagerly load associated objects via eager. If
specified, should be a proc that takes three arguments: a key hash (used
solely to enhance performance), an array of current model instances, and a
hash of dependent associations to eagerly load. The proc is responsible for
querying the database to retrieve all associated records for any of the model
instances (the second argument), and modifying the associations cache for each
record to correctly set the associated records for that record.

A custom loader to use when eagerly load associated objects via eager.
For many details and examples of custom eager loaders, please see the
{Advanced Associations page}[link:files/doc/advanced_associations_rdoc.html].
{Advanced Associations guide}[link:files/doc/advanced_associations_rdoc.html].

==== :eager_loader_key

Expand Down
48 changes: 28 additions & 20 deletions lib/sequel/model/associations.rb
Original file line number Diff line number Diff line change
Expand Up @@ -445,8 +445,8 @@ module AssociationDatasetMethods
# => {:type => :many_to_one, :name => :portfolio, :class_name => "Portfolio"}
#
# For a more in depth general overview, as well as a reference guide,
# see the {Association Basics page}[link:files/doc/association_basics_rdoc.html].
# For examples of advanced usage, see the {Advanced Associations page}[link:files/doc/advanced_associations_rdoc.html].
# see the {Association Basics guide}[link:files/doc/association_basics_rdoc.html].
# For examples of advanced usage, see the {Advanced Associations guide}[link:files/doc/advanced_associations_rdoc.html].
module ClassMethods
# All association reflections defined for this model (default: none).
attr_reader :association_reflections
Expand Down Expand Up @@ -525,9 +525,11 @@ def all_association_reflections
# Takes three arguments, a dataset, an alias to use for the table to graph for this association,
# and the alias that was used for the current table (since you can cascade associations),
# Should return a copy of the dataset with the association graphed into it.
# - :eager_loader - A proc to use to implement eager loading, overriding the default. Takes three arguments,
# a key hash (used solely to enhance performance), an array of records,
# and a hash of dependent associations. The associated records should
# - :eager_loader - A proc to use to implement eager loading, overriding the default. Takes one or three arguments.
# If three arguments, the first should be a key hash (used solely to enhance performance), the second an array of records,
# and the third a hash of dependent associations. If one argument, is passed a hash with keys :key_hash,
# :rows, and :associations, corresponding to the three arguments, and an additional key :self, which is
# the dataset doing the eager loading. In the proc, the associated records should
# be queried from the database and the associations cache for each
# record should be populated for this to work correctly.
# - :eager_loader_key - A symbol for the key column to use to populate the key hash
Expand Down Expand Up @@ -614,6 +616,7 @@ def associate(type, name, opts = {}, &block)
raise(Error, 'one_to_many association type with :one_to_one option removed, used one_to_one association type') if opts[:one_to_one] && type == :one_to_many
raise(Error, 'invalid association type') unless assoc_class = ASSOCIATION_TYPES[type]
raise(Error, 'Model.associate name argument must be a symbol') unless Symbol === name
raise(Error, ':eager_loader option must have an arity of 1 or 3') if opts[:eager_loader] && ![1, 3].include?(opts[:eager_loader].arity)

# merge early so we don't modify opts
orig_opts = opts.dup
Expand Down Expand Up @@ -653,7 +656,7 @@ def associations
end

# Modify and return eager loading dataset based on association options. Options:
def eager_loading_dataset(opts, ds, select, associations)
def eager_loading_dataset(opts, ds, select, associations, eager_options={})
ds = ds.select(*select) if select
if c = opts[:conditions]
ds = (c.is_a?(Array) && !Sequel.condition_specifier?(c)) ? ds.filter(*c) : ds.filter(c)
Expand Down Expand Up @@ -761,12 +764,12 @@ def def_many_to_many(opts)
opts[:dataset] ||= proc{opts.associated_class.inner_join(join_table, rcks.zip(opts.right_primary_keys) + lcks.zip(lcpks.map{|k| send(k)}))}
database = db

opts[:eager_loader] ||= proc do |key_hash, records, associations|
h = key_hash[left_pk]
records.each{|object| object.associations[name] = []}
opts[:eager_loader] ||= proc do |eo|
h = eo[:key_hash][left_pk]
eo[:rows].each{|object| object.associations[name] = []}
r = uses_rcks ? rcks.zip(opts.right_primary_keys) : [[right, opts.right_primary_key]]
l = uses_lcks ? [[lcks.map{|k| SQL::QualifiedIdentifier.new(join_table, k)}, SQL::SQLArray.new(h.keys)]] : [[left, h.keys]]
model.eager_loading_dataset(opts, opts.associated_class.inner_join(join_table, r + l), Array(opts.select), associations).all do |assoc_record|
model.eager_loading_dataset(opts, opts.associated_class.inner_join(join_table, r + l), Array(opts.select), eo[:associations], eo).all do |assoc_record|
hash_key = if uses_lcks
left_key_alias.map{|k| assoc_record.values.delete(k)}
else
Expand Down Expand Up @@ -827,16 +830,16 @@ def def_many_to_one(opts)
klass = opts.associated_class
klass.filter(opts.primary_keys.map{|k| SQL::QualifiedIdentifier.new(klass.table_name, k)}.zip(cks.map{|k| send(k)}))
end
opts[:eager_loader] ||= proc do |key_hash, records, associations|
h = key_hash[key]
opts[:eager_loader] ||= proc do |eo|
h = eo[:key_hash][key]
keys = h.keys
# Default the cached association to nil, so any object that doesn't have it
# populated will have cached the negative lookup.
records.each{|object| object.associations[name] = nil}
eo[:rows].each{|object| object.associations[name] = nil}
# Skip eager loading if no objects have a foreign key for this association
unless keys.empty?
klass = opts.associated_class
model.eager_loading_dataset(opts, klass.filter(uses_cks ? {opts.primary_keys.map{|k| SQL::QualifiedIdentifier.new(klass.table_name, k)}=>SQL::SQLArray.new(keys)} : {SQL::QualifiedIdentifier.new(klass.table_name, opts.primary_key)=>keys}), opts.select, associations).all do |assoc_record|
model.eager_loading_dataset(opts, klass.filter(uses_cks ? {opts.primary_keys.map{|k| SQL::QualifiedIdentifier.new(klass.table_name, k)}=>SQL::SQLArray.new(keys)} : {SQL::QualifiedIdentifier.new(klass.table_name, opts.primary_key)=>keys}), opts.select, eo[:associations], eo).all do |assoc_record|
hash_key = uses_cks ? opts.primary_keys.map{|k| assoc_record.send(k)} : assoc_record.send(opts.primary_key)
next unless objects = h[hash_key]
objects.each{|object| object.associations[name] = assoc_record}
Expand Down Expand Up @@ -877,16 +880,16 @@ def def_one_to_many(opts)
klass = opts.associated_class
klass.filter(cks.map{|k| SQL::QualifiedIdentifier.new(klass.table_name, k)}.zip(cpks.map{|k| send(k)}))
end
opts[:eager_loader] ||= proc do |key_hash, records, associations|
h = key_hash[primary_key]
opts[:eager_loader] ||= proc do |eo|
h = eo[:key_hash][primary_key]
if one_to_one
records.each{|object| object.associations[name] = nil}
eo[:rows].each{|object| object.associations[name] = nil}
else
records.each{|object| object.associations[name] = []}
eo[:rows].each{|object| object.associations[name] = []}
end
reciprocal = opts.reciprocal
klass = opts.associated_class
model.eager_loading_dataset(opts, klass.filter(uses_cks ? {cks.map{|k| SQL::QualifiedIdentifier.new(klass.table_name, k)}=>SQL::SQLArray.new(h.keys)} : {SQL::QualifiedIdentifier.new(klass.table_name, key)=>h.keys}), opts.select, associations).all do |assoc_record|
model.eager_loading_dataset(opts, klass.filter(uses_cks ? {cks.map{|k| SQL::QualifiedIdentifier.new(klass.table_name, k)}=>SQL::SQLArray.new(h.keys)} : {SQL::QualifiedIdentifier.new(klass.table_name, key)=>h.keys}), opts.select, eo[:associations], eo).all do |assoc_record|
hash_key = uses_cks ? cks.map{|k| assoc_record.send(k)} : assoc_record.send(key)
next unless objects = h[hash_key]
if one_to_one
Expand Down Expand Up @@ -1526,7 +1529,12 @@ def eager_load(a, eager_assoc=@opts[:eager])
end

reflections.each do |r|
r[:eager_loader].call(key_hash, a, eager_assoc[r[:name]])
loader = r[:eager_loader]
if loader.arity == 1
loader.call(:key_hash=>key_hash, :rows=>a, :associations=>eager_assoc[r[:name]], :self=>self)
else
loader.call(key_hash, a, eager_assoc[r[:name]])
end
a.each{|object| object.send(:run_association_callbacks, r, :after_load, object.associations[r[:name]])} unless r[:after_load].empty?
end
end
Expand Down
Loading

0 comments on commit 14e9bcd

Please sign in to comment.