[question] Using inverse ActiveRecord associations to avoid N+1 queries?

[martinliptak]

I've built a smart loader that can load any association using inverse ActiveRecord associations.

module Loaders
  class AssociationLoader < GraphQL::Batch::Loader
    def initialize(
      user,
      model,
      association,
      apply: [],
      aggregate: nil,
      aggregate_default_value: 0
    )
      @user = user
      @model = model
      @association = association
      @apply = apply
      @aggregate = aggregate
      @aggregate_default_value = aggregate_default_value
    end

    def perform(keys)
      # Loading article -> tags, target class will be Tag.
      unless @model.reflect_on_association(@association)
        raise(
          ArgumentError,
          "Association #{@association} doesn't exist on #{@model.name}.",
        )
      end
      target_class = @model.reflect_on_association(@association).klass

      # Why do we need inverse associations?
      #
      # Imagine loading article -> tags. We could do Article.joins(:tags), but this
      #   1) needlessly loads articles, which have already been
      #      loaded by the parent GraphQL query
      #   2) doesn't apply LIMIT to tags
      #
      # Using inverse association, we can do Tag.joins(:article) avoiding both issues.
      #
      unless @model.reflect_on_association(@association).inverse_of
        raise(
          ArgumentError,
          "Association #{@association} on #{@model.name} doesn't have an inverse association.",
        )
      end
      inverse_association = @model.reflect_on_association(@association).inverse_of.name

      # Make sure all loaded records are authorized 
      # (either with simple scopes on models or using a library like Pundit).
      scope = target_class.authorized_scope_for(@user)
      
      # Now doing Tag.joins(:articles).where(articles: { id: @model.id })
      scope = scope.joins(inverse_association)
      scope = scope.where(@model.arel_table[:id].in(keys))

      # Additional named scopes or where conditions.
      @apply.each do |method_and_params|
        scope = scope.send(*method_and_params)
      end

      if @aggregate
        # Group by Article.id
        scope = scope.group(@model.arel_table[:id])

        # For example, `count` aggregates tag count per article.
        scope = scope.send(*@aggregate)

        fulfill_aggregation(keys, scope)
      else
        # Select Article.id as __loader_key
        scope = scope.select(
          @model.arel_table[:id].as("__loader_key"),
          target_class.arel_table[Arel.star]
        )

        # Default limit for security
        scope = scope.limit(GalaxycodrSchema.default_max_page_size)

        if multiple_results_per_key?
          fulfill_multiple_results(keys, scope)
        else
          fulfill_single_result(keys, scope)
        end
      end
    end

    private

    def multiple_results_per_key?
      @association.to_s == @association.to_s.pluralize.to_s
    end

    def fulfill_aggregation(keys, scope)
      # Fulfill results
      scope.each do |key, result|
        fulfill(key, result)
      end

      # Default value is 0 or other value the user provides.
      keys.each do |key|
        next if fulfilled?(key)
        fulfill(key, @aggregate_default_value)
      end
    end

    def fulfill_multiple_results(keys, scope)
      # Group by __loader_key and fulfill keys
      scope
        .group_by { |record| record[:__loader_key] }
        .each { |key, records| fulfill(key, records) }

      # Default value is an empty array
      keys.each { |key| fulfill(key, []) unless fulfilled?(key) }
    end

    def fulfill_single_result(keys, scope)
      scope
        .each { |record| fulfill(record[:__loader_key], record) }

      # Default value is nil
      keys.each { |key| fulfill(key, nil) unless fulfilled?(key) }
    end
  end
end

Then I can do this:

module Types
  class ArticleType < Types::BaseObject
    ...
    field :author, Types::AuthorType, null: false
    def author
      AssociationLoader
        .for(context.user, Article, :author).load(object.id)
    end
    field :all_tags, [Types::TagType], null: false
    def all_tags
      AssociationLoader
        .for(context.user, Article, :tags).load(object.id)
    end
    field :visible_tags, Integer, null: false
    def visible_tags
      AssociationLoader.for(context.user, Article, :tags, {
        apply: [:without_deleted]
      }).load(object.id)
    end
    field :tag_count, Integer, null: false
    def tag_count
      AssociationLoader.for(context.user, Article, :tags, {
        aggregate: :count
      }).load(object.id)
    end
  end
end

All explained in detail here:
https://medium.com/@martinliptak/loading-activerecord-associations-with-graphql-210ec315da0b

Do you think this approach has any downsides I haven't noticed? Do you load more complex associations using something similar or do you implement a loader per every association? Would it be useful to put this into examples/? The current association loader from examples/ can preload associations, but doesn't allow you to specify custom scopes or run an aggregation.

[dylanahsmith]

One of the advantages of AssociationLoader in the examples is that it actually loads the association. Your example seems more like the RecordLoader example, which could more naturally be extended to support of the active record query interface (e.g. joins or named scopes).

The danger of doing too much in the way of filtering in the query is that it could result in multiple queries being performed on the same data in the database, where it could be more appropriate to do the additional filtering client-side to avoid doing an extra query.

We use a separate loader for calculations (i.e. aggregations), which is something we could probably add an example. I don't think it makes sense to combine that with the RecordLoader to make it more complicated by adding the internally divergent code path.

      # Make sure all loaded records are authorized 
      # (either with simple scopes on models or using a library like Pundit).
      scope = target_class.authorized_scope_for(@user)

This makes the example less general and adds an additional dependency for the example that makes it harder to use. I would prefer to leave this sort of customization to the library user.

Do you load more complex associations using something similar or do you implement a loader per every association?

Custom loaders have the advantage of adding a layer of abstraction, which can be useful to decouple the code using the loader from the underlying implementation when it is expected to change. When an abstraction isn't needed, then re-using more general associations makes sense.

[martinliptak]

@dylanahsmith Thank you for your extensive answer!

[fedefa]

Hi @dylanahsmith @martinliptak ! Thank you for your explanation. Can you explain me why loading the association using the preloader has any advantage?

One of the advantages of AssociationLoader in the examples is that it actually loads the association

From my point of view, the only important thing is to calculate the value and populate the <key,value> store. Also, using the preloader, record.association_name will be called only from the perform action.