diff --git a/.travis.yml b/.travis.yml index 9888b20..139d03d 100644 --- a/.travis.yml +++ b/.travis.yml @@ -2,4 +2,6 @@ sudo: false language: ruby rvm: - 2.3.4 +env: + - CI=true before_install: gem install bundler -v 1.15.3 diff --git a/CHANGELOG.md b/CHANGELOG.md index 41c1f1c..237c29f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,7 +16,7 @@ that you can set version constraints properly. * `Added`: `cache: false` option. * `Added`: `BatchLoader::Middleware`. -* `Added`: More docs and tests. +* `Added`: more docs and tests. #### [v0.1.0](https://github.com/exAspArk/batch-loader/compare/ed32edb...v0.1.0) – 2017-07-31 diff --git a/Gemfile b/Gemfile index d5d61f6..443e487 100644 --- a/Gemfile +++ b/Gemfile @@ -2,7 +2,7 @@ source "https://rubygems.org" git_source(:github) {|repo_name| "https://github.com/#{repo_name}" } -gem "pry" +gem 'coveralls', require: false # Specify your gem's dependencies in batch-loader.gemspec gemspec diff --git a/README.md b/README.md index f92a9f3..9a7ab9a 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,10 @@ # BatchLoader [![Build Status](https://travis-ci.org/exAspArk/batch-loader.svg?branch=master)](https://travis-ci.org/exAspArk/batch-loader) +[![Coverage Status](https://coveralls.io/repos/github/exAspArk/batch-loader/badge.svg)](https://coveralls.io/github/exAspArk/batch-loader) +[![Code Climate](https://img.shields.io/codeclimate/github/exAspArk/batch-loader.svg)](https://codeclimate.com/github/exAspArk/batch-loader) +[![Downloads](https://img.shields.io/gem/dt/batch-loader.svg)](https://rubygems.org/gems/batch-loader) +[![Latest Version](https://img.shields.io/gem/v/batch-loader.svg)](https://rubygems.org/gems/batch-loader) Simple tool to avoid N+1 DB queries, HTTP requests, etc. @@ -16,7 +20,6 @@ Simple tool to avoid N+1 DB queries, HTTP requests, etc. * [Caching](#caching) * [Installation](#installation) * [Implementation details](#implementation-details) -* [Testing](#testing) * [Development](#development) * [Contributing](#contributing) * [License](#license) @@ -96,7 +99,7 @@ users = load_posts(post.user) # ↓ U ↓ users.map { |u| user.name } # Users ``` -But the problem here is that `load_posts` now depends on the child association. Plus it'll preload the association every time, even if it's not necessary. Can we do better? Sure! +But the problem here is that `load_posts` now depends on the child association and knows that it has to preload the data for `load_users`. And it'll do it every time, even if it's not necessary. Can we do better? Sure! ### Basic example @@ -130,7 +133,7 @@ As we can see, batching is isolated and described right in a place where it's ne ### How it works -In general, `BatchLoader` returns an object which in other similar implementations is call Promise. Each Promise knows which data it needs to load and how to batch the query. When all the Promises are collected it's possible to resolve them once without N+1 queries. +In general, `BatchLoader` returns a lazy object. In other programming languages it usually called Promise, but I personally prefer to call it lazy, since Ruby already uses the name in standard library :) Each lazy object knows which data it needs to load and how to batch the query. When all the lazy objects are collected it's possible to resolve them once without N+1 queries. So, when we call `BatchLoader.for` we pass an item (`user_id`) which should be batched. For the `batch` method, we pass a block which uses all the collected items (`user_ids`): @@ -172,7 +175,7 @@ end class PostsController < ApplicationController def index posts = Post.limit(10) - serialized_posts = posts.map { |post| {id: post.id, rating: post.rating} } + serialized_posts = posts.map { |post| {id: post.id, rating: post.rating} } # N+1 HTTP requests for each post.rating render json: serialized_posts end @@ -182,7 +185,6 @@ end As we can see, the code above will make N+1 HTTP requests, one for each post. Let's batch the requests with a gem called [parallel](https://github.com/grosser/parallel): ```ruby -# app/models/post.rb class Post < ApplicationRecord def rating_lazy BatchLoader.for(post).batch do |posts, batch_loader| @@ -190,9 +192,7 @@ class Post < ApplicationRecord end end - def rating - HttpClient.request(:get, "https://example.com/ratings/#{id}") - end + # ... end ``` @@ -201,7 +201,6 @@ end Now we can resolve all `BatchLoader` objects in the controller: ```ruby -# app/controllers/posts_controller.rb class PostsController < ApplicationController def index posts = Post.limit(10) @@ -211,10 +210,9 @@ class PostsController < ApplicationController end ``` -`BatchLoader` caches the resolved values. To ensure that the cache is purged for each request in the app add the following middleware: +`BatchLoader` caches the resolved values. To ensure that the cache is purged between requests in the app add the following middleware to your `config/application.rb`: ```ruby -# config/application.rb config.middleware.use BatchLoader::Middleware ``` @@ -222,11 +220,120 @@ See the [Caching](#caching) section for more information. ### GraphQL example -TODO +With GraphQL using batching is particularly useful. You can't use usual techniques such as preloading associations in advance to avoid N+1 queries. +Since you don't know which fields user is going to ask in a query. + +Let's take a look at the simple [graphql-ruby](https://github.com/rmosolgo/graphql-ruby) schema example: + +```ruby +Schema = GraphQL::Schema.define do + query QueryType +end + +QueryType = GraphQL::ObjectType.define do + name "Query" + field :posts, !types[PostType], resolve: ->(obj, args, ctx) { Post.all } +end + +PostType = GraphQL::ObjectType.define do + name "Post" + field :user, !UserType, resolve: ->(post, args, ctx) { post.user } # N+1 queries +end + +UserType = GraphQL::ObjectType.define do + name "User" + field :name, !types.String +end +``` + +If we want to execute a simple query like: + +```ruby +query = " +{ + posts { + user { + name + } + } +} +" +Schema.execute(query, variables: {}, context: {}) +``` + +We will get N+1 queries for each `post.user`. To avoid this problem, all we have to do is to change the resolver to use `BatchLoader`: + +```ruby +PostType = GraphQL::ObjectType.define do + name "Post" + field :user, !UserType, resolve: ->(post, args, ctx) do + BatchLoader.for(post.user_id).batch do |user_ids, batch_loader| + User.where(id: user_ids).each { |user| batch_loader.load(user.id, user) } + end + end +end +``` + +And setup GraphQL with built-in `lazy_resolve` method: + +```ruby +Schema = GraphQL::Schema.define do + query QueryType + lazy_resolve BatchLoader, :sync +end +``` ### Caching -TODO +By default `BatchLoader` caches the resolved values. You can test it by running something like: + +```ruby +def user_lazy(id) + BatchLoader.for(id).batch do |ids, batch_loader| + User.where(id: ids).each { |user| batch_loader.load(user.id, user) } + end +end + +user_lazy(1) # no request +# => <#BatchLoader> + +user_lazy(1).sync # SELECT * FROM users WHERE id IN (1) +# => <#User> + +user_lazy(1).sync # no request +# => <#User> +``` + +To drop the cache manually you can run: + +```ruby +user_lazy(1).sync # SELECT * FROM users WHERE id IN (1) +user_lazy(1).sync # no request + +BatchLoader::Executor.clear_current + +user_lazy(1).sync # SELECT * FROM users WHERE id IN (1) +``` + +Usually, it's just enough to clear the cache between HTTP requests in the app. To do so, simply add the middleware: + +```ruby +# calls "BatchLoader::Executor.clear_current" after each request +use BatchLoader::Middleware +``` + +In some rare cases it's useful to disable caching for `BatchLoader`. For example, in tests or after data mutations: + +```ruby +def user_lazy(id) + BatchLoader.for(id).batch(cache: false) do |ids, batch_loader| + # ... + end +end + +user_lazy(1).sync # SELECT * FROM users WHERE id IN (1) +user_lazy(1).sync # SELECT * FROM users WHERE id IN (1) +``` ## Installation @@ -246,11 +353,7 @@ Or install it yourself as: ## Implementation details -TODO - -## Testing - -TODO +Coming soon ## Development diff --git a/batch-loader.gemspec b/batch-loader.gemspec index d13da2c..94dc8d5 100644 --- a/batch-loader.gemspec +++ b/batch-loader.gemspec @@ -26,4 +26,6 @@ Gem::Specification.new do |spec| spec.add_development_dependency "bundler", "~> 1.15" spec.add_development_dependency "rake", "~> 10.0" spec.add_development_dependency "rspec", "~> 3.0" + spec.add_development_dependency "graphql", "~> 1.6" + spec.add_development_dependency "pry-byebug", "~> 3.4" end diff --git a/lib/batch_loader.rb b/lib/batch_loader.rb index 053ed96..03173f0 100644 --- a/lib/batch_loader.rb +++ b/lib/batch_loader.rb @@ -6,22 +6,20 @@ class BatchLoader NoBatchError = Class.new(StandardError) BatchAlreadyExistsError = Class.new(StandardError) - class << self - def for(item) - new(item: item) - end + def self.for(item) + new(item: item) + end - def sync!(value) - case value - when Array - value.map! { |v| sync!(v) } - when Hash - value.each { |k, v| value[k] = sync!(v) } - when BatchLoader - sync!(value.sync) - else - value - end + def self.sync!(value) + case value + when Array + value.map! { |v| sync!(v) } + when Hash + value.each { |k, v| value[k] = sync!(v) } + when BatchLoader + sync!(value.sync) + else + value end end diff --git a/lib/batch_loader/executor.rb b/lib/batch_loader/executor.rb index 4cce088..5db0cda 100644 --- a/lib/batch_loader/executor.rb +++ b/lib/batch_loader/executor.rb @@ -6,7 +6,11 @@ def self.ensure_current Thread.current[NAMESPACE] ||= new end - def self.delete_current + def self.current + Thread.current[NAMESPACE] + end + + def self.clear_current Thread.current[NAMESPACE] = nil end diff --git a/lib/batch_loader/middleware.rb b/lib/batch_loader/middleware.rb index 2dafba3..e0a6bac 100644 --- a/lib/batch_loader/middleware.rb +++ b/lib/batch_loader/middleware.rb @@ -6,10 +6,9 @@ def initialize(app) def call(env) begin - BatchLoader::Executor.ensure_current @app.call(env) ensure - BatchLoader::Executor.delete_current + BatchLoader::Executor.clear_current end end end diff --git a/spec/batch_loader/middleware_spec.rb b/spec/batch_loader/middleware_spec.rb new file mode 100644 index 0000000..e798eda --- /dev/null +++ b/spec/batch_loader/middleware_spec.rb @@ -0,0 +1,24 @@ +require "spec_helper" + +RSpec.describe BatchLoader::Middleware do + describe '#call' do + it 'returns the result from the app' do + app = ->(_env) { 1 } + middleware = BatchLoader::Middleware.new(app) + + expect(middleware.call(nil)).to eq(1) + end + + it 'clears the Executor' do + app = ->(_) { nil } + middleware = BatchLoader::Middleware.new(app) + BatchLoader::Executor.ensure_current + + expect { + middleware.call(nil) + }.to change { + BatchLoader::Executor.current + }.to(nil) + end + end +end diff --git a/spec/batch_loader_spec.rb b/spec/batch_loader_spec.rb index 6238d97..00de416 100644 --- a/spec/batch_loader_spec.rb +++ b/spec/batch_loader_spec.rb @@ -1,10 +1,8 @@ require "spec_helper" RSpec.describe BatchLoader do - after { BatchLoader::Executor.delete_current } - describe '.sync!' do - it "does something useful" do + it "syncs all BatchLoaders" do user1 = User.save(id: 1) post1 = Post.new(user_id: user1.id) user2 = User.save(id: 2) @@ -26,7 +24,7 @@ }.to raise_error(BatchLoader::NoBatchError, "Please provide a batch block first") end - it 'caches the result between different BatchLoader instances' do + it 'caches the result even between different BatchLoader instances' do user = User.save(id: 1) post = Post.new(user_id: user.id) diff --git a/spec/fixtures/graphql_schema.rb b/spec/fixtures/graphql_schema.rb new file mode 100644 index 0000000..e5d50df --- /dev/null +++ b/spec/fixtures/graphql_schema.rb @@ -0,0 +1,19 @@ +UserType = GraphQL::ObjectType.define do + name "User" + field :id, !types.ID +end + +PostType = GraphQL::ObjectType.define do + name "Post" + field :user, !UserType, resolve: ->(post, args, ctx) { post.user_lazy } +end + +QueryType = GraphQL::ObjectType.define do + name "Query" + field :posts, !types[PostType], resolve: ->(obj, args, ctx) { Post.all } +end + +GraphqlSchema = GraphQL::Schema.define do + query QueryType + lazy_resolve BatchLoader, :sync +end diff --git a/spec/fixtures/models.rb b/spec/fixtures/models.rb index ddb74bb..4a69831 100644 --- a/spec/fixtures/models.rb +++ b/spec/fixtures/models.rb @@ -1,6 +1,15 @@ class Post attr_accessor :user_id, :user_lazy + def self.save(user_id:) + @@posts ||= [] + @@posts << new(user_id: user_id) + end + + def self.all + @@posts + end + def initialize(user_id:) self.user_id = user_id end diff --git a/spec/graphql_spec.rb b/spec/graphql_spec.rb new file mode 100644 index 0000000..ef92b82 --- /dev/null +++ b/spec/graphql_spec.rb @@ -0,0 +1,28 @@ +require "spec_helper" + +RSpec.describe 'GraphQL integration' do + it 'resolves BatchLoader fields lazily' do + user1 = User.save(id: "1") + user2 = User.save(id: "2") + Post.save(user_id: user1.id) + Post.save(user_id: user2.id) + query = <<~QUERY + { + posts { + user { id } + } + } + QUERY + + expect(User).to receive(:where).with(id: ["1", "2"]).once.and_call_original + + result = GraphqlSchema.execute(query) + + expect(result['data']).to eq({ + 'posts' => [ + {'user' => {'id' => "1"}}, + {'user' => {'id' => "2"}} + ] + }) + end +end diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 8ca4ec2..6142674 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -1,6 +1,16 @@ require "bundler/setup" + +require "graphql" + require "batch_loader" + require "fixtures/models" +require "fixtures/graphql_schema" + +if ENV['CI'] + require 'coveralls' + Coveralls.wear! +end RSpec.configure do |config| # Enable flags like --only-failures and --next-failure @@ -12,4 +22,8 @@ config.expect_with :rspec do |c| c.syntax = :expect end + + config.after do + BatchLoader::Executor.clear_current + end end