PostgreSQL Cursor is an extension to the ActiveRecord PostgreSQLAdapter for very large result sets. It provides a cursor open/fetch/close interface to access data without loading all rows into memory, and instead loads the result rows in “chunks” (default of 10_000 rows), buffers them, and returns the rows one at a time.
For web pages, an application would not want to process a large amount of data, usually employing a Pagination scheme to present it to the users. Background processes sometimes need to generate a large
Previous solutions employ pagination to fetch each block, then re-running the query for the next “page”. This plugin avoids re-executing the query by using the PostgreSQL cursors.
Like the #find_by_sql method, #find_cursor returns each row as a hash instead of an instantiated model class. The rationale for this is performance, though an option to return instances is available. Julian's benchmarks showed returning instances was a factor of 4 slower than return the hash.
A Rails/ActiveRecord plugin for the PostgreSQL database adapter that will add cursors to a find_cursor() method to process very large result sets.
the find_cursor method uses cursors to pull in one data block (of x records) at a time, and return each record as a Hash to a procedural block. When each data block is exhausted, it will fetch the next one.
find_cursor_by_sql takes a custom SQL statement and returns each row.
Account.find_cursor(:account_cursor, true, :all, :conditions=>["status = ?", 'good']) do |row| puts row.to_json end Account.find_cursor_by_sql(:account_cursor, "select ...", *args) do |row| puts row.to_json end
Allen Fair, email@example.com, github.com/afair
Thank you to:
Copyright © 2010 Allen M. Fair, released under the MIT license