Skip to content
master
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
lib
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

YajlRest

Sometimes you may end up having REST API endpoint returning huge JSON array of some object you need to parse and load into your database. However, if you are retrieving a really large amount of data, you can't just load it into memory and parse. And you shouldn't!

YajlRest is a streaming JSON array parser providing you with enumerator to lazily traverse through each object allocationg only memory required for the single object to place.

You can use it to enumerate JSON array acquired from HTTP URL or from local file. Local file should be plain JSON. HTTP endpoint may return plain or gzipped content.

You only need enough disk space to download the whole response into temporary file.

Requirements

You will need yajl2 library. See the link for details.

Installation

Add this line to your application's Gemfile:

gem 'yajl-rest'

And then execute:

$ bundle

Or install it yourself as:

$ gem install yajl-rest

Usage

Read data from REST API endpoint:

e = YajlRest::JsonEnumerator.new 'http://example.com/objects',
                                 accept: :json,
                                 authorization: 'Bearer token="asdf"'
e.each { |o| puts o['id'] }

Read data from local file:

e = YajlRest::JsonEnumerator.new '/tmp/rest-client.20180629-25287-1fgmf91'
e.each { |o| puts o['id'] }

Credits

Inspired by and based on yajl-ffi

About

Streaming JSON arrays parser for Ruby

Resources

License

Releases

No releases published

Packages

No packages published

Languages

You can’t perform that action at this time.