[Ruby] Loading Large JSON Files with red-arrow Gem

[ashishbista]

I am currently using the red-arrow gem to process JSON files within my Ruby application. While the gem works seamlessly with smaller JSON files, I have encountered an issue when attempting to process a larger JSON file with a size of 2.3MB.

Versions:

red-arrow: 14.0.2
OS:  macOS

The code snippet below illustrates how I am attempting to load the JSON file:

table = Arrow::Table.load(json_file, format: :json)

Unfortunately, executing this code results in the following error:

~/gems/ruby-3.3.0/gems/gobject-introspection-4.2.0/lib/gobject-introspection/loader.rb:705:in `invoke': [json-reader][read]: Invalid: straddling object straddles two block boundaries (try to increase block size?) (Arrow::Error::Invalid)
	from ~/gems/ruby-3.3.0/gems/gobject-introspection-4.2.0/lib/gobject-introspection/loader.rb:705:in `invoke'
	from ~/gems/ruby-3.3.0/gems/gobject-introspection-4.2.0/lib/gobject-introspection/loader.rb:573:in `read'
	from ~/gems/ruby-3.3.0/gems/red-arrow-14.0.2/lib/arrow/table-loader.rb:258:in `block in load_as_json'
	from ~/gems/ruby-3.3.0/gems/red-arrow-14.0.2/lib/arrow/table-loader.rb:155:in `open_input_stream'
	from ~/gems/ruby-3.3.0/gems/red-arrow-14.0.2/lib/arrow/table-loader.rb:256:in `load_as_json'
	from ~/gems/ruby-3.3.0/gems/red-arrow-14.0.2/lib/arrow/table-loader.rb:98:in `load_by_reader'
	from ~/.rvm/gems/ruby-3.3.0/gems/red-arrow-14.0.2/lib/arrow/table-loader.rb:76:in `load_from_file'
	from ~/.rvm/gems/ruby-3.3.0/gems/red-arrow-14.0.2/lib/arrow/table-loader.rb:51:in `block in load'
	from ~/.rvm/gems/ruby-3.3.0/gems/red-arrow-14.0.2/lib/arrow/table-loader.rb:49:in `each'
	from ~/gems/ruby-3.3.0/gems/red-arrow-14.0.2/lib/arrow/table-loader.rb:49:in `load'
	from ~/gems/ruby-3.3.0/gems/red-arrow-14.0.2/lib/arrow/table-loader.rb:26:in `load'
	from ~/gems/ruby-3.3.0/gems/red-arrow-14.0.2/lib/arrow/table.rb:30:in `load'
	....

After researching the issue, I attempted to address it by adjusting the block size using the block_size parameter in the load method:

block_size = 1024 * 1024 * 1024 # Tried different values
table = Arrow::Table.load(json_file, format: :json, block_size: block_size)

However, it seems that the block_size parameter might not be honored by the load method, as suggested by my analysis of the source code.

Therefore, I would like to seek your guidance on whether there is an existing method to adjust the block size when reading files with the red-arrow gem. If this feature is not currently supported, I kindly request consideration for the addition of a feature that either dynamically handles the block size internally or exposes an API to adjust the block size.

Let me know if you need anything else from my side.

Component(s)

Ruby