Ruby Gem for reading and writing Apache Optimized Row Columnar (ORC) files. This gem can also be paired using the factory_girl gem.
Must use jruby.
Add this line to your application’s Gemfile:
gem 'orc_file'
And then execute:
$ bundle install
Or install it yourself as:
$ gem install orc_file
To write a file, you will need to initialize the OrcFileWriter class. This object needs a table schema, your dataset, the path to store the file, and an optional configuration hash.
OrcFileWriter.new(table_schema, data_set, path, *options={})
The table_schema must be a hash containing the column name and datatype as the key-value pair.
Valid datatypes are:
-
integer
-
decimal
-
float
-
date
-
datetime
-
time
-
string
table_schema = {:id => :integer, :amount => :decimal, :rate => :float}
The data_set must contain a hash with the column name and data value as the key-value pair.
For one row in the dataset:
data_set = {:id => 1, :amount => 1000.01, :rate => 0.0005}
For multiple rows in the dataset:
dataset = [{:id => 1, :amount => 1000.01, :rate => 0.0005}, {:id => 2, :amount => 2500.5, :rate => 0.1}, {:id => 3, :amount => 10.12, :rate => 10.0134}]
The path should be the full file path or relative to your working directory. You must also specify the file name.
path = '/temp/orc_file.orc'
Options is an optional hash parameter containing 5 configurable settings for writing an ORC file.
`:stripe_size` defines the size of the stripe, defaulted as 67,108,864 bytes <br> `:row_index_stride` defines the number of rows between row index entries, defaulted as 10,000 <br> `:buffer_size` defines the orc buffer size, defaulted as 262,144 bytes <br> `:compression` defines the compression codec (NONE,ZLIB,SNAPPY,LZO), defaulted as ZLIB. <br>
Define the options parameter has a hash
options = {:stripe_size => 70000000, :compression => 'SNAPPY'}
Once you have the OrcFileWriter object initialized you must call write_to_orc to write out the file
OrcFileWriter.new(table_schema, data_set, path, options).write_to_orc
To read a file, you will need to initialize the OrcFileReader class. This object needs a table schema, and the path of the file to be read.
OrcFileReader.new(table_schema, path)
The table_schema must be a hash containing the column name and datatype as the key-value pair.
Valid datatypes are:
-
integer
-
decimal
-
float
-
date
-
datetime
-
time
-
string
table_schema = {:id => :integer, :amount => :decimal, :rate => :float}
The path should be the full file path or relative to your working directory. You must also specify the file name.
path = '/temp/orc_file.orc'