Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-33749: [Ruby] Add Arrow::RecordBatch#each_raw_record #37137

Merged
merged 23 commits into from
Sep 5, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
8dd6ff6
[Ruby] Add Arrow::Table#each_raw_record for iterable raw record access
otegami Jul 25, 2023
3318abc
Introduce RawRecordProducer for specialized row-wise processing
otegami Aug 14, 2023
d3e5f3f
Refactor RawRecordsProducer for Row-Wise Processing in raw-records.cpp
otegami Aug 15, 2023
93adaac
Add EachRawRecordBasicArraysTest#test_boolean
otegami Aug 16, 2023
a16c4c4
Add the additonal test cases about the other data types to EachRawRec…
otegami Aug 17, 2023
b354011
Implemented RecordBatch#each_raw_record
otegami Aug 17, 2023
92b4b76
Fix typo test_tring -> test_string
otegami Aug 17, 2023
66ea580
Avoid needless reference count increment
otegami Aug 18, 2023
b0731a7
Refactor n_columns definition in produce methods
otegami Aug 18, 2023
34b53e0
Support multi-chunk ArrowTable structures for RawRecordsProducer#produce
otegami Aug 19, 2023
8305926
Initialized all member variables
otegami Aug 21, 2023
c0a07e2
Update data retrieval logic to handle varying chunk layouts
otegami Aug 21, 2023
1089b8c
Add empty chunk test case for the edge case
otegami Aug 21, 2023
97ff172
Refactor: Make number of columns and rows constants
otegami Aug 21, 2023
98a3599
Add test cases about dense union arrays
otegami Aug 27, 2023
516ef62
Add test cases about dictionary arrays
otegami Aug 28, 2023
7050dcc
Add test cases about list arrays
otegami Aug 30, 2023
72cc0b4
Add test cases about map arrays
otegami Aug 30, 2023
4456002
Add test cases about multiple columns
otegami Aug 30, 2023
e1e1cd2
Add test cases about sparse union arrays
otegami Sep 2, 2023
c7f4135
Add test cases about struct arrays
otegami Sep 2, 2023
595f0c9
Add test cases about table
otegami Sep 2, 2023
3f173ff
Improved the test case for multiple columns
otegami Sep 4, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions ruby/red-arrow/ext/arrow/arrow.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,9 @@ extern "C" void Init_arrow() {
rb_define_method(cArrowTable, "raw_records",
reinterpret_cast<rb::RawMethod>(red_arrow::table_raw_records),
0);
rb_define_method(cArrowTable, "each_raw_record",
reinterpret_cast<rb::RawMethod>(red_arrow::table_each_raw_record),
0);

red_arrow::cDate = rb_const_get(rb_cObject, rb_intern("Date"));

Expand Down
60 changes: 56 additions & 4 deletions ruby/red-arrow/ext/arrow/raw-records.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,9 @@ namespace red_arrow {
explicit RawRecordsBuilder(VALUE records, int n_columns)
: Converter(),
records_(records),
n_columns_(n_columns) {
record_(Qnil),
n_columns_(n_columns),
is_produce_mode_(false) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about creating RawRecordProducer instead of adding a new "produce" mode to RawRecordsBuilder?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix: 3318abc
Sounds pretty nice to me!
I tried to split RawRecordProducer into its own source file because some methods had to know the row-wise processing situation.
What do you think of it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I should have written RawRecordsProducer not RawRecordProducer because it produces multiple raw records.

So I think that we can still use raw-records.cpp instead of raw-record.cpp because it also handles multiple raw records not just one raw record.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix: d3e5f3f

}

void build(const arrow::RecordBatch& record_batch) {
Expand Down Expand Up @@ -68,6 +70,29 @@ namespace red_arrow {
});
}

void produce(const arrow::Table& table) {
rb::protect([&] {
is_produce_mode_ = true;
const auto n_rows = table.num_rows();
for (int64_t i = 0; i < n_rows; ++i) {
row_offset_ = i;
record_ = rb_ary_new_capa(n_columns_);

for (int i = 0; i < n_columns_; ++i) {
const auto& chunked_array = table.column(i).get();
column_index_ = i;

for (const auto array : chunked_array->chunks()) {
check_status(array->Accept(this),
"[table][each-raw-record]");
}
}
rb_yield(record_);
}
return Qnil;
});
}

#define VISIT(TYPE) \
arrow::Status Visit(const arrow::TYPE ## Array& array) override { \
convert(array); \
Expand Down Expand Up @@ -125,16 +150,23 @@ namespace red_arrow {
rb_ary_store(record, column_index_, value);
}
} else {
for (int64_t i = 0, ii = row_offset_; i < n; ++i, ++ii) {
auto record = rb_ary_entry(records_, ii);
rb_ary_store(record, column_index_, convert_value(array, i));
if (is_produce_mode_) {
rb_ary_store(record_, column_index_, convert_value(array, row_offset_));
} else {
for (int64_t i = 0, ii = row_offset_; i < n; ++i, ++ii) {
auto record = rb_ary_entry(records_, ii);
rb_ary_store(record, column_index_, convert_value(array, i));
}
}
}
}

// Destination for converted records.
VALUE records_;

// Destination for converted record.
VALUE record_;

// The current column index.
int column_index_;

Expand All @@ -143,6 +175,8 @@ namespace red_arrow {

// The number of columns.
const int n_columns_;

bool is_produce_mode_;
};
}

Expand Down Expand Up @@ -181,4 +215,22 @@ namespace red_arrow {

return records;
}

VALUE
table_each_raw_record(VALUE rb_table) {
auto garrow_table = GARROW_TABLE(RVAL2GOBJ(rb_table));
auto table = garrow_table_get_raw(garrow_table).get();
const auto n_rows = table->num_rows();
const auto n_columns = table->num_columns();
auto records = rb_ary_new_capa(n_rows);

try {
RawRecordsBuilder builder(records, n_columns);
builder.produce(*table);
} catch (rb::State& state) {
state.jump();
}

return Qnil;
}
}
1 change: 1 addition & 0 deletions ruby/red-arrow/ext/arrow/red-arrow.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ namespace red_arrow {

VALUE record_batch_raw_records(VALUE obj);
VALUE table_raw_records(VALUE obj);
VALUE table_each_raw_record(VALUE obj);

inline VALUE time_unit_to_scale(const arrow::TimeUnit::type unit) {
switch (unit) {
Expand Down