ARROW-1712: [C++] Add method to BinaryBuilder to reserve space for value data #1481

xuepanchen · 2018-01-16T16:09:56Z

Modified BinaryBuilder::Resize(int64_t) so that when building BinaryArrays with a known size, space is also reserved for value_data_builder_ to prevent internal reallocation.

Syncing from original

When building BinaryArrays with a known size using Resize and Reserve methods, space is also reserved for value_data_builder_ to prevent internal reallocation

Update BinaryBuilder::Resize(int64_t capacity) in builder.cc

xuepanchen · 2018-01-16T16:11:37Z

Plz let me know if I should create a new method and add allocation of value data in the new function, instead of directly putting inside the Resize method.

wesm · 2018-01-16T17:46:57Z

@xuepanchen thank you for your contribution. We need to add a new method to BinaryBuilder for this use case.

To give an example, suppose that we anticipate building an array with 1000 elements, each of which has an expected size of around 100 bytes. You would want to write something like:

RETURN_NOT_OK(builder.Reserve(1000));
RETURN_NOT_OK(builder.ReserveData(100 * 1000));

(@xhochy do you have an opinion on what to call this?)

Please also add a method to return the capacity of the internal value_data_builder_ and add unit tests to array-test.cc. Thanks!

…naryBuilder

…ilder

wesm · 2018-01-16T22:50:00Z

@xuepanchen note that each time you push any commits to GitHub on an open PR, it creates CI builds in our Travis CI and Appveyor queues, so small incremental pushes can impact other developers who are waiting on their builds to run. In general, it's a good practice to wait to open a PR on a WIP patch until you're ready to validate a completed patch and/or need code review.

If you enable Travis CI and Appveyor on your fork of Arrow, you can see CI builds on your branches without having to open a PR to the Arrow repo. e.g. https://travis-ci.org/wesm/arrow/branches

xuepanchen · 2018-01-16T23:18:04Z

@wesm thank you for the reminder. Will pay more attention next time.

xhochy · 2018-01-17T19:39:40Z

cpp/src/arrow/builder.cc

+
+Status BinaryBuilder::ReserveData(int64_t capacity) {
+  DCHECK_LT(capacity, std::numeric_limits<int32_t>::max());
+  return value_data_builder_.Resize(capacity * sizeof(int64_t));


Why do we multiply here with int64_t? I would expect that ReserveData(x) will lead to value_data_capacity() = x.

I think this should be Resize(capacity + value_data_builder_.length()). The sizeof(int64_t) looks like a copy-paste error from implementation of BinaryBuilder::Resize above (and that should be sizeof(int32_t) there, so we should fix that)

We should check that extra_capacity + length does not exceed INT32_MAX but probably return Status::Invalid since overflowing a BinaryBuilder is likely to happen somewhat more regularly

I note also that BufferBuilder and TypedBufferBuilder<T> don't have a shrink_to_fit option in their Resize method, that would be good to add to avoid unnecessary reallocations

Opened https://issues.apache.org/jira/browse/ARROW-2004

Thank you two for the comments.

@xhochy you are right. I will make the change accordingly.

@wesm I have modified BinaryBuilder::Resize to use sizeof(int32_t) in previous commit. Will change to Resize(extra_capacity + curr_length)

Replace parameter name from "bytes" to "capacity" to avoid confusion.

Add TestCapacityReserve to test space reservation for BinaryBuilder

…eservation

data_capacity_ represents the indicated capacity for value_data_builder and it is always smaller than or equal to the actual capacity of underlying value_data_builder (data_capacity_ <= value_data_builder.capacity()). That's because when we say: ReserveData(capacity); The new capacity is max(data_capacity_, data length + capacity), and data_capacity_ is set to be equal to new capacity but underlying buffer size is set to BitUtil::RoundUpToMultipleOf64(new capacity) to ensure that the capacity of the buffer is a multiple of 64 bytes as defined in Layout.md. That's why data_capacity_ is needed to show the indicated capacity of the BinaryBuilder, just like ArrayBuilder::capacity_ indicates the indicated capacity of ArrayBuilder. A safety check is added in BinaryBuilder::Append() to update data_capacity_ if data length is greater than data_capacity_. The reason is that data_capacity is updated in ResearveData(). But if users make mistakes to append too much data, data length might be larger than data_capacity_ (data length <= actual capacity of underlying value_data_builder). If this happens data_capacity_ is set equal to data length to avoid confusion.

Update ReserveData(int64_t) method for BinaryBuilder

xuepanchen · 2018-01-19T16:15:01Z

Update ReserveData method based on feedbacks and add test case for BinaryBuilder.

Syncing from original

wesm · 2018-01-19T17:31:26Z

cpp/src/arrow/array-test.cc

+  ASSERT_EQ(builder_->length(), length);
+  ASSERT_EQ(builder_->capacity(), BitUtil::NextPower2(capacity));
+  ASSERT_EQ(builder_->value_data_length(), data_length);
+  ASSERT_EQ(builder_->value_data_capacity(), capacity);


Why is this not a power of 2?

wesm · 2018-01-19T17:34:30Z

cpp/src/arrow/array-test.cc

+    if (data_length <= capacity) {
+      ASSERT_EQ(builder_->value_data_capacity(), capacity);
+    } else {
+      ASSERT_EQ(builder_->value_data_capacity(), data_length);


It's not clear to me why these assertions would hold true

ASSERT_EQ(builder_->capacity(), BitUtil::NextPower2(capacity)); ASSERT_EQ(builder_->value_data_capacity(), capacity);

I would think that value_data_capacity() is always the power of 2 greater than or equal to the amount of data appended so far, i.e. ASSERT_EQ(BitUtil::NextPower2(data_length), builder_->value_data_capacity())

Can you make the strings you are appending much larger, at least 10 length each?

@wesm value_data_capacity() is actually always a multiple of 64 greater than or equal to the amount of data appended so far because the underlying buffer size is set to ensure that the capacity of the buffer is a multiple of 64 bytes as defined in Layout.md, i.e.

ASSERT_EQ(BitUtil::RoundUpToMultipleOf64(data_length), builder_->value_data_capacity())

So if you call ReserveData(capacity) at the very beginning, then we have

ASSERT_EQ(BitUtil::RoundUpToMultipleOf64(capacity), builder_->value_data_capacity())

wesm · 2018-01-19T17:35:05Z

cpp/src/arrow/array-test.cc

+  ASSERT_OK(builder_->Reserve(capacity));
+  ASSERT_OK(builder_->ReserveData(capacity));
+
+  ASSERT_EQ(builder_->length(), length);


In googletest, the 1st parameter passed to ASSERT_EQ should be the expected result, so flip the argument order here and below

wesm · 2018-01-19T17:37:18Z

cpp/src/arrow/array-test.cc

+  int64_t capacity = N;
+
+  ASSERT_OK(builder_->Reserve(capacity));
+  ASSERT_OK(builder_->ReserveData(capacity));


These reservations should be viewed as different, because the size of the offsets buffer and the data buffer grow at different rates. The way this unit test should read is:

Call ReserveData with enough space for some large-ish amount of data (say 4K bytes or so)

Append <= N bytes incrementally

Check that the capacity remains invariant at the end (i.e. the initial ReserveData made sure that no additional reallocations took place)

wesm · 2018-01-19T17:37:57Z

cpp/src/arrow/builder.cc

@@ -1208,7 +1208,7 @@ ArrayBuilder* ListBuilder::value_builder() const {
 // String and binary

 BinaryBuilder::BinaryBuilder(const std::shared_ptr<DataType>& type, MemoryPool* pool)
-    : ArrayBuilder(type, pool), offsets_builder_(pool), value_data_builder_(pool) {}
+    : ArrayBuilder(type, pool), offsets_builder_(pool), value_data_builder_(pool), data_capacity_(0) {}


I don't think we need this extra member; we can just use value_data_capacity() wherever data_capacity_ is currently being used

wesm · 2018-01-19T17:38:41Z

cpp/src/arrow/builder.cc

+      return Status::Invalid("Cannot reserve capacity larger than 2^31 - 1 in length for binary data");
+    }    
+
+    RETURN_NOT_OK(value_data_builder_.Resize(value_data_length() + capacity));


If you rebase on master, you can use value_data_builder_.Reserve(capacity) here

wesm · 2018-01-19T17:38:48Z

cpp/src/arrow/builder.cc

+    }    
+
+    RETURN_NOT_OK(value_data_builder_.Resize(value_data_length() + capacity));
+    data_capacity_ = value_data_length() + capacity;


wesm · 2018-01-19T17:39:04Z

cpp/src/arrow/builder.cc

@@ -1241,6 +1253,9 @@ Status BinaryBuilder::Append(const uint8_t* value, int32_t length) {
  RETURN_NOT_OK(Reserve(1));
  RETURN_NOT_OK(AppendNextOffset());
  RETURN_NOT_OK(value_data_builder_.Append(value, length));
+  if (data_capacity_ < value_data_length()) {
+    data_capacity_ = value_data_length();
+  }


Remove this

wesm · 2018-01-19T17:40:25Z

cpp/src/arrow/builder.h

@@ -682,10 +682,13 @@ class ARROW_EXPORT BinaryBuilder : public ArrayBuilder {

  Status Init(int64_t elements) override;
  Status Resize(int64_t capacity) override;
+  Status ReserveData(int64_t capacity);


Can you add a doxygen comment, since this is a new API

I note that the Reserve-type methods in this header have different semantics from their STL counterparts. They are reserving additional space rather than absolute space (e.g. std::vector::reserve takes an absolute length as argument)

Can we also call the argument to ReserveData size (or elements) instead of capacity (to avoid confusion about whether we are passing an incremental value vs. absolute)

…dify test case

wesm

Thanks, this is looking pretty close, just a couple of minor comments

wesm · 2018-01-20T23:17:49Z

cpp/src/arrow/builder.h

@@ -682,10 +682,15 @@ class ARROW_EXPORT BinaryBuilder : public ArrayBuilder {

  Status Init(int64_t elements) override;
  Status Resize(int64_t capacity) override;
+  /// Ensures there is enough space for adding the number of value elements 
+  /// by checking value buffer capacity and resizing if necessary.


Can you add \brief to the method description? Can we rephrase this to "Ensures there is enough allocated capacity to append the indicated number of bytes to the value data buffer without additional allocations"

wesm · 2018-01-20T23:21:27Z

cpp/src/arrow/array-test.cc

+      ASSERT_EQ(length, builder_->value_data_length());
+      ASSERT_EQ(BitUtil::RoundUpToMultipleOf64(capacity), builder_->value_data_capacity());
+    }
+  }


Can you add another call to ReserveData here, like builder_->ReserveData(500) to show that the input argument is an incremental amount rather than an absolute amount?

Syncing from original

…nd change arguments for offsets_builder_.Resize()

wesm

+1 -- thank you. I will merge once the style issues are fixed and the build is passing. There are a number of linting problems, see https://github.com/apache/arrow/tree/master/cpp#continuous-integration

wesm · 2018-01-22T23:19:54Z

In the future, it is better to make pull requests from a branch on your fork, so that you can work on multiple changes at once. Here are some links from the pandas project about this

http://pandas.pydata.org/pandas-docs/stable/contributing.html#creating-a-branch . This includes instructions for how to update a branch with the latest changes from the upstream branch
http://pandas.pydata.org/pandas-docs/stable/contributing.html#contributing-your-changes-to-pandas note that squashing commits yourself is not necessary in Apache Arrow because our merge tool does that when merging the PR at the end

wesm · 2018-01-23T22:59:29Z

@kou do you know what is causing this error?

Building dependency tree...
Reading state information...
ccache is already the newest version (3.1.9-1).
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 libgirepository1.0-dev : Depends: libgirepository-1.0-1 (= 1.40.0-1) but 1.40.0-1ubuntu0.2 is to be installed
                          Depends: gobject-introspection (= 1.40.0-1) but it is not going to be installed
                          Depends: gir1.2-glib-2.0 (= 1.40.0-1) but 1.40.0-1ubuntu0.2 is to be installed
                          Depends: gir1.2-freedesktop (= 1.40.0-1) but 1.40.0-1ubuntu0.2 is to be installed
E: Unable to correct problems, you have held broken packages.

wesm · 2018-01-23T23:01:38Z

cpp/src/arrow/array-test.cc

 TEST_F(TestBinaryBuilder, TestCapacityReserve) {
-  vector<string> strings = {"aaaaa", "bbbbbbbbbb", "ccccccccccccccc", "dddddddddddddddddddd", "eeeeeeeeee"};
+  vector<string> strings = {"aaaaa", "bbbbbbbbbb", "ccccccccccccccc", "dddddddddd"};


In the future, you can run make format (which uses clang-format) to fix these long lines without having to make code changes

kou · 2018-01-24T01:49:01Z

@wesm I think that it's a network problem:

https://travis-ci.org/apache/arrow/jobs/332472808#L552

E: Failed to fetch http://us-central1.gce.archive.ubuntu.com/ubuntu/dists/trusty-updates/main/binary-amd64/Packages.gz Hash Sum mismatch

We will be able to reduce the case by changing the apt-get update command line with the following:

for i in {1..3}; do
  sudo -E apt-get -yq update &>> ~/apt-get-update.log && break
done

wesm · 2018-01-24T03:06:48Z

@kou I see. These commands are initiated by Travis CI from https://github.com/apache/arrow/blob/master/.travis.yml#L21. We could install our package toolchain outside of Travis CI's built-in commands, if that might help improve the flakiness

I opened https://issues.apache.org/jira/browse/ARROW-2021

xuepanchen added 4 commits January 15, 2018 13:40

Merge pull request #1 from apache/master

c2f8dc4

Syncing from original

Update BinaryBuilder::Resize(int64_t capacity) in builder.cc

232024e

When building BinaryArrays with a known size using Resize and Reserve methods, space is also reserved for value_data_builder_ to prevent internal reallocation

Merge pull request #2 from xuepanchen/xuepanchen-arrow-1712

d021c54

Update BinaryBuilder::Resize(int64_t capacity) in builder.cc

Update again BinaryBuilder::Resize(int64_t capacity) in builder.cc

5b73c1c

xuepanchen changed the title ~~[C++] Add method to BinaryBuilder to reserve space for value data~~ ARROW-1712: [C++] Add method to BinaryBuilder to reserve space for value data Jan 16, 2018

xuepanchen added 4 commits January 16, 2018 16:40

Add capacity() method for TypedBufferBuilder

5ebfb32

Add ReserveData(int64_t) and value_data_capacity() for methods for Bi…

e0434e6

…naryBuilder

Implement ReserveData(int64_t) method for BinaryBuilder

de318f4

Remove override keyword from ReserveData(int64_t) method for BinaryBu…

b002e0b

…ilder

xhochy requested changes Jan 17, 2018

View reviewed changes

xuepanchen added 9 commits January 17, 2018 15:56

Update builder.cc

8dd5eaa

Update ReserveData(int64_t) method signature for BinaryBuilder

9b5e805

Replace parameter name from "bytes" to "capacity" to avoid confusion.

Update again ReserveData(int64_t) method for BinaryBuilder

5a5593e

Add test case for array-test.cc

15e045c

Add TestCapacityReserve to test space reservation for BinaryBuilder

ARROW-1945: [C++] Update test case for BinaryBuild data value space r…

bbc6527

…eservation

ARROW-1945: [C++] Add data_capacity_ to track capacity of value data

0b07895

ARROW-1945: [C++] Fix a small typo

d3c8202

Merge pull request #3 from xuepanchen/xuepanchen-arrow-1712

8e4c892

Update ReserveData(int64_t) method for BinaryBuilder

Merge pull request #4 from apache/master

5a5b70e

Syncing from original

wesm reviewed Jan 19, 2018

View reviewed changes

ARROW-1712: [C++] Remove unneeded data member in BinaryBuilder and mo…

bc5db7d

…dify test case

wesm reviewed Jan 20, 2018

View reviewed changes

xuepanchen added 2 commits January 22, 2018 11:32

Merge pull request #5 from apache/master

77f8f3c

Syncing from original

ARROW-1712: [C++] Modify test case for BinaryBuilder::ReserveData() a…

d4bbd15

…nd change arguments for offsets_builder_.Resize()

Merge branch 'master' of https://github.com/xuepanchen/arrow

360e601

wesm reviewed Jan 22, 2018

View reviewed changes

ARROW-1712: [C++] Fix lint errors

707b67b

wesm reviewed Jan 23, 2018

View reviewed changes

wesm closed this in 0a49022 Jan 24, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-1712: [C++] Add method to BinaryBuilder to reserve space for value data #1481

ARROW-1712: [C++] Add method to BinaryBuilder to reserve space for value data #1481

xuepanchen commented Jan 16, 2018

xuepanchen commented Jan 16, 2018

wesm commented Jan 16, 2018

wesm commented Jan 16, 2018

xuepanchen commented Jan 16, 2018 •

edited

Loading

xhochy Jan 17, 2018

wesm Jan 17, 2018

wesm Jan 17, 2018

xuepanchen Jan 17, 2018

xuepanchen commented Jan 19, 2018

wesm Jan 19, 2018

wesm Jan 19, 2018

xuepanchen Jan 19, 2018

wesm Jan 19, 2018

wesm Jan 19, 2018

wesm Jan 19, 2018

wesm Jan 19, 2018

wesm Jan 19, 2018

wesm Jan 19, 2018

wesm Jan 19, 2018

wesm Jan 19, 2018

wesm left a comment

wesm Jan 20, 2018

wesm Jan 20, 2018

wesm left a comment

wesm commented Jan 22, 2018

wesm commented Jan 23, 2018

wesm Jan 23, 2018

kou commented Jan 24, 2018

wesm commented Jan 24, 2018

ARROW-1712: [C++] Add method to BinaryBuilder to reserve space for value data #1481

ARROW-1712: [C++] Add method to BinaryBuilder to reserve space for value data #1481

Conversation

xuepanchen commented Jan 16, 2018

xuepanchen commented Jan 16, 2018

wesm commented Jan 16, 2018

wesm commented Jan 16, 2018

xuepanchen commented Jan 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xuepanchen commented Jan 19, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wesm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wesm left a comment

Choose a reason for hiding this comment

wesm commented Jan 22, 2018

wesm commented Jan 23, 2018

Choose a reason for hiding this comment

kou commented Jan 24, 2018

wesm commented Jan 24, 2018

xuepanchen commented Jan 16, 2018 •

edited

Loading