Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ARROW-1011: [FORMAT] fix typo and mistakes in Layout.md #673

Closed
wants to merge 1 commit into from
Closed

ARROW-1011: [FORMAT] fix typo and mistakes in Layout.md #673

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
22 changes: 11 additions & 11 deletions format/Layout.md
View file
Open in desktop
Expand Up @@ -51,7 +51,7 @@ Base requirements

* A physical memory layout enabling zero-deserialization data interchange
amongst a variety of systems handling flat and nested columnar data, including
such systems as Spark, Drill, Impala, Kudu, Ibis, Spark, ODBC protocols, and
such systems as Spark, Drill, Impala, Kudu, Ibis, ODBC protocols, and
proprietary systems that utilize the open source components.
* All array slots are accessible in constant time, with complexity growing
linearly in the nesting level
Expand Down Expand Up @@ -114,7 +114,7 @@ data-structures over 64 bytes (which will be a common case for Arrow Arrays).

Requiring padding to a multiple of 64 bytes allows for using [SIMD][4] instructions
consistently in loops without additional conditional checks.
This should allow for simpler and more efficient code.
This should allow for simpler and more efficient code.
The specific padding length was chosen because it matches the largest known
SIMD instruction registers available as of April 2016 (Intel AVX-512).
Guaranteed padding can also allow certain compilers
Expand Down Expand Up @@ -146,7 +146,7 @@ signed integer, as it may be as large as the array length.
Any relative type can have null value slots, whether primitive or nested type.

An array with nulls must have a contiguous memory buffer, known as the null (or
validity) bitmap, whose length is a multiple of 64 bytes (as discussed above)
validity) bitmap, whose length is a multiple of 64 bytes (as discussed above)
and large enough to have at least 1 bit for each array
slot.

Expand Down Expand Up @@ -205,7 +205,7 @@ Would look like:

|Byte 0 (validity bitmap) | Bytes 1-63 |
|-------------------------|-----------------------|
|00011011 | 0 (padding) |
| 00011011 | 0 (padding) |

* Value Buffer:

Expand Down Expand Up @@ -378,16 +378,16 @@ The layout for [{'joe', 1}, {null, 2}, null, {'mark', 4}] would be:
* Length: 4, Null count: 1
* Null bitmap buffer:

| Byte 0 (validity bitmap) | Bytes 1-7 | Bytes 8-63 |
|--------------------------|-------------|-------------|
| 00001011 | 0 (padding) | unspecified |
|Byte 0 (validity bitmap) | Bytes 1-63 |
|-------------------------|-----------------------|
| 00001011 | 0 (padding) |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC this was a deliberate choice so that one can rely upon accurate null counts using a byte-wise hardware popcount instruction. Though I'm not sure how safe that is in general since we can do:

array->Slice(0, 1)

which would yield a slice of length 1, whose additional bytes should not be inspected in algorithms. In an IPC setting, the padding to byte 63 should be all zeros (see e.g. https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/writer.cc#L220), so I think this is fine. We will want to make sure in the IPC writers that we do not write possibly unspecified in-memory bytes from the validity bitmap, instead truncating the buffer to the effective byte range and writing zero padding. It might be worth adding some language to this document about this.

Per our prior discussion about padding -- since this document is about what data gets written to stream or file format, it may be good to be more rigid about zero-ing out the padding bytes that go on the wire. This places a slight additional burden on the writers, though

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think ideally we only need to use zero padding for null bitmap in a word(8 bytes) boundary, which means the previous code is accurate.

However, to make the rule simpler, I think it also makes sense to require all the padding bytes to be 0 for null bitmap. This is also what we specified in the "Null bitmaps" section:

A 1 (set bit) for index j indicates that the value is not null, while a 0 (bit not set) indicates that it is null. Bitmaps are to be initialized to be all unset at allocation time (this includes padding).


* Children arrays:
* field-0 array (`List<char>`):
* Length: 4, Null count: 1
* Null bitmap buffer:

| Byte 0 (validity bitmap) | Bytes 1-7 |
| Byte 0 (validity bitmap) | Bytes 1-63 |
|--------------------------|-----------------------|
| 00001101 | 0 (padding) |

Expand Down Expand Up @@ -447,7 +447,7 @@ of overhead for each value. Its physical layout is as follows:
* One child array for each relative type
* Types buffer: A buffer of 8-bit signed integers, enumerated from 0 corresponding
to each type. A union with more then 127 possible types can be modeled as a
union of unions.
union of unions.
* Offsets buffer: A buffer of signed int32 values indicating the relative offset
into the respective child array for the type in a given slot. The respective
offsets for each child value array must be in order / increasing.
Expand Down Expand Up @@ -555,7 +555,7 @@ will have the following layout:

|Byte 0 (validity bitmap) | Bytes 1-63 |
|-------------------------|-----------------------|
|00001010 | 0 (padding) |
| 00001010 | 0 (padding) |

* Value buffer:

Expand Down Expand Up @@ -636,7 +636,7 @@ type: List<String>

## References

Apache Drill Documentation - [Value Vectors][6]
Apache Drill Documentation - [Value Vectors][6]

[1]: https://en.wikipedia.org/wiki/Bit_numbering
[2]: https://software.intel.com/en-us/articles/practical-intel-avx-optimization-on-2nd-generation-intel-core-processors
Expand Down