Our strategy for integration testing between Arrow implementations is:
- Test datasets are specified in a custom human-readable, JSON-based format designed exclusively for Arrow's integration tests
- Each implementation provides a testing executable capable of converting between the JSON and the binary Arrow file representation
- The test executable is also capable of validating the contents of a binary file against a corresponding JSON file
The integration test data generator and runner uses archery
, a Python script
that requires Python 3.6 or higher. You can create a standalone Python
distribution and environment for running the tests by using
miniconda. On Linux this is:
MINICONDA_URL=https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
wget -O miniconda.sh $MINICONDA_URL
bash miniconda.sh -b -p miniconda
export PATH=`pwd`/miniconda/bin:$PATH
conda create -n arrow-integration python=3.6 nomkl numpy six
conda activate arrow-integration
If you are on macOS, instead use the URL:
MINICONDA_URL=https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
Once you have Python, you can install archery
pip install -e dev/archery
The integration tests are run using the archery integration
command.
archery integration --help
In order to run integration tests, you'll first need to build each component you want to include. See the respective developer docs for C++, Java, etc. for instructions on building those.
Some languages may require additional build options to enable integration
testing. For C++, for example, you need to add -DARROW_BUILD_INTEGRATION=ON
to your cmake command.
Depending on which components you have built, you can enable and add them to the archery test run. For example, if you only have the C++ project built, run:
archery integration --with-cpp=1
For Java, it may look like:
VERSION=0.11.0-SNAPSHOT
export ARROW_JAVA_INTEGRATION_JAR=$JAVA_DIR/tools/target/arrow-tools-$VERSION-jar-with-dependencies.jar
archery integration --with-cpp=1 --with-java=1
To run all tests, including Flight integration tests, do:
archery integration --with-all --run-flight
Note that we run these tests in continuous integration, and the CI job uses docker-compose. You may also run the docker-compose job locally, or at least refer to it if you have questions about how to build other languages or enable certain tests.
See :ref:`integration` for more information about the project's
docker-compose
configuration.
A JSON representation of Arrow columnar data is provided for cross-language integration testing purposes. This representation is not canonical but it provides a human-readable way of verifying language implementations.
See here for some examples of this JSON data.
The high level structure of a JSON integration test files is as follows:
Data file
{ "schema": /*Schema*/, "batches": [ /*RecordBatch*/ ], "dictionaries": [ /*DictionaryBatch*/ ], }
All files contain schema
and batches
, while dictionaries
is only
present if there are dictionary type fields in the schema.
Schema
{ "fields" : [ /* Field */ ], "metadata" : /* Metadata */ }
Field
{ "name" : "name_of_the_field", "nullable" : /* boolean */, "type" : /* Type */, "children" : [ /* Field */ ], "dictionary": { "id": /* integer */, "indexType": /* Type */, "isOrdered": /* boolean */ }, "metadata" : /* Metadata */ }
The dictionary
attribute is present if and only if the Field
corresponds to a
dictionary type, and its id
maps onto a column in the DictionaryBatch
. In this
case the type
attribute describes the value type of the dictionary.
For primitive types, children
is an empty array.
Metadata
null | [ { "key": /* string */, "value": /* string */ } ]
A key-value mapping of custom metadata. It may be omitted or null, in which case it is
considered equivalent to []
(no metadata). Duplicated keys are not forbidden here.
Type:
{ "name" : "null|struct|list|largelist|fixedsizelist|union|int|floatingpoint|utf8|largeutf8|binary|largebinary|fixedsizebinary|bool|decimal|date|time|timestamp|interval|duration|map" }
A Type
will have other fields as defined in
Schema.fbs
depending on its name.
Int:
{ "name" : "int", "bitWidth" : /* integer */, "isSigned" : /* boolean */ }
FloatingPoint:
{ "name" : "floatingpoint", "precision" : "HALF|SINGLE|DOUBLE" }
FixedSizeBinary:
{ "name" : "fixedsizebinary", "byteWidth" : /* byte width */ }
Decimal:
{ "name" : "decimal", "precision" : /* integer */, "scale" : /* integer */ }
Timestamp:
{ "name" : "timestamp", "unit" : "$TIME_UNIT", "timezone": "$timezone" }
$TIME_UNIT
is one of "SECOND|MILLISECOND|MICROSECOND|NANOSECOND"
"timezone" is an optional string.
Duration:
{ "name" : "duration", "unit" : "$TIME_UNIT" }
Date:
{ "name" : "date", "unit" : "DAY|MILLISECOND" }
Time:
{ "name" : "time", "unit" : "$TIME_UNIT", "bitWidth": /* integer: 32 or 64 */ }
Interval:
{ "name" : "interval", "unit" : "YEAR_MONTH|DAY_TIME" }
Union:
{ "name" : "union", "mode" : "Sparse|Dense", "typeIds" : [ /* integer */ ] }
The typeIds
field in Union
are the codes used to denote which member of
the union is active in each array slot. Note that in general these discriminants are not identical
to the index of the corresponding child array.
List:
{ "name": "list" }
The type that the list is a "list of" will be included in the Field
's
"children" member, as a single Field
there. For example, for a list of
int32
,
{ "name": "list_nullable", "type": { "name": "list" }, "nullable": true, "children": [ { "name": "item", "type": { "name": "int", "isSigned": true, "bitWidth": 32 }, "nullable": true, "children": [] } ] }
FixedSizeList:
{ "name": "fixedsizelist", "listSize": /* integer */ }
This type likewise comes with a length-1 "children" array.
Struct:
{ "name": "struct" }
The Field
's "children" contains an array of Fields
with meaningful
names and types.
Map:
{ "name": "map", "keysSorted": /* boolean */ }
The Field
's "children" contains a single struct
field, which itself
contains 2 children, named "key" and "value".
Null:
{ "name": "null" }
RecordBatch:
{ "count": /* integer number of rows */, "columns": [ /* FieldData */ ] }
DictionaryBatch:
{ "id": /* integer */, "data": [ /* RecordBatch */ ] }
FieldData:
{ "name": "field_name", "count" "field_length", "$BUFFER_TYPE": /* BufferData */ ... "$BUFFER_TYPE": /* BufferData */ "children": [ /* FieldData */ ] }
The "name" member of a Field
in the Schema
corresponds to the "name"
of a FieldData
contained in the "columns" of a RecordBatch
.
For nested types (list, struct, etc.), Field
's "children" each have a
"name" that corresponds to the "name" of a FieldData
inside the
"children" of that FieldData
.
For FieldData
inside of a DictionaryBatch
, the "name" field does not
correspond to anything.
Here $BUFFER_TYPE
is one of VALIDITY
, OFFSET
(for
variable-length types, such as strings and lists), TYPE
(for unions),
or DATA
.
BufferData
is encoded based on the type of buffer:
VALIDITY
: a JSON array of 1 (valid) and 0 (null). Data for non-nullableField
still has aVALIDITY
array, even though all values are 1.OFFSET
: a JSON array of integers for 32-bit offsets or string-formatted integers for 64-bit offsetsTYPE
: a JSON array of integersDATA
: a JSON array of encoded values
The value encoding for DATA
is different depending on the logical
type:
- For boolean type: an array of 1 (true) and 0 (false)
- For integer-based types (including timestamps): an array of integers
- For 64-bit integers: an array of integers formatted as JSON strings to avoid loss of precision
- For floating point types: as is. Values are limited to 3 decimal places to avoid loss of precision
- For Binary types, a hex-string is produced to encode a variable- or fixed-size binary value
For "list" type, BufferData
has VALIDITY
and OFFSET
, and the
rest of the data is inside "children". These child FieldData
contain all
of the same attributes as non-child data, so in the example of a list of
int32
, the child data has VALIDITY
and DATA
.
For "fixedsizelist", there is no OFFSET
member because the offsets are
implied by the field's "listSize".
Note that the "count" for these child data may not match the parent "count".
For example, if a RecordBatch
has 7 rows and contains a FixedSizeList
of listSize
4, then the data inside the "children" of that FieldData
will have count 28.
For "null" type, BufferData
does not contain any buffers.