Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARROW-648: [C++] Support multiarch on Debian #398

Closed
wants to merge 2 commits into from

Conversation

kou
Copy link
Member

@kou kou commented Mar 17, 2017

On multiarch enabled Debian, we need to install libraries into
${CMAKE_INSTALL_PREFIX}/lib/${ARCH}/ instead of
${CMAKE_INSTALL_PREFIX}/lib/.

See also: https://wiki.debian.org/Multiarch/HOWTO

On multiarch enabled Debian, we need to install libraries into
${CMAKE_INSTALL_PREFIX}/lib/${ARCH}/ instead of
${CMAKE_INSTALL_PREFIX}/lib/.
Copy link
Member

@xhochy xhochy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM

@kou
Copy link
Member Author

kou commented Mar 17, 2017

Sorry... The original changes have a bug... I've added a commit that fixes the bug.

@asfgit asfgit closed this in f5157a0 Mar 18, 2017
@wesm
Copy link
Member

wesm commented Mar 22, 2017

@kou do you know how other projects handle this issue? We need to be able to install these libraries in a deterministic location -- when the user passes -DCMAKE_INSTALL_PREFIX it is not respected. For example, I have been installing in $HOME/lib, but now the packages are going in

/home/wesm/local/lib/x86_64-linux-gnu/libarrow_io.so.0.0.0

This does not look right to me https://issues.apache.org/jira/browse/ARROW-680

@kou kou deleted the debian-support-multiarch branch March 22, 2017 00:53
@kou
Copy link
Member Author

kou commented Mar 22, 2017

Umm... I couldn't reproduce it.

I'm using cmake .. -DCMAKE_INSTALL_PREFIX=/tmp/local and libarrow_io.so.0.0.0 is installed into /tmp/local/lib/libarrow_ipc.so.0.0.0.

I'm on Debian GNU/Linux.

@wesm
Copy link
Member

wesm commented Mar 22, 2017

From a clean out of source build, using gcc 4.9.2 not obtained from apt:

Install the project...
-- Install configuration: "DEBUG"
-- Installing: /home/wesm/local/lib/x86_64-linux-gnu/libarrow.so.0.0.0
-- Up-to-date: /home/wesm/local/lib/x86_64-linux-gnu/libarrow.so.0
-- Up-to-date: /home/wesm/local/lib/x86_64-linux-gnu/libarrow.so
-- Installing: /home/wesm/local/lib/x86_64-linux-gnu/libarrow.a
-- Up-to-date: /home/wesm/local/include/arrow/allocator.h
-- Up-to-date: /home/wesm/local/include/arrow/api.h
-- Up-to-date: /home/wesm/local/include/arrow/array.h
-- Up-to-date: /home/wesm/local/include/arrow/buffer.h
-- Up-to-date: /home/wesm/local/include/arrow/builder.h
-- Up-to-date: /home/wesm/local/include/arrow/column.h
-- Up-to-date: /home/wesm/local/include/arrow/compare.h
-- Up-to-date: /home/wesm/local/include/arrow/loader.h
-- Up-to-date: /home/wesm/local/include/arrow/memory_pool.h
-- Up-to-date: /home/wesm/local/include/arrow/pretty_print.h
-- Up-to-date: /home/wesm/local/include/arrow/schema.h
-- Up-to-date: /home/wesm/local/include/arrow/status.h
-- Up-to-date: /home/wesm/local/include/arrow/table.h
-- Up-to-date: /home/wesm/local/include/arrow/type.h
-- Up-to-date: /home/wesm/local/include/arrow/type_fwd.h
-- Up-to-date: /home/wesm/local/include/arrow/type_traits.h
-- Up-to-date: /home/wesm/local/include/arrow/test-util.h
-- Installing: /home/wesm/local/lib/x86_64-linux-gnu/pkgconfig/arrow.pc
-- Installing: /home/wesm/local/lib/x86_64-linux-gnu/libarrow_io.so.0.0.0
-- Up-to-date: /home/wesm/local/lib/x86_64-linux-gnu/libarrow_io.so.0
-- Up-to-date: /home/wesm/local/lib/x86_64-linux-gnu/libarrow_io.so
-- Removed runtime path from "/home/wesm/local/lib/x86_64-linux-gnu/libarrow_io.so.0.0.0"
-- Installing: /home/wesm/local/lib/x86_64-linux-gnu/libarrow_io.a
-- Up-to-date: /home/wesm/local/include/arrow/io/file.h
-- Up-to-date: /home/wesm/local/include/arrow/io/hdfs.h
-- Up-to-date: /home/wesm/local/include/arrow/io/interfaces.h
-- Up-to-date: /home/wesm/local/include/arrow/io/memory.h
-- Installing: /home/wesm/local/lib/x86_64-linux-gnu/pkgconfig/arrow-io.pc
-- Up-to-date: /home/wesm/local/include/arrow/util/bit-util.h
-- Up-to-date: /home/wesm/local/include/arrow/util/logging.h
-- Up-to-date: /home/wesm/local/include/arrow/util/macros.h
-- Up-to-date: /home/wesm/local/include/arrow/util/random.h
-- Up-to-date: /home/wesm/local/include/arrow/util/visibility.h
-- Installing: /home/wesm/local/lib/x86_64-linux-gnu/libarrow_jemalloc.so.0.0.0
-- Installing: /home/wesm/local/lib/x86_64-linux-gnu/libarrow_jemalloc.so.0
-- Installing: /home/wesm/local/lib/x86_64-linux-gnu/libarrow_jemalloc.so
-- Removed runtime path from "/home/wesm/local/lib/x86_64-linux-gnu/libarrow_jemalloc.so.0.0.0"
-- Installing: /home/wesm/local/lib/x86_64-linux-gnu/libarrow_jemalloc.a
-- Installing: /home/wesm/local/include/arrow/jemalloc/memory_pool.h
-- Installing: /home/wesm/local/lib/x86_64-linux-gnu/pkgconfig/arrow-jemalloc.pc
-- Installing: /home/wesm/local/lib/x86_64-linux-gnu/libarrow_ipc.so.0.0.0
-- Up-to-date: /home/wesm/local/lib/x86_64-linux-gnu/libarrow_ipc.so.0
-- Up-to-date: /home/wesm/local/lib/x86_64-linux-gnu/libarrow_ipc.so
-- Removed runtime path from "/home/wesm/local/lib/x86_64-linux-gnu/libarrow_ipc.so.0.0.0"
-- Installing: /home/wesm/local/lib/x86_64-linux-gnu/libarrow_ipc.a
-- Up-to-date: /home/wesm/local/include/arrow/ipc/api.h
-- Up-to-date: /home/wesm/local/include/arrow/ipc/feather.h
-- Up-to-date: /home/wesm/local/include/arrow/ipc/json.h
-- Up-to-date: /home/wesm/local/include/arrow/ipc/metadata.h
-- Up-to-date: /home/wesm/local/include/arrow/ipc/reader.h
-- Up-to-date: /home/wesm/local/include/arrow/ipc/writer.h
-- Installing: /home/wesm/local/lib/x86_64-linux-gnu/pkgconfig/arrow-ipc.pc

@wesm
Copy link
Member

wesm commented Mar 22, 2017

this is on Ubuntu 14.04 LTS

@wesm
Copy link
Member

wesm commented Mar 22, 2017

also, in case it's relevant:

$ cmake --version
cmake version 2.8.12.2

@wesm
Copy link
Member

wesm commented Mar 22, 2017

Looks like the problem is that my GNUInstallDirs.cmake is kind of old, and I have i386 enabled on my x86_64 machine, see the commit:

Kitware/CMake@620939e

I can install a newer cmake to solve this issue, but it would be unfortunate to have this failure mode out of the box on 14.04 LTS

@kou
Copy link
Member Author

kou commented Mar 22, 2017

I've created a pull request as #419 for Ubuntu 14.04.

wesm added a commit to wesm/arrow that referenced this pull request Sep 2, 2018
…ed data, refactor parquet/arrow/reader

We did not have very consistent logic around reading values from leaf nodes versus reading semantic records where the repetition level is greater than zero. This introduces a reader class that reads from column chunks until it identifies the end of records. It also reads values (with spaces, if required by the schema) into internal buffers. This permitted a substantial refactoring and simplification of the code in parquet::arrow where we were handling the interpretation of batch reads as records manually.

As follow up patch, we should be able to take a collection of record readers from the same "tree" in a nested type and reassemble the intermediate Arrow structure and dealing with any redundant structure information in repetition and definition levels. This should a allow a unification of our nested data read code path so that we can read arbitrary nested structures.

Author: Wes McKinney <wes.mckinney@twosigma.com>

Closes apache#398 from wesm/PARQUET-1100 and squashes the following commits:

9ea85d9 [Wes McKinney] Revert to const args
f4dc0fe [Wes McKinney] Make parquet::schema::Node non-copyable. Use const-refs instead of const-ptr for non-nullable argument
0d859cc [Wes McKinney] Code review comments, scrubbing some flakes
1368415 [Wes McKinney] Fix more MSVC warnings
eccb84c [Wes McKinney] Give macro more accurate name
0eaada0 [Wes McKinney] Use int64_t instead of int for batch sizes
79c3709 [Wes McKinney] Add documentation. Remove RecordReader from public API
8fa619b [Wes McKinney] Initialize memory in DecodeSpaced to avoid undefined behavior
5a0c860 [Wes McKinney] Remove non-repeated branch from DelimitRecords
c754e6e [Wes McKinney] Refactor to skip record delimiting for non-repeated data
ed2a03f [Wes McKinney] Move more code into TypedRecordReader
2e934e9 [Wes McKinney] Set some integers as const
58d3a0f [Wes McKinney] Do not index into levels arrays
b766371 [Wes McKinney] Add RecordReader::Reserve to preallocate, fixing perf regression. cpplint
1bf3e8f [Wes McKinney] Refactor to create stateful parquet::RecordReader class to better support nested data. Shift value buffering logic from parquet/arrow/reader into RecordReader. Fix bug described in PARQUET-1100
wesm added a commit to wesm/arrow that referenced this pull request Sep 4, 2018
…ed data, refactor parquet/arrow/reader

We did not have very consistent logic around reading values from leaf nodes versus reading semantic records where the repetition level is greater than zero. This introduces a reader class that reads from column chunks until it identifies the end of records. It also reads values (with spaces, if required by the schema) into internal buffers. This permitted a substantial refactoring and simplification of the code in parquet::arrow where we were handling the interpretation of batch reads as records manually.

As follow up patch, we should be able to take a collection of record readers from the same "tree" in a nested type and reassemble the intermediate Arrow structure and dealing with any redundant structure information in repetition and definition levels. This should a allow a unification of our nested data read code path so that we can read arbitrary nested structures.

Author: Wes McKinney <wes.mckinney@twosigma.com>

Closes apache#398 from wesm/PARQUET-1100 and squashes the following commits:

9ea85d9 [Wes McKinney] Revert to const args
f4dc0fe [Wes McKinney] Make parquet::schema::Node non-copyable. Use const-refs instead of const-ptr for non-nullable argument
0d859cc [Wes McKinney] Code review comments, scrubbing some flakes
1368415 [Wes McKinney] Fix more MSVC warnings
eccb84c [Wes McKinney] Give macro more accurate name
0eaada0 [Wes McKinney] Use int64_t instead of int for batch sizes
79c3709 [Wes McKinney] Add documentation. Remove RecordReader from public API
8fa619b [Wes McKinney] Initialize memory in DecodeSpaced to avoid undefined behavior
5a0c860 [Wes McKinney] Remove non-repeated branch from DelimitRecords
c754e6e [Wes McKinney] Refactor to skip record delimiting for non-repeated data
ed2a03f [Wes McKinney] Move more code into TypedRecordReader
2e934e9 [Wes McKinney] Set some integers as const
58d3a0f [Wes McKinney] Do not index into levels arrays
b766371 [Wes McKinney] Add RecordReader::Reserve to preallocate, fixing perf regression. cpplint
1bf3e8f [Wes McKinney] Refactor to create stateful parquet::RecordReader class to better support nested data. Shift value buffering logic from parquet/arrow/reader into RecordReader. Fix bug described in PARQUET-1100

Change-Id: I94c1dd3f7936e30036e9c42eef689e86fb5fe7fa
wesm added a commit to wesm/arrow that referenced this pull request Sep 6, 2018
…ed data, refactor parquet/arrow/reader

We did not have very consistent logic around reading values from leaf nodes versus reading semantic records where the repetition level is greater than zero. This introduces a reader class that reads from column chunks until it identifies the end of records. It also reads values (with spaces, if required by the schema) into internal buffers. This permitted a substantial refactoring and simplification of the code in parquet::arrow where we were handling the interpretation of batch reads as records manually.

As follow up patch, we should be able to take a collection of record readers from the same "tree" in a nested type and reassemble the intermediate Arrow structure and dealing with any redundant structure information in repetition and definition levels. This should a allow a unification of our nested data read code path so that we can read arbitrary nested structures.

Author: Wes McKinney <wes.mckinney@twosigma.com>

Closes apache#398 from wesm/PARQUET-1100 and squashes the following commits:

9ea85d9 [Wes McKinney] Revert to const args
f4dc0fe [Wes McKinney] Make parquet::schema::Node non-copyable. Use const-refs instead of const-ptr for non-nullable argument
0d859cc [Wes McKinney] Code review comments, scrubbing some flakes
1368415 [Wes McKinney] Fix more MSVC warnings
eccb84c [Wes McKinney] Give macro more accurate name
0eaada0 [Wes McKinney] Use int64_t instead of int for batch sizes
79c3709 [Wes McKinney] Add documentation. Remove RecordReader from public API
8fa619b [Wes McKinney] Initialize memory in DecodeSpaced to avoid undefined behavior
5a0c860 [Wes McKinney] Remove non-repeated branch from DelimitRecords
c754e6e [Wes McKinney] Refactor to skip record delimiting for non-repeated data
ed2a03f [Wes McKinney] Move more code into TypedRecordReader
2e934e9 [Wes McKinney] Set some integers as const
58d3a0f [Wes McKinney] Do not index into levels arrays
b766371 [Wes McKinney] Add RecordReader::Reserve to preallocate, fixing perf regression. cpplint
1bf3e8f [Wes McKinney] Refactor to create stateful parquet::RecordReader class to better support nested data. Shift value buffering logic from parquet/arrow/reader into RecordReader. Fix bug described in PARQUET-1100

Change-Id: I94c1dd3f7936e30036e9c42eef689e86fb5fe7fa
wesm added a commit to wesm/arrow that referenced this pull request Sep 7, 2018
…ed data, refactor parquet/arrow/reader

We did not have very consistent logic around reading values from leaf nodes versus reading semantic records where the repetition level is greater than zero. This introduces a reader class that reads from column chunks until it identifies the end of records. It also reads values (with spaces, if required by the schema) into internal buffers. This permitted a substantial refactoring and simplification of the code in parquet::arrow where we were handling the interpretation of batch reads as records manually.

As follow up patch, we should be able to take a collection of record readers from the same "tree" in a nested type and reassemble the intermediate Arrow structure and dealing with any redundant structure information in repetition and definition levels. This should a allow a unification of our nested data read code path so that we can read arbitrary nested structures.

Author: Wes McKinney <wes.mckinney@twosigma.com>

Closes apache#398 from wesm/PARQUET-1100 and squashes the following commits:

9ea85d9 [Wes McKinney] Revert to const args
f4dc0fe [Wes McKinney] Make parquet::schema::Node non-copyable. Use const-refs instead of const-ptr for non-nullable argument
0d859cc [Wes McKinney] Code review comments, scrubbing some flakes
1368415 [Wes McKinney] Fix more MSVC warnings
eccb84c [Wes McKinney] Give macro more accurate name
0eaada0 [Wes McKinney] Use int64_t instead of int for batch sizes
79c3709 [Wes McKinney] Add documentation. Remove RecordReader from public API
8fa619b [Wes McKinney] Initialize memory in DecodeSpaced to avoid undefined behavior
5a0c860 [Wes McKinney] Remove non-repeated branch from DelimitRecords
c754e6e [Wes McKinney] Refactor to skip record delimiting for non-repeated data
ed2a03f [Wes McKinney] Move more code into TypedRecordReader
2e934e9 [Wes McKinney] Set some integers as const
58d3a0f [Wes McKinney] Do not index into levels arrays
b766371 [Wes McKinney] Add RecordReader::Reserve to preallocate, fixing perf regression. cpplint
1bf3e8f [Wes McKinney] Refactor to create stateful parquet::RecordReader class to better support nested data. Shift value buffering logic from parquet/arrow/reader into RecordReader. Fix bug described in PARQUET-1100

Change-Id: I94c1dd3f7936e30036e9c42eef689e86fb5fe7fa
wesm added a commit to wesm/arrow that referenced this pull request Sep 8, 2018
…ed data, refactor parquet/arrow/reader

We did not have very consistent logic around reading values from leaf nodes versus reading semantic records where the repetition level is greater than zero. This introduces a reader class that reads from column chunks until it identifies the end of records. It also reads values (with spaces, if required by the schema) into internal buffers. This permitted a substantial refactoring and simplification of the code in parquet::arrow where we were handling the interpretation of batch reads as records manually.

As follow up patch, we should be able to take a collection of record readers from the same "tree" in a nested type and reassemble the intermediate Arrow structure and dealing with any redundant structure information in repetition and definition levels. This should a allow a unification of our nested data read code path so that we can read arbitrary nested structures.

Author: Wes McKinney <wes.mckinney@twosigma.com>

Closes apache#398 from wesm/PARQUET-1100 and squashes the following commits:

9ea85d9 [Wes McKinney] Revert to const args
f4dc0fe [Wes McKinney] Make parquet::schema::Node non-copyable. Use const-refs instead of const-ptr for non-nullable argument
0d859cc [Wes McKinney] Code review comments, scrubbing some flakes
1368415 [Wes McKinney] Fix more MSVC warnings
eccb84c [Wes McKinney] Give macro more accurate name
0eaada0 [Wes McKinney] Use int64_t instead of int for batch sizes
79c3709 [Wes McKinney] Add documentation. Remove RecordReader from public API
8fa619b [Wes McKinney] Initialize memory in DecodeSpaced to avoid undefined behavior
5a0c860 [Wes McKinney] Remove non-repeated branch from DelimitRecords
c754e6e [Wes McKinney] Refactor to skip record delimiting for non-repeated data
ed2a03f [Wes McKinney] Move more code into TypedRecordReader
2e934e9 [Wes McKinney] Set some integers as const
58d3a0f [Wes McKinney] Do not index into levels arrays
b766371 [Wes McKinney] Add RecordReader::Reserve to preallocate, fixing perf regression. cpplint
1bf3e8f [Wes McKinney] Refactor to create stateful parquet::RecordReader class to better support nested data. Shift value buffering logic from parquet/arrow/reader into RecordReader. Fix bug described in PARQUET-1100

Change-Id: I94c1dd3f7936e30036e9c42eef689e86fb5fe7fa
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants