ARROW-17817: [C++] Let ORC compile on MSVC if it is activated #14208

LouisClt · 2022-09-22T14:26:48Z

No description provided.

kou · 2022-09-22T20:56:17Z

Could you enable ORC in our CI jobs?

diff --git a/.github/workflows/cpp.yml b/.github/workflows/cpp.yml
index 2642a6ec1a..07243ffc4e 100644
--- a/.github/workflows/cpp.yml
+++ b/.github/workflows/cpp.yml
@@ -213,6 +213,7 @@ jobs:
       ARROW_HOME: /usr
       ARROW_JEMALLOC: OFF
       ARROW_MIMALLOC: ON
+      ARROW_ORC: ON
       ARROW_PARQUET: ON
       ARROW_USE_GLOG: OFF
       ARROW_VERBOSE_THIRDPARTY_BUILD: OFF
diff --git a/ci/appveyor-cpp-build.bat b/ci/appveyor-cpp-build.bat
index 4b90e49257..d8eb9d2c11 100644
--- a/ci/appveyor-cpp-build.bat
+++ b/ci/appveyor-cpp-build.bat
@@ -68,6 +68,7 @@ cmake -G "%GENERATOR%" %CMAKE_ARGS% ^
       -DARROW_FLIGHT_SQL=%ARROW_BUILD_FLIGHT_SQL% ^
       -DARROW_GANDIVA=%ARROW_BUILD_GANDIVA% ^
       -DARROW_MIMALLOC=ON ^
+      -DARROW_ORC=ON ^
       -DARROW_PARQUET=ON ^
       -DARROW_PYTHON=ON ^
       -DARROW_S3=%ARROW_S3% ^

github-actions · 2022-09-22T21:30:33Z

https://issues.apache.org/jira/browse/ARROW-17817

github-actions · 2022-09-22T21:30:35Z

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

LouisClt · 2022-09-23T07:49:30Z

Hello, I made the changes you suggested. I don't know if this will compile fine as my only compile tests were with vcpkg.

kou

Thanks.

It seems that there are some warnings with Visual C++:

https://github.com/apache/arrow/actions/runs/3111153495/jobs/5043117996#step:11:1325

 D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(49): error C2220: the following warning is treated as an error
D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(49): warning C4244: 'initializing': conversion from 'double' to 'int64_t', possible loss of data
D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(49): warning C4244: 'initializing': conversion from 'double' to 'const int64_t', possible loss of data
D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(50): warning C4244: 'initializing': conversion from 'double' to 'int64_t', possible loss of data
D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(50): warning C4244: 'initializing': conversion from 'double' to 'const int64_t', possible loss of data
D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(51): warning C4244: 'initializing': conversion from 'double' to 'int64_t', possible loss of data
D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(51): warning C4244: 'initializing': conversion from 'double' to 'const int64_t', possible loss of data
D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(52): warning C4244: 'initializing': conversion from 'double' to 'int64_t', possible loss of data
D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(52): warning C4244: 'initializing': conversion from 'double' to 'const int64_t', possible loss of data
D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(53): warning C4244: 'initializing': conversion from 'double' to 'int64_t', possible loss of data
D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(53): warning C4244: 'initializing': conversion from 'double' to 'const int64_t', possible loss of data
D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(54): warning C4244: 'initializing': conversion from 'double' to 'int64_t', possible loss of data
D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(54): warning C4244: 'initializing': conversion from 'double' to 'const int64_t', possible loss of data
D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(60): error C2061: syntax error: identifier 'ssize_t'
D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(61): error C2065: 'capacity': undeclared identifier
D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(89): error C2131: expression did not evaluate to a constant
D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(89): note: failure was caused by a read of a variable outside its lifetime
D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(89): note: see usage of 'length'
D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(92): error C3863: array type 'int32_t [length]' is not assignable
D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(292): error C2664: 'arrow::MemoryOutputStream::MemoryOutputStream(const arrow::MemoryOutputStream &)': cannot convert argument 1 from 'const int' to 'const arrow::MemoryOutputStream &'
D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(292): note: Reason: cannot convert from 'const int' to 'const arrow::MemoryOutputStream'
D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(292): note: No constructor could take the source type, or constructor overload resolution was ambiguous
D:\a\arrow\arrow\cpp\src\arrow\adapters\orc\adapter_test.cc(84): note: see declaration of 'arrow::MemoryOutputStream::MemoryOutputStream'

Could you fix them? Or do you want me to fix them?

ci/appveyor-cpp-build.bat

LouisClt · 2022-09-23T19:21:30Z

I can try to fix them, at least for the easy ones. How's that these warnings did not fire before ? It is specific to Visual C++ ?

kou · 2022-09-23T19:59:30Z

I think so.

kou · 2022-09-23T20:03:38Z

Could you apply this for the AppVeyor job failure?

diff --git a/ci/conda_env_cpp.txt b/ci/conda_env_cpp.txt
index dd313f19d7..40871f560d 100644
--- a/ci/conda_env_cpp.txt
+++ b/ci/conda_env_cpp.txt
@@ -37,6 +37,7 @@ ninja
 # Required by google-cloud-cpp, the Conda package is missing the dependency:
 #    https://github.com/conda-forge/google-cloud-cpp-feedstock/issues/28
 nlohmann_json
+orc
 pkg-config
 python
 rapidjson

https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/44867881#L740

CMake Error at cmake_modules/FindORC.cmake:56 (message):
  ORC library was required in toolchain and unable to locate
Call Stack (most recent call first):
  cmake_modules/ThirdpartyToolchain.cmake:280 (find_package)
  cmake_modules/ThirdpartyToolchain.cmake:4339 (resolve_dependency)
  CMakeLists.txt:579 (include)

LouisClt · 2022-09-27T07:05:08Z

I made the changes for the AppVeyor job failure. However this still does not work because it can't find a correct version of protobuf (from looking at the logs).
There is also a problem in the "C++ / AMD64 Windows 2019 C++17" job, it seems it can't find some Zstd symbols during linking.

kou · 2022-10-13T03:58:48Z

I've fixed CI failures on AppVeyor and GitHub Actions.

I'll merge this tomorrow if nobody objects it.

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

kou · 2022-10-14T14:01:37Z

@github-actions crossbow submit java-jars

kou · 2022-10-14T21:49:59Z

@github-actions crossbow submit java-jars

github-actions · 2022-10-14T21:52:47Z

Revision: b7b3f6e

Submitted crossbow builds: ursacomputing/crossbow @ actions-e1f7effb08

Task	Status
java-jars

kou

+1

@davisusanibar FYI: This PR also enables ORC for Windows' JNI.

ursabot · 2022-10-17T15:12:29Z

Benchmark runs are scheduled for baseline = f0cf5c2 and contender = aeba616. aeba616 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.56% ⬆️0.0%] test-mac-arm
[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.61% ⬆️0.04%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] aeba6166 ec2-t3-xlarge-us-east-2
[Failed] aeba6166 test-mac-arm
[Finished] aeba6166 ursa-i9-9960x
[Finished] aeba6166 ursa-thinkcentre-m75q
[Finished] f0cf5c20 ec2-t3-xlarge-us-east-2
[Failed] f0cf5c20 test-mac-arm
[Finished] f0cf5c20 ursa-i9-9960x
[Finished] f0cf5c20 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

ursabot · 2022-10-17T15:12:44Z

['Python', 'R'] benchmarks have high level of regressions.
test-mac-arm
ursa-i9-9960x

LouisClt changed the title ~~Let ORC compile on MSVC if it is activated~~ [C++] Let ORC compile on MSVC if it is activated Sep 22, 2022

LouisClt changed the title ~~[C++] Let ORC compile on MSVC if it is activated~~ ARROW-17817: [C++] Let ORC compile on MSVC if it is activated Sep 22, 2022

pitrou requested a review from kou September 22, 2022 17:06

github-actions bot added the Component: C++ label Sep 22, 2022

kou reviewed Sep 23, 2022

View reviewed changes

ci/appveyor-cpp-build.bat Outdated Show resolved Hide resolved

kou mentioned this pull request Sep 25, 2022

ARROW-17427: [Java] Add Windows build script that produces DLLs #14203

Merged

kou force-pushed the FullORCSupportOnWindows branch from cf1b358 to c3aff64 Compare October 12, 2022 05:02

LouisClt and others added 14 commits October 14, 2022 23:00

Let ORC compile on MSVC if it is activated

4297f86

Enable ORC in CI job

0046f88

Update ci/appveyor-cpp-build.bat

7e67f72

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

Correct appveyor

175c8d6

Attemp to correct Visual C++ CI job

c677497

Fix typo

e82c7bb

Typo

8d7c64e

Use more suitable types

1413d45

Remove needless workarounds

48e5ec9

Add missing cast

504e438

Set TZDIR

3315890

Use Conda's tzdata

0cd3834

Set in build.bat

aa64f46

Add missing Zstd dependency

8b4955f

kou added 4 commits October 14, 2022 23:00

Fix bundled ORC dependencies

301071e

Simplify

b881394

Set TZDIR

2af176f

Enable ORC for jar

f5ede2c

kou force-pushed the FullORCSupportOnWindows branch from 90338ab to f5ede2c Compare October 14, 2022 14:01

This comment was marked as outdated.

Sign in to view

Set TZDIR for ORC

b7b3f6e

kou approved these changes Oct 15, 2022

View reviewed changes

kou merged commit aeba616 into apache:master Oct 15, 2022

LouisClt deleted the FullORCSupportOnWindows branch December 1, 2022 14:30

asfimport mentioned this pull request Oct 17, 2022

[CI][C++] Builds with ORC enabled fail on ubuntu 18.04 jobs #33276

Closed

h-vetinari mentioned this pull request Jan 27, 2023

Build for windows conda-forge/orc-feedstock#54

Closed

jorisvandenbossche mentioned this pull request Mar 19, 2024

[Python] BUG: Reading ORC segfaults on windows (if TZDIR isn't set) #36026

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-17817: [C++] Let ORC compile on MSVC if it is activated #14208

ARROW-17817: [C++] Let ORC compile on MSVC if it is activated #14208

LouisClt commented Sep 22, 2022 •

edited by kou

Loading

kou commented Sep 22, 2022

github-actions bot commented Sep 22, 2022

github-actions bot commented Sep 22, 2022

LouisClt commented Sep 23, 2022

kou left a comment

LouisClt commented Sep 23, 2022

kou commented Sep 23, 2022

kou commented Sep 23, 2022 •

edited

Loading

LouisClt commented Sep 27, 2022

kou commented Oct 13, 2022

kou commented Oct 14, 2022

This comment was marked as outdated.

kou commented Oct 14, 2022

github-actions bot commented Oct 14, 2022

kou left a comment

ursabot commented Oct 17, 2022

ursabot commented Oct 17, 2022

ARROW-17817: [C++] Let ORC compile on MSVC if it is activated #14208

ARROW-17817: [C++] Let ORC compile on MSVC if it is activated #14208

Conversation

LouisClt commented Sep 22, 2022 • edited by kou Loading

kou commented Sep 22, 2022

github-actions bot commented Sep 22, 2022

github-actions bot commented Sep 22, 2022

LouisClt commented Sep 23, 2022

kou left a comment

Choose a reason for hiding this comment

LouisClt commented Sep 23, 2022

kou commented Sep 23, 2022

kou commented Sep 23, 2022 • edited Loading

LouisClt commented Sep 27, 2022

kou commented Oct 13, 2022

kou commented Oct 14, 2022

This comment was marked as outdated.

kou commented Oct 14, 2022

github-actions bot commented Oct 14, 2022

kou left a comment

Choose a reason for hiding this comment

ursabot commented Oct 17, 2022

ursabot commented Oct 17, 2022

LouisClt commented Sep 22, 2022 •

edited by kou

Loading

kou commented Sep 23, 2022 •

edited

Loading