Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

arrow-cpp: 0.16.0 -> 0.17.1 #87638

Merged
merged 3 commits into from May 20, 2020
Merged

arrow-cpp: 0.16.0 -> 0.17.1 #87638

merged 3 commits into from May 20, 2020

Conversation

@veprbl
Copy link
Member

veprbl commented May 12, 2020

  • Linux
    • arrow-cpp builds
    • python3Packages.pyarrow builds
    • python38Packages.pyarrow builds
    • pkgsi686Linux.arrow-cpp builds
    • pkgsStatic.arrow-cpp builds
  • Darwin
    • arrow-cpp builds
    • python3Packages.pyarrow builds
    • python38Packages.pyarrow builds (python38Packages.numexpr broken)
@veprbl veprbl force-pushed the veprbl:pr/arrow_cpp_0_17_0 branch from e112d56 to f149c70 May 12, 2020
@veprbl veprbl marked this pull request as ready for review May 12, 2020
@veprbl veprbl requested review from FRidh and jonringer as code owners May 12, 2020
@veprbl
Copy link
Member Author

veprbl commented May 12, 2020

@tobim Thank you for the preemptive musl fix.

@jonringer
Copy link
Contributor

jonringer commented May 12, 2020

python3Packages.awkward is now failing due to the deprecation warning, do you mind fixing that package and then I think this is good to merge

https://github.com/NixOS/nixpkgs/pull/87638
6 package failed to build:
python37Packages.awkward python37Packages.uproot python37Packages.uproot-methods python38Packages.awkward python38Packages.uproot python38Packages.uproot-methods

8 package built:
arrow-cpp python37Packages.google_cloud_bigquery python37Packages.ibis-framework python37Packages.intake python37Packages.pyarrow python38Packages.google_cloud_bigquery python38Packages.intake python38Packages.pyarrow
@veprbl
Copy link
Member Author

veprbl commented May 12, 2020

@jonringer I think I will just disable the pyarrow support as per upstream
scikit-hep/awkward-array@220bd8d
Note that pythonPackages.awkward is to be EOL and superseded by pythonPackages.awkward1 sometime in the future.

@tobim
Copy link
Contributor

tobim commented May 12, 2020

Thank you for the preemptive musl fix.

I only fixed the build, unfortunately the static build is not usable atm and requires a bit more work:
apache/arrow#6220. We could disable jemalloc support for static builds in the meantime.

@jonringer
Copy link
Contributor

jonringer commented May 12, 2020

@veprbl Maybe so. But I would like to avoid introducing regressions.

do you mind applying this diff?

diff --git a/pkgs/development/python-modules/awkward/default.nix b/pkgs/development/python-modules/awkward/default.nix
index bd933205f6a..a1cc0f4431b 100644
--- a/pkgs/development/python-modules/awkward/default.nix
+++ b/pkgs/development/python-modules/awkward/default.nix
@@ -19,7 +19,7 @@ buildPythonPackage rec {
   };

   nativeBuildInputs = [ pytestrunner ];
-  checkInputs = [ pandas pyarrow pytest h5py ];
+  checkInputs = [ pandas pytest h5py ];
   propagatedBuildInputs = [ numpy ];

   checkPhase = ''
It appears that upstream doesn't want to support this anymore:
scikit-hep/awkward-array@220bd8d
@ofborg ofborg bot requested a review from costrouc May 12, 2020
@veprbl
Copy link
Member Author

veprbl commented May 12, 2020

We could disable jemalloc support for static builds in the meantime.

Not sure if that would work. There are other non-optional vendored libraries like uriparser.

@veprbl
Copy link
Member Author

veprbl commented May 20, 2020

Apache Arrow 0.17.1 (14 May 2020)

Bug

  • ARROW-8503 - [Packaging][deb] Can't build apache-arrow-archive-keyring for RC
  • ARROW-8505 - [Release][C#] "sourcelink test" is failed by Apache.Arrow.AssemblyInfo.cs
  • ARROW-8584 - [Packaging][C++] Protobuf link error in deb builds
  • ARROW-8608 - [C++] Update vendored mpark/variant.h to latest to fix NVCC compilation issues
  • ARROW-8609 - [C++] ORC JNI bridge crashed on null arrow buffer
  • ARROW-8641 - [Python] Regression in feather: no longer supports permutation in column selection
  • ARROW-8657 - [Python][C++][Parquet] Forward compatibility issue from 0.16 to 0.17 when using version='2.0'
  • ARROW-8684 - [Python] "SystemError: Bad call flags in _PyMethodDef_RawFastCallDict" in Python 3.7.7 on macOS when using pyarrow wheel
  • ARROW-8694 - [Python][Parquet] parquet.read_schema() fails when loading wide table created from Pandas DataFrame
  • ARROW-8704 - [C++] Fix Parquet crash on invalid input (OSS-Fuzz)
  • ARROW-8706 - [C++][Parquet] Tracking JIRA for PARQUET-1857 (unencrypted INT16_MAX Parquet row group limit)
  • ARROW-8728 - [C++] Bitmap operation may cause buffer overflow
  • ARROW-8741 - [Python][Packaging] Keep VS2015 with for the windows wheels
  • ARROW-8750 - [Python] pyarrow.feather.write_feather does not default to lz4 compression if it's available
  • PARQUET-1857 - [C++][Parquet] ParquetFileReader unable to read files with more than 32767 row groups

Improvement

  • ARROW-8501 - [Packaging][RPM] Upgrade devtoolset to 8 on CentOS 6
  • ARROW-8549 - [R] Assorted post-0.17 release cleanups
  • ARROW-8699 - [R] Fix automatic r_to_py conversion
  • ARROW-8758 - [R] Updates for compatibility with dplyr 1.0
  • ARROW-8786 - [Packaging][rpm] Use bundled zstd in the CentOS 8 build

@GrahamcOfBorg build arrow-cpp python3Packages.pyarrow
@GrahamcOfBorg build python38Packages.pyarrow
@GrahamcOfBorg build pkgsi686Linux.arrow-cpp
@GrahamcOfBorg build pkgsStatic.arrow-cpp

@tobim
Copy link
Contributor

tobim commented May 20, 2020

@veprbl jemalloc is the only vendored dependency that is built in a separate archive (the others are linked into libarrow.a directly).

@veprbl
Copy link
Member Author

veprbl commented May 20, 2020

@tobim My understanding is that the problem is already present in the current 0.16.0? If that is the case, we can merge this and address the static build later.

@tobim
tobim approved these changes May 20, 2020
@veprbl veprbl changed the title arrow-cpp: 0.16.0 -> 0.17.0 arrow-cpp: 0.16.0 -> 0.17.1 May 20, 2020
@veprbl veprbl merged commit 4a302d1 into NixOS:master May 20, 2020
26 of 30 checks passed
26 of 30 checks passed
editorconfig editorconfig
Details
arrow-cpp, arrow-cpp.passthru.tests on x86_64-darwin
Details
pkgsStatic.arrow-cpp on x86_64-darwin
Details
pkgsi686Linux.arrow-cpp on x86_64-darwin
Details
python38Packages.pyarrow on x86_64-darwin
Details
arrow-cpp, python3Packages.pyarrow on aarch64-linux Failure
Details
arrow-cpp, python3Packages.pyarrow on x86_64-darwin Timed out, unknown build status
Details
pkgsi686Linux.arrow-cpp on aarch64-linux No attempt
Details
pkgsi686Linux.arrow-cpp on x86_64-linux Failure
Details
python38Packages.pyarrow on aarch64-linux Failure
Details
Evaluation Performance Report Evaluator Performance Report
Details
arrow-cpp, arrow-cpp.passthru.tests on aarch64-linux Success
Details
arrow-cpp, arrow-cpp.passthru.tests on x86_64-linux Success
Details
arrow-cpp, python3Packages.pyarrow on x86_64-linux Success
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="1fc90ff"; rev="1fc90ff164aa977ceb893371c34261209a8c220d"; } ./pkgs/t
Details
grahamcofborg-eval-lib-tests nix-build --arg pkgs import ./. {} ./lib/tests/release.nix
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="1fc90ff"; rev="1fc90ff164aa977ceb893371c34261209a8c220d"; } ./nixos/
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="1fc90ff"; rev="1fc90ff164aa977ceb893371c34261209a8c220d"; } ./nixos/
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="1fc90ff"; rev="1fc90ff164aa977ceb893371c34261209a8c220d"; } ./nixos/
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="1fc90ff"; rev="1fc90ff164aa977ceb893371c34261209a8c220d"; } ./pkgs/t
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="1fc90ff"; rev="1fc90ff164aa977ceb893371c34261209a8c220d"; } ./pkgs/t
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="1fc90ff"; rev="1fc90ff164aa977ceb893371c34261209a8c220d"; } ./pkgs/t
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
pkgsStatic.arrow-cpp on aarch64-linux Success
Details
pkgsStatic.arrow-cpp on x86_64-linux Success
Details
python38Packages.pyarrow on x86_64-linux Success
Details
@veprbl veprbl deleted the veprbl:pr/arrow_cpp_0_17_0 branch May 20, 2020
@HuwCampbell
Copy link

HuwCampbell commented Jun 9, 2020

I'm getting build failures for arrow-cpp on darwin at the moment. It looks related to this change.

CMake Error at cmake_modules/ThirdpartyToolchain.cmake:1860 (get_target_property):
  get_target_property() called with non-existent target "ZSTD::zstd".
@veprbl veprbl self-assigned this Jun 9, 2020
@veprbl
Copy link
Member Author

veprbl commented Jun 9, 2020

@HuwCampbell
0.17.1 was building on darwin at some point: https://hydra.nixos.org/build/119292492
Let's wait for apache/arrow#7388

@veprbl
Copy link
Member Author

veprbl commented Jun 15, 2020

Blocked by another regression fixed by #90218

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.