Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Python][Docs] Nightly build instructions install release version #32483

Closed
asfimport opened this issue Jul 22, 2022 · 5 comments · Fixed by #37948
Closed

[Python][Docs] Nightly build instructions install release version #32483

asfimport opened this issue Jul 22, 2022 · 5 comments · Fixed by #37948

Comments

@asfimport
Copy link

The Python installation documentation provides the following instructions to instal nightly builds of pyarrow:

Install the development version of PyArrow from [arrow-nightlies|https://anaconda.org/arrow-nightlies/pyarrow] conda channel:
conda install -c arrow-nightlies pyarrow
The result of this seems to be installation of the release version, not a nightly build:

 (test-nightlies) todd@pop-os:~/arrow/docs$ python -c "import pyarrow; pyarrow.show_versions()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pyarrow'
(test-nightlies) todd@pop-os:~/arrow/docs$ conda install -c arrow-nightlies pyarrow
Collecting package metadata (current_repodata.json): done
Solving environment: done## Package Plan ##  environment location: /home/todd/miniconda3/envs/test-nightlies  added / updated specs:
    - pyarrow
The following NEW packages will be INSTALLED:  abseil-cpp         pkgs/main/linux-64::abseil-cpp-20211102.0-hd4dd3e8_0
  arrow-cpp          pkgs/main/linux-64::arrow-cpp-8.0.0-py310h3098874_0
  aws-c-common       pkgs/main/linux-64::aws-c-common-0.4.57-he6710b0_1
  aws-c-event-stream pkgs/main/linux-64::aws-c-event-stream-0.1.6-h2531618_5
  aws-checksums      pkgs/main/linux-64::aws-checksums-0.1.9-he6710b0_0
  aws-sdk-cpp        pkgs/main/linux-64::aws-sdk-cpp-1.8.185-hce553d0_0
  blas               pkgs/main/linux-64::blas-1.0-mkl
  boost-cpp          pkgs/main/linux-64::boost-cpp-1.73.0-h7f8727e_12
  brotli             pkgs/main/linux-64::brotli-1.0.9-he6710b0_2
  c-ares             pkgs/main/linux-64::c-ares-1.18.1-h7f8727e_0
  gflags             pkgs/main/linux-64::gflags-2.2.2-he6710b0_0
  glog               pkgs/main/linux-64::glog-0.5.0-h2531618_0
  grpc-cpp           pkgs/main/linux-64::grpc-cpp-1.46.1-h33aed49_0
  icu                pkgs/main/linux-64::icu-58.2-he6710b0_3
  intel-openmp       pkgs/main/linux-64::intel-openmp-2021.4.0-h06a4308_3561
  krb5               pkgs/main/linux-64::krb5-1.19.2-hac12032_0
  libboost           pkgs/main/linux-64::libboost-1.73.0-h28710b8_12
  libcurl            pkgs/main/linux-64::libcurl-7.82.0-h0b77cf5_0
  libedit            pkgs/main/linux-64::libedit-3.1.20210910-h7f8727e_0
  libev              pkgs/main/linux-64::libev-4.33-h7f8727e_1
  libevent           pkgs/main/linux-64::libevent-2.1.12-h8f2d780_0
  libnghttp2         pkgs/main/linux-64::libnghttp2-1.46.0-hce63b2e_0
  libprotobuf        pkgs/main/linux-64::libprotobuf-3.20.1-h4ff587b_0
  libssh2            pkgs/main/linux-64::libssh2-1.10.0-h8f2d780_0
  libthrift          pkgs/main/linux-64::libthrift-0.15.0-hcc01f38_0
  lz4-c              pkgs/main/linux-64::lz4-c-1.9.3-h295c915_1
  mkl                pkgs/main/linux-64::mkl-2021.4.0-h06a4308_640
  mkl-service        pkgs/main/linux-64::mkl-service-2.4.0-py310h7f8727e_0
  mkl_fft            pkgs/main/linux-64::mkl_fft-1.3.1-py310hd6ae3a3_0
  mkl_random         pkgs/main/linux-64::mkl_random-1.2.2-py310h00e6091_0
  numpy              pkgs/main/linux-64::numpy-1.22.3-py310hfa59a62_0
  numpy-base         pkgs/main/linux-64::numpy-base-1.22.3-py310h9585f30_0
  orc                pkgs/main/linux-64::orc-1.7.4-h07ed6aa_0
  pyarrow            pkgs/main/linux-64::pyarrow-8.0.0-py310h468efa6_0
  re2                pkgs/main/linux-64::re2-2022.04.01-h295c915_0
  snappy             pkgs/main/linux-64::snappy-1.1.9-h295c915_0
  utf8proc           pkgs/main/linux-64::utf8proc-2.6.1-h27cfd23_0
  zstd               pkgs/main/linux-64::zstd-1.5.2-ha4553b6_0
Proceed ([y]/n)? yPreparing transaction: done
Verifying transaction: done
Executing transaction: done
(test-nightlies) todd@pop-os:~/arrow/docs$ python -c "import pyarrow; pyarrow.show_versions()"
pyarrow version info
--------------------
Package kind              : not indicated
Arrow C++ library version : 8.0.0   
Arrow C++ compiler        : GNU 11.2.0
Arrow C++ compiler flags  : -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/todd/miniconda3/envs/test-nightlies/include -fdebug-prefix-map=/opt/conda/conda-bld/arrow-cpp_1657131305338/work=/usr/local/src/conda/arrow-cpp-8.0.0 -fdebug-prefix-map=/home/todd/miniconda3/envs/test-nightlies=/usr/local/src/conda-prefix -fdiagnostics-color=always -O3 -DNDEBUG
Arrow C++ git revision    :         
Arrow C++ git description :         
Arrow C++ build type      : release 
(test-nightlies) todd@pop-os:~/arrow/docs$ 

The documentation should be updated to correctly guide installation of nightly builds.

Reporter: Todd Farmer / @toddfarmer

Note: This issue was originally created as ARROW-17189. Please see the migration documentation for further details.

@asfimport
Copy link
Author

Todd Farmer / @toddfarmer:
It's worth noting that the instructions on anaconda.org differ, but are not successful:

(test-nightlies) todd@pop-os:~/arrow/docs$ conda install --channel "arrow-nightlies" package
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.PackagesNotFoundError: The following packages are not available from current channels:  - packageCurrent channels:  - https://conda.anaconda.org/arrow-nightlies/linux-64
  - https://conda.anaconda.org/arrow-nightlies/noarch
  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarchTo search for alternate channels that may provide the conda package you're
looking for, navigate to    https://anaconda.organd use the search bar at the top of the page.
(test-nightlies) todd@pop-os:~/arrow/docs$  

It may be expected that experienced conda users will know how to work around this; that assumption does not apply to me. ;)

@asfimport
Copy link
Author

Kouhei Sutou / @kou:
BTW, I think that we should move the description to development documents from user documents to follow the ASF policy (It seems that "Use them at your own risk" isn't suitable for the ASF policy):

https://www.apache.org/legal/release-policy.html#publication

Projects SHALL publish official releases and SHALL NOT publish unreleased materials outside the development community.

During the process of developing software and preparing a release, various packages are made available to the development community for testing purposes. Projects MUST direct outsiders towards official releases rather than raw source repositories, nightly builds, snapshots, release candidates, or any other similar packages. Projects SHOULD make available developer resources to support individuals actively participating in development or following the dev list and thus aware of the conditions placed on unreleased materials.

@asfimport
Copy link
Author

Todd Farmer / @toddfarmer:
Thanks @kou - sounds reasonable.

Just documenting here the steps taken to make this work for me:

(test-nightlies) todd@pop-os:~/arrow/docs$ conda create -n test -c conda-forge python=3.9
Collecting package metadata (current_repodata.json): done
Solving environment: done## Package Plan ##  environment location: /home/todd/miniconda3/envs/test  added / updated specs:
    - python=3.9
The following NEW packages will be INSTALLED_libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-2_gnu
  bzip2              conda-forge/linux-64::bzip2-1.0.8-h7f98852_4
  ca-certificates    conda-forge/linux-64::ca-certificates-2022.6.15-ha878542_0
  ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.36.1-hea4e1c9_2
  libffi             conda-forge/linux-64::libffi-3.4.2-h7f98852_5
  libgcc-ng          conda-forge/linux-64::libgcc-ng-12.1.0-h8d9b700_16
  libgomp            conda-forge/linux-64::libgomp-12.1.0-h8d9b700_16
  libnsl             conda-forge/linux-64::libnsl-2.0.0-h7f98852_0
  libuuid            conda-forge/linux-64::libuuid-2.32.1-h7f98852_1000
  libzlib            conda-forge/linux-64::libzlib-1.2.12-h166bdaf_2
  ncurses            conda-forge/linux-64::ncurses-6.3-h27087fc_1
  openssl            conda-forge/linux-64::openssl-3.0.5-h166bdaf_0
  pip                conda-forge/noarch::pip-22.2-pyhd8ed1ab_0
  python             conda-forge/linux-64::python-3.9.13-h2660328_0_cpython
  python_abi         conda-forge/linux-64::python_abi-3.9-2_cp39
  readline           conda-forge/linux-64::readline-8.1.2-h0f457ee_0
  setuptools         conda-forge/linux-64::setuptools-63.2.0-py39hf3d152e_0
  sqlite             conda-forge/linux-64::sqlite-3.39.2-h4ff8645_0
  tk                 conda-forge/linux-64::tk-8.6.12-h27826a3_0
  tzdata             conda-forge/noarch::tzdata-2022a-h191b570_0
  wheel              conda-forge/noarch::wheel-0.37.1-pyhd8ed1ab_0
  xz                 conda-forge/linux-64::xz-5.2.5-h516909a_1
  zlib               conda-forge/linux-64::zlib-1.2.12-h166bdaf_2
Proceed ([y]/n)? yPreparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate test
#
# To deactivate an active environment, use
#
#     $ conda deactivate(test-nightlies) todd@pop-os:~/arrow/docs$ conda activate test
(test) todd@pop-os:~/arrow/docs$ conda install -c arrow-nightlies -c conda-forge 'pyarrow>9.0.0.dev'
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: done## Package Plan ##  environment location: /home/todd/miniconda3/envs/test  added / updated specs:
    - pyarrow[version='>9.0.0.dev']
The following NEW packages will be INSTALLEDabseil-cpp         conda-forge/linux-64::abseil-cpp-20210324.2-h9c3ff4c_0
  arrow-cpp          arrow-nightlies/linux-64::arrow-cpp-9.0.0.dev476-py39h5f82830_0_cpu
  aws-c-cal          conda-forge/linux-64::aws-c-cal-0.5.11-h95a6274_0
  aws-c-common       conda-forge/linux-64::aws-c-common-0.6.2-h7f98852_0
  aws-c-event-stream conda-forge/linux-64::aws-c-event-stream-0.2.7-h3541f99_13
  aws-c-io           conda-forge/linux-64::aws-c-io-0.10.5-hfb6a706_0
  aws-checksums      conda-forge/linux-64::aws-checksums-0.1.11-ha31a3da_7
  aws-sdk-cpp        conda-forge/linux-64::aws-sdk-cpp-1.8.186-hb4091e7_3
  c-ares             conda-forge/linux-64::c-ares-1.18.1-h7f98852_0
  gflags             conda-forge/linux-64::gflags-2.2.2-he1b5a44_1004
  glog               conda-forge/linux-64::glog-0.5.0-h48cff8f_0
  grpc-cpp           conda-forge/linux-64::grpc-cpp-1.43.2-h9e046d8_3
  keyutils           conda-forge/linux-64::keyutils-1.6.1-h166bdaf_0
  krb5               conda-forge/linux-64::krb5-1.19.3-h3790be6_0
  libblas            conda-forge/linux-64::libblas-3.9.0-15_linux64_openblas
  libbrotlicommon    conda-forge/linux-64::libbrotlicommon-1.0.9-h166bdaf_7
  libbrotlidec       conda-forge/linux-64::libbrotlidec-1.0.9-h166bdaf_7
  libbrotlienc       conda-forge/linux-64::libbrotlienc-1.0.9-h166bdaf_7
  libcblas           conda-forge/linux-64::libcblas-3.9.0-15_linux64_openblas
  libcrc32c          conda-forge/linux-64::libcrc32c-1.1.2-h9c3ff4c_0
  libcurl            conda-forge/linux-64::libcurl-7.83.1-h7bff187_0
  libedit            conda-forge/linux-64::libedit-3.1.20191231-he28a2e2_2
  libev              conda-forge/linux-64::libev-4.33-h516909a_1
  libevent           conda-forge/linux-64::libevent-2.1.10-h9b69904_4
  libgfortran-ng     conda-forge/linux-64::libgfortran-ng-12.1.0-h69a702a_16
  libgfortran5       conda-forge/linux-64::libgfortran5-12.1.0-hdcd56e2_16
  libgoogle-cloud    conda-forge/linux-64::libgoogle-cloud-1.35.0-h6945097_2
  liblapack          conda-forge/linux-64::liblapack-3.9.0-15_linux64_openblas
  libnghttp2         conda-forge/linux-64::libnghttp2-1.47.0-h727a467_0
  libopenblas        conda-forge/linux-64::libopenblas-0.3.20-pthreads_h78a6416_0
  libprotobuf        conda-forge/linux-64::libprotobuf-3.19.4-h780b84a_0
  libssh2            conda-forge/linux-64::libssh2-1.10.0-ha56f1ee_2
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-12.1.0-ha89aaad_16
  libthrift          conda-forge/linux-64::libthrift-0.15.0-he6d91bd_1
  libutf8proc        conda-forge/linux-64::libutf8proc-2.7.0-h7f98852_0
  lz4-c              conda-forge/linux-64::lz4-c-1.9.3-h9c3ff4c_1
  numpy              conda-forge/linux-64::numpy-1.23.1-py39hba7629e_0
  orc                conda-forge/linux-64::orc-1.7.3-h1be678f_0
  parquet-cpp        conda-forge/noarch::parquet-cpp-1.5.1-2
  pyarrow            arrow-nightlies/linux-64::pyarrow-9.0.0.dev476-py39hff6fa39_0_cpu
  re2                conda-forge/linux-64::re2-2022.02.01-h9c3ff4c_0
  s2n                conda-forge/linux-64::s2n-1.0.10-h9b69904_0
  snappy             conda-forge/linux-64::snappy-1.1.9-hbd366e4_1
  zstd               conda-forge/linux-64::zstd-1.5.2-h8a70e8d_2The following packages will be DOWNGRADEDopenssl                                  3.0.5-h166bdaf_0 --> 1.1.1q-h166bdaf_0
  python                          3.9.13-h2660328_0_cpython --> 3.9.13-h9a8a25e_0_cpython
Proceed ([y]/n)? yPreparing transaction: done
Verifying transaction: done
Executing transaction: done
(test) todd@pop-os:~/arrow/docs$ conda list
# packages in environment at /home/todd/miniconda3/envs/test:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
abseil-cpp                20210324.2           h9c3ff4c_0    conda-forge
arrow-cpp                 9.0.0.dev476    py39h5f82830_0_cpu    arrow-nightlies
aws-c-cal                 0.5.11               h95a6274_0    conda-forge
aws-c-common              0.6.2                h7f98852_0    conda-forge
aws-c-event-stream        0.2.7               h3541f99_13    conda-forge
aws-c-io                  0.10.5               hfb6a706_0    conda-forge
aws-checksums             0.1.11               ha31a3da_7    conda-forge
aws-sdk-cpp               1.8.186              hb4091e7_3    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2022.6.15            ha878542_0    conda-forge
gflags                    2.2.2             he1b5a44_1004    conda-forge
glog                      0.5.0                h48cff8f_0    conda-forge
grpc-cpp                  1.43.2               h9e046d8_3    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.19.3               h3790be6_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
libblas                   3.9.0           15_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h166bdaf_7    conda-forge
libbrotlidec              1.0.9                h166bdaf_7    conda-forge
libbrotlienc              1.0.9                h166bdaf_7    conda-forge
libcblas                  3.9.0           15_linux64_openblas    conda-forge
libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
libcurl                   7.83.1               h7bff187_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               h9b69904_4    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.1.0              h8d9b700_16    conda-forge
libgfortran-ng            12.1.0              h69a702a_16    conda-forge
libgfortran5              12.1.0              hdcd56e2_16    conda-forge
libgomp                   12.1.0              h8d9b700_16    conda-forge
libgoogle-cloud           1.35.0               h6945097_2    conda-forge
liblapack                 3.9.0           15_linux64_openblas    conda-forge
libnghttp2                1.47.0               h727a467_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.20          pthreads_h78a6416_0    conda-forge
libprotobuf               3.19.4               h780b84a_0    conda-forge
libssh2                   1.10.0               ha56f1ee_2    conda-forge
libstdcxx-ng              12.1.0              ha89aaad_16    conda-forge
libthrift                 0.15.0               he6d91bd_1    conda-forge
libutf8proc               2.7.0                h7f98852_0    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libzlib                   1.2.12               h166bdaf_2    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
numpy                     1.23.1           py39hba7629e_0    conda-forge
openssl                   1.1.1q               h166bdaf_0    conda-forge
orc                       1.7.3                h1be678f_0    conda-forge
parquet-cpp               1.5.1                         2    conda-forge
pip                       22.2               pyhd8ed1ab_0    conda-forge
pyarrow                   9.0.0.dev476    py39hff6fa39_0_cpu    arrow-nightlies
python                    3.9.13          h9a8a25e_0_cpython    conda-forge
python_abi                3.9                      2_cp39    conda-forge
re2                       2022.02.01           h9c3ff4c_0    conda-forge
readline                  8.1.2                h0f457ee_0    conda-forge
s2n                       1.0.10               h9b69904_0    conda-forge
setuptools                63.2.0           py39hf3d152e_0    conda-forge
snappy                    1.1.9                hbd366e4_1    conda-forge
sqlite                    3.39.2               h4ff8645_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tzdata                    2022a                h191b570_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.12               h166bdaf_2    conda-forge
zstd                      1.5.2                h8a70e8d_2    conda-forge
(test) todd@pop-os:~/arrow/docs$  

In short, I needed to include conda-forge explicitly as a channel, and I needed to specify a version filter that included .dev. I found the correct version to use via conda list-c "arrow-nightlies" (important here because the version had been bumped in nightlies).

@asfimport
Copy link
Author

Apache Arrow JIRA Bot:
This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned per project policy. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.

@jorisvandenbossche
Copy link
Member

At the moment this works fine for me (but I also have conda-forge set up as default channel, so no need for me to specify that explicitly). We can indeed update the docs to mention you need to use conda-forge).

jorisvandenbossche added a commit to jorisvandenbossche/arrow that referenced this issue Sep 29, 2023
jorisvandenbossche added a commit that referenced this issue Oct 5, 2023
…alling nightly conda package (#37948)

### Rationale for this change

Our nightly packages don't work with the defaults channel from Anaconda.

* Closes: #32483

Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
@jorisvandenbossche jorisvandenbossche added this to the 14.0.0 milestone Oct 5, 2023
JerAguilon pushed a commit to JerAguilon/arrow that referenced this issue Oct 23, 2023
…r installing nightly conda package (apache#37948)

### Rationale for this change

Our nightly packages don't work with the defaults channel from Anaconda.

* Closes: apache#32483

Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
loicalleyne pushed a commit to loicalleyne/arrow that referenced this issue Nov 13, 2023
…r installing nightly conda package (apache#37948)

### Rationale for this change

Our nightly packages don't work with the defaults channel from Anaconda.

* Closes: apache#32483

Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
dgreiss pushed a commit to dgreiss/arrow that referenced this issue Feb 19, 2024
…r installing nightly conda package (apache#37948)

### Rationale for this change

Our nightly packages don't work with the defaults channel from Anaconda.

* Closes: apache#32483

Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment