Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tomnewton/test azurite flakes4 #9

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from
+19 −1
Draft

Tomnewton/test azurite flakes4 #9

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
18b2961
Add missing export that prevented any pyarrow tests from running
Tom-Newton Feb 14, 2024
d944abf
Enable pyarrow tests on sdist_test
Tom-Newton Feb 14, 2024
9f0cd0b
Enable Azure on python macos builds
Tom-Newton Feb 14, 2024
b699483
Enable azure in conda builds and install dependencies
Tom-Newton Feb 14, 2024
b8f6cb5
Install node from conda instead of apt
Tom-Newton Feb 14, 2024
40f1db1
Update comment
Tom-Newton Feb 15, 2024
e740c3a
Pin specific versions to avoid segfault
Tom-Newton Feb 17, 2024
92bd99a
Less restrictive pin and add a comment about the issue
Tom-Newton Feb 18, 2024
7ed0910
Correct comment typo
Tom-Newton Feb 18, 2024
bccc256
Only run blob emulator
Tom-Newton Feb 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/python.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,7 @@ jobs:
timeout-minutes: 60
env:
ARROW_HOME: /usr/local
ARROW_AZURE: ON
ARROW_DATASET: ON
ARROW_FLIGHT: ON
ARROW_GANDIVA: ON
Expand Down
7 changes: 7 additions & 0 deletions ci/conda_env_cpp.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@
# under the License.

aws-sdk-cpp=1.11.68
# There is a problem with the 1.11.0 conda release of azure-core-cpp https://github.com/conda-forge/admin-requests/pull/911
azure-core-cpp>=1.10.3,<1.11.0
azure-identity-cpp>=1.6.0
azure-storage-blobs-cpp>=12.10.0
azure-storage-common-cpp>=12.5.0
azure-storage-files-datalake-cpp>=12.9.0
benchmark>=1.6.0
boost-cpp>=1.68.0
brotli
Expand All @@ -34,6 +40,7 @@ libutf8proc
lz4-c
make
ninja
nodejs
orc
pkg-config
python
Expand Down
8 changes: 8 additions & 0 deletions ci/docker/conda-cpp.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,13 @@ RUN mamba install -q -y \
valgrind && \
mamba clean --all

# Ensure npm, node and azurite are on path. npm and node are required to install azurite, which will then need to
# be on the path for the tests to run.
ENV PATH=/opt/conda/envs/arrow/bin:$PATH

COPY ci/scripts/install_azurite.sh /arrow/ci/scripts/
RUN /arrow/ci/scripts/install_azurite.sh

# We want to install the GCS testbench using the same Python binary that the Conda code will use.
COPY ci/scripts/install_gcs_testbench.sh /arrow/ci/scripts
RUN /arrow/ci/scripts/install_gcs_testbench.sh default
Expand All @@ -50,6 +57,7 @@ COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
RUN /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin

ENV ARROW_ACERO=ON \
ARROW_AZURE=ON \
ARROW_BUILD_TESTS=ON \
ARROW_DATASET=ON \
ARROW_DEPENDENCY_SOURCE=CONDA \
Expand Down
1 change: 1 addition & 0 deletions ci/scripts/python_sdist_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ export PARQUET_TEST_DATA=${arrow_dir}/cpp/submodules/parquet-testing/data
export PYARROW_CMAKE_GENERATOR=${CMAKE_GENERATOR:-Ninja}
export PYARROW_BUILD_TYPE=${CMAKE_BUILD_TYPE:-debug}
export PYARROW_WITH_ACERO=${ARROW_ACERO:-ON}
export PYARROW_WITH_AZURE=${ARROW_AZURE:-OFF}
export PYARROW_WITH_S3=${ARROW_S3:-OFF}
export PYARROW_WITH_ORC=${ARROW_ORC:-OFF}
export PYARROW_WITH_CUDA=${ARROW_CUDA:-OFF}
Expand Down
1 change: 1 addition & 0 deletions ci/scripts/python_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ fi
: ${PYARROW_TEST_S3:=${ARROW_S3:-ON}}

export PYARROW_TEST_ACERO
export PYARROW_TEST_AZURE
export PYARROW_TEST_CUDA
export PYARROW_TEST_DATASET
export PYARROW_TEST_FLIGHT
Expand Down
2 changes: 1 addition & 1 deletion cpp/src/arrow/filesystem/azurefs_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ class AzuriteEnv : public AzureEnvImpl<AzuriteEnv> {
new AzuriteEnv("devstoreaccount1",
"Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/"
"K1SZFPTOtr/KBHBeksoGMGw=="));
auto exe_path = bp::search_path("azurite");
auto exe_path = bp::search_path("azurite-blob");
if (exe_path.empty()) {
return Status::Invalid("Could not find Azurite emulator.");
}
Expand Down
Loading