ARROW-14892: [Python][C++] GCS Bindings #12763

emkornfield · 2022-03-31T08:26:00Z

Incorporate GCS file system into python and other bug fixes.

Bugs/Other changes:

Add GCS bindings mostly based on AWS bindings in Python and associated unit tests
Tell was incorrect, it double counted when the stream was constructed with an offset.
Missed setting the define in config.cmake which means FileSystemFromUri was never tested and didn't compile this is now fixed
Refine logic for GetFileInfo with a single path to recognize prefixes followed by a slash as a directory. This allows datasets to work as expected with a toy dataset generated on local-filesystem and copied to the cloud (I believe this is typical of how other systems write to GCS as well.
Switch convention for creating directories to always end in "/" and make use of this as another indicator. From testing with a sample iceberg table it appears this is the convention used for hive-partitioning, so I assume this is common practice for other Hive related writers (i.e. what we want to support).
Fix bug introduced in a5e45ce which caused failures when a deletion occurred on a bucket (not an object in the bucket).
Ensure output streams are closed on destruction (this is consistent with S3)

emkornfield · 2022-03-31T08:27:09Z

@pitrou @kszucs this still isn't ready for review but I have it compiling locally. I was wondering if there are cross-bow or other actions I should be taking to verify packaging?

github-actions · 2022-03-31T09:11:37Z

https://issues.apache.org/jira/browse/ARROW-14892

github-actions · 2022-03-31T09:11:39Z

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

emkornfield · 2022-04-02T06:26:14Z

CC @coryan for C++ changes.

coryan · 2022-04-02T13:15:40Z

cpp/src/arrow/filesystem/gcsfs.cc

@@ -536,8 +546,7 @@ class GcsFileSystem::Impl {
                                                           gcs::ReadFromOffset offset) {
    auto stream = client_.ReadObject(path.bucket, path.object, generation, offset);
    ARROW_GCS_RETURN_NOT_OK(stream.status());
-    return std::make_shared<GcsInputStream>(std::move(stream), path, gcs::Generation(),
-                                            offset, client_);
+    return std::make_shared<GcsInputStream>(std::move(stream), path, generation, client_);


Assume offset into this function is 1000. Without the offset parameter passed to GcsInputStream its Tell() function will return 0 when you are in fact reading byte 1000. That seems like the wrong semantics to me, but maybe it is the expected behavior?

Empirically the underlying tell returns 1000. I was observing doubling of expected tell value. The python test that found this wrote N bytes then seeked to N/2 and tried reading. The reading called the FS tell which returned N which caused zeo bytes to be read

I looked at the GCS code and it does appear to keep the ReadAt offset to return for tell but I might have missed something

@emkornfield I'm curious this wasn't caught by the C++ tests. Is it possible to enhance the generic filesystem tests to cover this?

Oh, sorry for the tell case we can certainly add a test case. Filed ARROW-16226 to track this.

cpp/src/arrow/filesystem/gcsfs.cc

emkornfield · 2022-04-04T08:02:53Z

when running docker locally the python tests seem to hang forever when attempting to reach GCS test_bench which explains the timeouts, not sure if this could be some sort of config missing for docker? (all tests pass when run without docker).

This could also be a test-bench versioning issue.

It turns out this was testbench not getting installed properly into the conda env.

emkornfield · 2022-04-05T07:05:56Z

Also CC @jorisvandenbossche if you have time to look

emkornfield · 2022-04-07T16:09:30Z

@github-actions crossbow submit -g python

emkornfield · 2022-04-07T18:56:52Z

@github-actions crossbow submit -g wheel

emkornfield · 2022-04-14T19:10:34Z

Ping @pitrou to see if you have time to review. Also @rok it seems you have been looking at GCS stuff recently.

voutilad · 2022-04-15T13:25:09Z

FWIW, @emkornfield I had to revert your last commit (608b6ec) in order to get tests to work. It seems if the testbench suite doesn't properly initialize, the GcsFileSystem tests hang until the ctest timeout. (This is using archery docker build ubuntu-cpp.)

coryan · 2022-04-15T13:37:17Z

FWIW, @emkornfield I had to revert your last commit (608b6ec) in order to get tests to work. It seems if the testbench suite doesn't properly initialize,

You may be running into postmanlabs/httpbin#673 which we worked around in googleapis/storage-testbench#301 . The latest release should have these fixes.

pitrou

Thank you @emkornfield . You'll find a bunch of comments below.
Also, can you ensure you rebase on the latest git master?

pitrou · 2022-04-18T12:32:04Z

.github/workflows/cpp.yml

@@ -128,6 +128,8 @@ jobs:
      ARROW_GANDIVA: ON
      ARROW_HDFS: ON
      ARROW_HOME: /usr/local
+      # TODO(ARROW-16102): Enable this once we can figure out builds.
+      ARROW_GCS: OFF


@emkornfield Did you mean to add this? ARROW-16102 is fixed already.

yes, ARROW-16102 was fixed in between this was posted for review and it got reviewed. this will be removed in the rebase.

pitrou · 2022-04-18T12:32:40Z

ci/scripts/python_test.sh

 export PYARROW_TEST_HDFS
 export PYARROW_TEST_ORC
 export PYARROW_TEST_PARQUET
 export PYARROW_TEST_S3

+# DO NOT SUBMIT


Why is this there? Is this PR actually ready? Did you perhaps forget to push some followup changes?

This seemed like an ugly hack. I was hoping someone could point me to a better solution. I will try removing after the rebase to see if the tests now pass, as the prior install command (install_gcs_testbench.sh) has been updated slightly at HEAD

cpp/src/arrow/filesystem/gcsfs.h

pitrou · 2022-04-18T12:37:42Z

cpp/src/arrow/filesystem/gcsfs.cc

-struct GcsCredentials {
-  explicit GcsCredentials(std::shared_ptr<google::cloud::Credentials> c)
+struct GcsCredentialsHolder {
+  explicit GcsCredentialsHolder(std::shared_ptr<google::cloud::Credentials> c)


I don't think this constructor is necessary, just let C++ define it implicitly for you?

needed for std::make_shared

cpp/src/arrow/filesystem/gcsfs.cc

python/pyarrow/_gcsfs.pyx

pitrou · 2022-04-18T13:45:53Z

python/pyarrow/tests/conftest.py

+    try:
+        proc = subprocess.Popen(args, env=env)
+    except OSError:
+        pytest.skip('`gcs test bench` command cannot be located')


Why this message? Is "gcs test bench" an actual command?

most copy and past from s3, I rephrased slightly.

emkornfield · 2022-04-20T02:41:43Z

@pitrou thanks for the thorough review. I think I addressed most comments with the exception of the cleanup of directory logic, which I will try to address in a little bit (I left TODO place markers there). I've also rebased, and changed the destructor on OutputStream to close the file it is isn't closed.

emkornfield · 2022-04-20T04:13:07Z

@pitrou I addressed the TODO's I left in there for simplification. Please let me know if the code is now easier to read, there was a bunch of superfluous code. I am still not sure why testbench needs to be installed twice.

pitrou

Thanks for the update @emkornfield !

python/pyarrow/tests/conftest.py

python/pyarrow/_gcsfs.pyx

pitrou · 2022-04-20T07:33:30Z

dev/archery/archery/cli.py

@@ -183,6 +183,8 @@ def _apply_options(cmd, options):
 @click.option("--with-r", default=None, type=BOOL,
              help="Build the Arrow R extensions. This is not a CMake option, "
              "it will toggle required options")
+@click.option("--with-gcs", default=None, type=BOOL,


It seems the option is duplicated now? (I'm curious that click doesn't complain about it)

pitrou · 2022-04-20T07:35:55Z

cpp/src/arrow/filesystem/gcsfs.h


 /// Options for the GcsFileSystem implementation.
 struct ARROW_EXPORT GcsOptions {
-  std::shared_ptr<GcsCredentials> credentials;
+  GcsCredentials credentials;


Do you plan on solving this TODO here?

pitrou · 2022-04-20T07:36:41Z

cpp/src/arrow/filesystem/gcsfs.h


 /// Options for the GcsFileSystem implementation.
 struct ARROW_EXPORT GcsOptions {
-  std::shared_ptr<GcsCredentials> credentials;
+  GcsCredentials credentials;


Note a default-constructed credentials could trivially be anonymous...

pitrou · 2022-04-20T07:37:53Z

ci/scripts/python_test.sh

 export PYARROW_TEST_HDFS
 export PYARROW_TEST_ORC
 export PYARROW_TEST_PARQUET
 export PYARROW_TEST_S3

+# Without this, install_gcs_test_bench.sh doesn't seem to be put in a
+# place that the python env can find it.
+python -m pip install "https://github.com/googleapis/storage-testbench/archive/v0.16.0.tar.gz"


I left a comment about this in conftest.py below.

I changed this to check for the existence of the script and use the existing install comment. I tried multiple approaches with docker but could get the environment variable set.

pitrou · 2022-04-20T07:38:08Z

ci/scripts/python_build.sh

 export PYARROW_WITH_HDFS=${ARROW_HDFS:-ON}
 export PYARROW_WITH_ORC=${ARROW_ORC:-OFF}
 export PYARROW_WITH_PLASMA=${ARROW_PLASMA:-OFF}
 export PYARROW_WITH_PARQUET=${ARROW_PARQUET:-OFF}
 export PYARROW_WITH_PARQUET_ENCRYPTION=${PARQUET_REQUIRE_ENCRYPTION:-ON}
+export PYARROW_WITH_GCS=${ARROW_GCS:-OFF}


Seems like this one is duplicate?

pitrou · 2022-04-20T15:14:09Z

cpp/src/arrow/filesystem/gcsfs_test.cc

@@ -50,6 +50,7 @@ namespace bp = boost::process;
 namespace gc = google::cloud;


Can you add a test for FileSystemFromUri somewhere in this file?

added a test to make sure it can instantiate the file system. there are detailed test for the FromUri implementations on GCS already.

kou · 2022-04-20T21:15:40Z

.github/workflows/cpp.yml

-      # ARROW_GCS: ON
+      ARROW_GCS: OFF


Could you revert this change because ARROW_GCS is OFF by default in ci/scripts/cpp_build.sh?

kou · 2022-04-20T21:17:44Z

cpp/src/arrow/util/config.h.cmake

@@ -45,6 +45,7 @@
 #cmakedefine ARROW_IPC
 #cmakedefine ARROW_JSON

+#cmakedefine ARROW_GCS


Could you also update document for FileSystemFromUrilike https://github.com/apache/arrow/pull/12932/files#diff-82a2d9a8424dfb436344c0802dc36275cd0fb87e2e9eb6399745fba43799f7e5 ?

kou · 2022-04-20T21:21:03Z

dev/tasks/tasks.yml

@@ -431,7 +431,7 @@ tasks:
 {############################## Wheel OSX ####################################}

 # enable S3 support from macOS 10.13 so we don't need to bundle curl, crypt and ssl
-{% for macos_version, macos_codename, arrow_s3 in [("10.9", "mavericks", "OFF"),
+{% for macos_version, macos_codename, arrow_s3_gcs in [("10.9", "mavericks", "OFF"),


I have a strong opinion for this but I like adding a new variable for gcs for readability:

{% for macos_version, macos_codename, arrow_s3, arrow_gcs in [("10.9", "mavericks", "OFF", "OFF"), ("10.13", "high-sierra", "ON", "ON")] %}

Thanks!

(I had a typo: "I have a strong opinion" -> "I don't have a strong opinion")

cpp/src/arrow/filesystem/gcsfs.cc

emkornfield · 2022-04-21T02:34:24Z

@github-actions crossbow submit -g wheel

emkornfield · 2022-04-21T02:35:52Z

@pitrou and @kou I believe I addressed all feedback you provided on the last review (sorry I had to force push my branch). One sticking point seems to be GCS Test bench, CI/Crossbow is running now there might be a few other places it needs to be installed.

pitrou · 2022-06-11T08:32:37Z

@github-actions crossbow submit wheel-macos-big-sur*

github-actions · 2022-06-11T08:33:41Z

Revision: 922e4ef

Submitted crossbow builds: ursacomputing/crossbow @ actions-2176483478

Task	Status
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp310-universal2
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-big-sur-cp39-universal2

pitrou · 2022-06-11T08:44:04Z

Ok, the situation on our macOS wheel builds is a bit horrible, but it's pointless to try to improve it here, so I'll revert the last two commits.

This reverts commit 922e4ef.

This reverts commit 28158ee.

pitrou · 2022-06-11T08:46:25Z

@github-actions crossbow submit wheel-macos-*

github-actions · 2022-06-11T08:47:27Z

Revision: 0f1f1da

Submitted crossbow builds: ursacomputing/crossbow @ actions-041204a60b

Task	Status
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp310-universal2
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-big-sur-cp39-universal2
wheel-macos-high-sierra-cp310-amd64
wheel-macos-high-sierra-cp37-amd64
wheel-macos-high-sierra-cp38-amd64
wheel-macos-high-sierra-cp39-amd64
wheel-macos-mavericks-cp310-amd64
wheel-macos-mavericks-cp37-amd64
wheel-macos-mavericks-cp38-amd64
wheel-macos-mavericks-cp39-amd64

pitrou · 2022-06-11T08:52:40Z

@github-actions crossbow submit -g nightly

github-actions · 2022-06-11T08:53:56Z

Revision: 0f1f1da

Submitted crossbow builds: ursacomputing/crossbow @ actions-8c3920b06e

Task	Status
almalinux-8-amd64
almalinux-8-arm64
almalinux-9-amd64
almalinux-9-arm64
amazon-linux-2-amd64
centos-7-amd64
centos-8-stream-amd64
centos-8-stream-arm64
conan-maximum
conan-minimum
conda-clean
conda-linux-gcc-py310-arm64
conda-linux-gcc-py310-cpu
conda-linux-gcc-py310-cuda
conda-linux-gcc-py310-ppc64le
conda-linux-gcc-py37-arm64
conda-linux-gcc-py37-cpu-r40
conda-linux-gcc-py37-cpu-r41
conda-linux-gcc-py37-cuda
conda-linux-gcc-py37-ppc64le
conda-linux-gcc-py38-arm64
conda-linux-gcc-py38-cpu
conda-linux-gcc-py38-cuda
conda-linux-gcc-py38-ppc64le
conda-linux-gcc-py39-arm64
conda-linux-gcc-py39-cpu
conda-linux-gcc-py39-cuda
conda-linux-gcc-py39-ppc64le
conda-osx-arm64-clang-py310
conda-osx-arm64-clang-py38
conda-osx-arm64-clang-py39
conda-osx-clang-py310
conda-osx-clang-py37-r40
conda-osx-clang-py37-r41
conda-osx-clang-py38
conda-osx-clang-py39
conda-win-vs2017-py310
conda-win-vs2017-py37-r40
conda-win-vs2017-py37-r41
conda-win-vs2017-py38
conda-win-vs2017-py39
debian-bookworm-amd64
debian-bookworm-arm64
debian-bullseye-amd64
debian-bullseye-arm64
debian-buster-amd64
debian-buster-arm64
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
homebrew-cpp
homebrew-r-autobrew
homebrew-r-brew
java-jars
nuget
python-sdist
test-build-cpp-fuzz
test-build-vcpkg-win
test-conda-cpp
test-conda-cpp-valgrind
test-conda-python-3.10
test-conda-python-3.7
test-conda-python-3.7-hdfs-2.9.2
test-conda-python-3.7-hdfs-3.2.1
test-conda-python-3.7-kartothek-latest
test-conda-python-3.7-kartothek-master
test-conda-python-3.7-pandas-0.24
test-conda-python-3.7-pandas-latest
test-conda-python-3.7-spark-v3.1.2
test-conda-python-3.8
test-conda-python-3.8-hypothesis
test-conda-python-3.8-pandas-latest
test-conda-python-3.8-pandas-nightly
test-conda-python-3.8-spark-v3.2.0
test-conda-python-3.9
test-conda-python-3.9-dask-latest
test-conda-python-3.9-dask-master
test-conda-python-3.9-pandas-master
test-conda-python-3.9-spark-master
test-debian-10-cpp-amd64
test-debian-10-cpp-i386
test-debian-11-cpp-amd64
test-debian-11-cpp-i386
test-debian-11-go-1.16
test-debian-11-python-3
test-debian-c-glib
test-debian-ruby
test-fedora-35-cpp
test-fedora-35-python-3
test-fedora-r-clang-sanitizer
test-r-arrow-backwards-compatibility
test-r-depsource-bundled
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-valgrind
test-r-minimal-build
test-r-offline-maximal
test-r-offline-minimal
test-r-rhub-debian-gcc-devel-lto-latest
test-r-rhub-debian-gcc-release-custom-ccache
test-r-rhub-ubuntu-gcc-release-latest
test-r-rocker-r-base-latest
test-r-rstudio-r-base-4.1-opensuse153
test-r-rstudio-r-base-4.2-centos7-devtoolset-8
test-r-rstudio-r-base-4.2-focal
test-r-ubuntu-22.04
test-r-versions
test-skyhook-integration
test-ubuntu-18.04-cpp
test-ubuntu-18.04-cpp-release
test-ubuntu-18.04-cpp-static
test-ubuntu-18.04-r-sanitizer
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-14
test-ubuntu-20.04-cpp-17
test-ubuntu-20.04-cpp-bundled
test-ubuntu-20.04-cpp-thread-sanitizer
test-ubuntu-20.04-python-3
test-ubuntu-22.04-cpp
test-ubuntu-c-glib
test-ubuntu-default-docs
test-ubuntu-ruby
ubuntu-bionic-amd64
ubuntu-bionic-arm64
ubuntu-focal-amd64
ubuntu-focal-arm64
ubuntu-impish-amd64
ubuntu-impish-arm64
ubuntu-jammy-amd64
ubuntu-jammy-arm64
verify-rc-source-cpp-linux-almalinux-8-amd64
verify-rc-source-cpp-linux-conda-latest-amd64
verify-rc-source-cpp-linux-ubuntu-18.04-amd64
verify-rc-source-cpp-linux-ubuntu-20.04-amd64
verify-rc-source-cpp-linux-ubuntu-22.04-amd64
verify-rc-source-cpp-macos-amd64
verify-rc-source-cpp-macos-arm64
verify-rc-source-cpp-macos-conda-amd64
verify-rc-source-csharp-linux-almalinux-8-amd64
verify-rc-source-csharp-linux-conda-latest-amd64
verify-rc-source-csharp-linux-ubuntu-18.04-amd64
verify-rc-source-csharp-linux-ubuntu-20.04-amd64
verify-rc-source-csharp-linux-ubuntu-22.04-amd64
verify-rc-source-csharp-macos-amd64
verify-rc-source-csharp-macos-arm64
verify-rc-source-go-linux-almalinux-8-amd64
verify-rc-source-go-linux-conda-latest-amd64
verify-rc-source-go-linux-ubuntu-18.04-amd64
verify-rc-source-go-linux-ubuntu-20.04-amd64
verify-rc-source-go-linux-ubuntu-22.04-amd64
verify-rc-source-go-macos-amd64
verify-rc-source-go-macos-arm64
verify-rc-source-integration-linux-almalinux-8-amd64
verify-rc-source-integration-linux-conda-latest-amd64
verify-rc-source-integration-linux-ubuntu-18.04-amd64
verify-rc-source-integration-linux-ubuntu-20.04-amd64
verify-rc-source-integration-linux-ubuntu-22.04-amd64
verify-rc-source-integration-macos-amd64
verify-rc-source-integration-macos-arm64
verify-rc-source-integration-macos-conda-amd64
verify-rc-source-java-linux-almalinux-8-amd64
verify-rc-source-java-linux-conda-latest-amd64
verify-rc-source-java-linux-ubuntu-18.04-amd64
verify-rc-source-java-linux-ubuntu-20.04-amd64
verify-rc-source-java-linux-ubuntu-22.04-amd64
verify-rc-source-java-macos-amd64
verify-rc-source-js-linux-almalinux-8-amd64
verify-rc-source-js-linux-conda-latest-amd64
verify-rc-source-js-linux-ubuntu-18.04-amd64
verify-rc-source-js-linux-ubuntu-20.04-amd64
verify-rc-source-js-linux-ubuntu-22.04-amd64
verify-rc-source-js-macos-amd64
verify-rc-source-js-macos-arm64
verify-rc-source-python-linux-almalinux-8-amd64
verify-rc-source-python-linux-conda-latest-amd64
verify-rc-source-python-linux-ubuntu-18.04-amd64
verify-rc-source-python-linux-ubuntu-20.04-amd64
verify-rc-source-python-linux-ubuntu-22.04-amd64
verify-rc-source-python-macos-amd64
verify-rc-source-python-macos-arm64
verify-rc-source-python-macos-conda-amd64
verify-rc-source-ruby-linux-almalinux-8-amd64
verify-rc-source-ruby-linux-conda-latest-amd64
verify-rc-source-ruby-linux-ubuntu-18.04-amd64
verify-rc-source-ruby-linux-ubuntu-20.04-amd64
verify-rc-source-ruby-linux-ubuntu-22.04-amd64
verify-rc-source-ruby-macos-amd64
verify-rc-source-ruby-macos-arm64
verify-rc-source-windows
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp310-universal2
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-big-sur-cp39-universal2
wheel-macos-high-sierra-cp310-amd64
wheel-macos-high-sierra-cp37-amd64
wheel-macos-high-sierra-cp38-amd64
wheel-macos-high-sierra-cp39-amd64
wheel-macos-mavericks-cp310-amd64
wheel-macos-mavericks-cp37-amd64
wheel-macos-mavericks-cp38-amd64
wheel-macos-mavericks-cp39-amd64
wheel-manylinux2010-cp310-amd64
wheel-manylinux2010-cp37-amd64
wheel-manylinux2010-cp38-amd64
wheel-manylinux2010-cp39-amd64
wheel-manylinux2014-cp310-amd64
wheel-manylinux2014-cp310-arm64
wheel-manylinux2014-cp37-amd64
wheel-manylinux2014-cp37-arm64
wheel-manylinux2014-cp38-amd64
wheel-manylinux2014-cp38-arm64
wheel-manylinux2014-cp39-amd64
wheel-manylinux2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp37-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

kou · 2022-06-11T21:40:26Z

We need to fix wheel-*-cp37-* failures such as wheel-manylinux2014-cp37-amd64:

https://github.com/ursacomputing/crossbow/runs/6842267198?check_suite_focus=true

=================================== FAILURES ===================================
_______________ TestConvertMetadata.test_rangeindex_doesnt_warn ________________
self = <pyarrow.tests.test_pandas.TestConvertMetadata object at 0x7f9961d90d90>
    def test_rangeindex_doesnt_warn(self):
        # ARROW-5606: pandas 0.25 deprecated private _start/stop/step
        # attributes -> can be removed if support < pd 0.25 is dropped
        df = pd.DataFrame(np.random.randn(4, 2), columns=['a', 'b'])
        with pytest.warns(None) as record:
            _check_pandas_roundtrip(df, preserve_index=True)
>       assert len(record) == 0
E       assert 4 == 0
E        +  where 4 = len(WarningsChecker(record=True))
usr/local/lib/python3.7/site-packages/pyarrow/tests/test_pandas.py:229: AssertionError
_______________ TestConvertMetadata.test_multiindex_doesnt_warn ________________
self = <pyarrow.tests.test_pandas.TestConvertMetadata object at 0x7f9961d2a710>
    def test_multiindex_doesnt_warn(self):
        # ARROW-3953: pandas 0.24 rename of MultiIndex labels to codes
        columns = pd.MultiIndex.from_arrays([['one', 'two'], ['X', 'Y']])
        df = pd.DataFrame([(1, 'a'), (2, 'b'), (3, 'c')], columns=columns)
        with pytest.warns(None) as record:
            _check_pandas_roundtrip(df, preserve_index=True)
>       assert len(record) == 0
E       assert 6 == 0
E        +  where 6 = len(WarningsChecker(record=True))
usr/local/lib/python3.7/site-packages/pyarrow/tests/test_pandas.py:280: AssertionError

emkornfield · 2022-06-11T23:45:52Z

These seem unrelated to this PR? If so I can open up JIRA to track

pitrou · 2022-06-12T08:29:27Z

@kou Those seem unrelated to this PR. I need to do a last review pass and then this PR can be merged.

pitrou · 2022-06-12T08:43:52Z

@github-actions crossbow submit -g wheel--cp37-

github-actions · 2022-06-12T08:45:04Z

Invalid group(s) {'wheel-*-cp37-*'}. Must be one of {'linux-amd64', 'verify-rc', 'conda', 'nightly-packaging', 'packaging', 'vcpkg', 'linux', 'nightly', 'example', 'fuzz', 'r', 'verify-rc-source-macos', 'wheel', 'verify-rc-source', 'verify-rc-jars', 'cpp', 'integration', 'c-glib', 'ruby', 'python', 'test', 'homebrew', 'verify-rc-wheels', 'linux-arm64', 'example-cpp', 'conan', 'example-python', 'verify-rc-source-linux', 'nightly-release', 'nightly-tests', 'verify-rc-binaries'}
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/2482888639

pitrou · 2022-06-12T08:46:57Z

@github-actions crossbow submit wheel--cp37-

github-actions · 2022-06-12T08:47:59Z

Revision: b8336ef

Submitted crossbow builds: ursacomputing/crossbow @ actions-d604965942

Task	Status
wheel-macos-high-sierra-cp37-amd64
wheel-macos-mavericks-cp37-amd64
wheel-manylinux2010-cp37-amd64
wheel-manylinux2014-cp37-amd64
wheel-manylinux2014-cp37-arm64
wheel-windows-cp37-amd64

pitrou · 2022-06-12T09:50:55Z

Ok, some JIRAs will probably have to be opened for the wheel test failures.

kou · 2022-06-12T12:11:40Z

Oh, sorry.

emkornfield · 2022-06-13T05:08:34Z

@kou @pitrou thank you very much for all your help on this one.

grisaitis · 2022-06-19T22:54:33Z

for documentation, should i open a new issue? v excited about this :)

the docs source: https://github.com/apache/arrow/blob/master/docs/source/python/filesystems.rst

kou · 2022-06-20T01:37:21Z

Yes, please.
If you can work on it, please submit a pull request too.

emkornfield marked this pull request as draft March 31, 2022 08:26

github-actions bot added Component: C++ Component: Python labels Mar 31, 2022

emkornfield marked this pull request as ready for review April 2, 2022 06:23

coryan approved these changes Apr 2, 2022

View reviewed changes

emkornfield force-pushed the gcsp branch from ed28a84 to 608b6ec Compare April 5, 2022 07:05

kou mentioned this pull request Apr 7, 2022

ARROW-16102: [C++] Add support for building with system gRPC and bundled GCS #12795

Closed

pitrou self-requested a review April 13, 2022 17:00

pitrou reviewed Apr 18, 2022

View reviewed changes

emkornfield force-pushed the gcsp branch from 5952ed7 to 76f2b27 Compare April 19, 2022 05:49

pitrou reviewed Apr 20, 2022

View reviewed changes

pitrou mentioned this pull request Apr 20, 2022

ARROW-16248: [C++][FS] Actually enable GCS support in FileSystemFromUri() #12932

Closed

kou reviewed Apr 20, 2022

View reviewed changes

emkornfield force-pushed the gcsp branch from 5e6adfb to bc74b83 Compare April 21, 2022 02:29

Try to rationalize uses of arch

922e4ef

pitrou force-pushed the gcsp branch from 3dd473f to 922e4ef Compare June 11, 2022 08:32

pitrou added 2 commits June 11, 2022 10:44

Revert "Try to rationalize uses of arch"

3a7c769

This reverts commit 922e4ef.

Revert "Try to understand arch issues"

0f1f1da

This reverts commit 28158ee.

Make error more informative in warnings check

b8336ef

pitrou approved these changes Jun 12, 2022

View reviewed changes

pitrou merged commit 7b5912d into apache:master Jun 12, 2022

h-vetinari mentioned this pull request Sep 12, 2022

Back-sync features from upstream recipe conda-forge/arrow-cpp-feedstock#853

Merged

This was referenced Jun 20, 2022

[Python] Add bindings for GCS filesystem #30417

Closed

[R] Add GCS tests using testbench #32202

Closed

raulcd mentioned this pull request Apr 18, 2023

[Python] Windows wheel is built without GCS support #35193

Closed

		@@ -50,6 +50,7 @@ namespace bp = boost::process;
		namespace gc = google::cloud;

ARROW-14892: [Python][C++] GCS Bindings #12763

ARROW-14892: [Python][C++] GCS Bindings #12763

Conversation

emkornfield commented Mar 31, 2022 • edited Loading

emkornfield commented Mar 31, 2022

github-actions bot commented Mar 31, 2022

github-actions bot commented Mar 31, 2022

emkornfield commented Apr 2, 2022

Choose a reason for hiding this comment

emkornfield Apr 2, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emkornfield Apr 19, 2022 • edited Loading

Choose a reason for hiding this comment

emkornfield commented Apr 4, 2022 • edited Loading

emkornfield commented Apr 5, 2022

emkornfield commented Apr 7, 2022

emkornfield commented Apr 7, 2022

emkornfield commented Apr 14, 2022

voutilad commented Apr 15, 2022

coryan commented Apr 15, 2022

pitrou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emkornfield commented Apr 20, 2022

emkornfield commented Apr 20, 2022

pitrou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emkornfield commented Apr 21, 2022

emkornfield commented Apr 21, 2022

pitrou commented Jun 11, 2022

github-actions bot commented Jun 11, 2022

pitrou commented Jun 11, 2022

pitrou commented Jun 11, 2022

github-actions bot commented Jun 11, 2022

pitrou commented Jun 11, 2022

github-actions bot commented Jun 11, 2022

kou commented Jun 11, 2022

emkornfield commented Jun 11, 2022

pitrou commented Jun 12, 2022 • edited Loading

pitrou commented Jun 12, 2022

github-actions bot commented Jun 12, 2022

pitrou commented Jun 12, 2022

github-actions bot commented Jun 12, 2022

pitrou commented Jun 12, 2022

kou commented Jun 12, 2022

emkornfield commented Jun 13, 2022

grisaitis commented Jun 19, 2022

kou commented Jun 20, 2022

emkornfield commented Mar 31, 2022 •

edited

Loading

emkornfield Apr 2, 2022 •

edited

Loading

emkornfield Apr 19, 2022 •

edited

Loading

emkornfield commented Apr 4, 2022 •

edited

Loading

pitrou commented Jun 12, 2022 •

edited

Loading