Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Python][CI] GCFS python tests fail due to updated version of google-cloud-cpp #36352

Closed
raulcd opened this issue Jun 28, 2023 · 2 comments · Fixed by #36376
Closed

[Python][CI] GCFS python tests fail due to updated version of google-cloud-cpp #36352

raulcd opened this issue Jun 28, 2023 · 2 comments · Fixed by #36376
Assignees
@raulcd
Labels
Component: Continuous Integration Component: Python Type: bug
Milestone
13.0.0

Comments

@raulcd
Copy link
Member

raulcd commented Jun 28, 2023

Describe the bug, including details regarding any error messages, version, and platform.

As discussed here:
#36119 (comment)

*originally from @raulcd *
I am unsure if there's something else to be done but it seems the nightlies for:

Failed with:

E   pyarrow.lib.ArrowInvalid: google::cloud::Status(INVALID_ARGUMENT: missing project id error_info={reason=INVALID_ARGUMENT, domain=gcloud-cpp, metadata={gcloud-cpp.source.function=CreateBucket, gcloud-cpp.source.line=402, gcloud-cpp.source.filename=/build/cpp/google_cloud_cpp_ep-install/include/google/cloud/storage/client.h, gcloud-cpp.version=v2.12.0}}). Detail: [errno 22] Invalid argument

Is there something else to be done? Should I create a new issue for those failures?

and
*originally from @raulcd *
No worries, from my understanding we have to add the new project_id to our GcsFileSystem constructor:
https://github.com/apache/arrow/blob/main/python/pyarrow/_gcsfs.pyx#L83-L89
And adding that to the underlying CGcsOptions.
We migt have to update the __reduce__ function on that file and update the tests on python/pyarrow/tests/test_fs.py:
https://github.com/apache/arrow/blob/main/python/pyarrow/tests/test_fs.py#L211
I'll create an issue

Component(s)

Continuous Integration, Python
@raulcd raulcd mentioned this issue Jun 28, 2023
Merged
@kou
Copy link
Member

kou commented Jun 28, 2023

Oh, sorry. I missed it.

@raulcd
Copy link
Member Author

raulcd commented Jun 29, 2023 •
edited
Loading

@kou does this change mean we should require a mandatory project_id when creating a GcsFileSystem?

After attempting to solve it I understood a little more about it, the question is not relevant :)

raulcd added a commit to raulcd/arrow that referenced this issue Jun 29, 2023
@raulcd
apacheGH-36352: [Python] Add project_id to GcsFileSystem options
6242dcc
@github-actions github-actions bot mentioned this issue Jun 29, 2023
Merged
raulcd added a commit that referenced this issue Jul 4, 2023
@raulcd @jorisvandenbossche
GH-36352: [Python] Add project_id to GcsFileSystem options (#36376)
cd1ed18 
### Rationale for this change
Some of our Python CI tests for GCS are failing due to the new project_id option added for GcsFileSystem here: #36228

### What changes are included in this PR?

Added option

### Are these changes tested?

Will be tested on CI.

### Are there any user-facing changes?

Yes, there is a new project_id option when defining a GcsFileSystem.
* Closes: #36352

Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
@raulcd raulcd added this to the 13.0.0 milestone Jul 4, 2023
westonpace pushed a commit to westonpace/arrow that referenced this issue Jul 7, 2023
@raulcd @jorisvandenbossche @westonpace
apacheGH-36352: [Python] Add project_id to GcsFileSystem options (apa…
61cc907 
…che#36376)

### Rationale for this change
Some of our Python CI tests for GCS are failing due to the new project_id option added for GcsFileSystem here: apache#36228

### What changes are included in this PR?

Added option

### Are these changes tested?

Will be tested on CI.

### Are there any user-facing changes?

Yes, there is a new project_id option when defining a GcsFileSystem.
* Closes: apache#36352

Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@raulcd raulcd

Labels
Component: Continuous Integration Component: Python Type: bug
Projects
None yet
Milestone
13.0.0
Development

Successfully merging a pull request may close this issue.

GH-36352: [Python] Add project_id to GcsFileSystem options raulcd/arrow
2 participants
@kou @raulcd