Skip to content

Conversation

@svetakvsundhar
Copy link
Contributor

@svetakvsundhar svetakvsundhar commented Nov 18, 2023

When granting IAM permissions to roles/storage.objectUser (or higher) on Google Cloud, the error in question does not surface. Adding a line on IAM permissions in logging, to better the error message.

Note: can't share the code to prove this here since it's proprietary, but happy to demo on a call via sandbox.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@svetakvsundhar
Copy link
Contributor Author

R: @AnandInguva (since I saw you recently updated Python File IO). Feel free to reassign if preferred.

@github-actions
Copy link
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

@codecov
Copy link

codecov bot commented Nov 18, 2023

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (c2816c8) 38.38% compared to head (0b00614) 37.82%.
Report is 225 commits behind head on master.

Files Patch % Lines
sdks/python/apache_beam/io/fileio.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master   #29485      +/-   ##
==========================================
- Coverage   38.38%   37.82%   -0.56%     
==========================================
  Files         687      686       -1     
  Lines      101745   100987     -758     
==========================================
- Hits        39051    38196     -855     
- Misses      61113    61210      +97     
  Partials     1581     1581              
Flag Coverage Δ
python 29.01% <0.00%> (-0.98%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@AnandInguva AnandInguva requested a review from Abacn November 19, 2023 01:40
@svetakvsundhar svetakvsundhar marked this pull request as draft November 19, 2023 02:25
@svetakvsundhar svetakvsundhar marked this pull request as ready for review November 20, 2023 19:03
Copy link
Contributor

@Abacn Abacn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM

@Abacn
Copy link
Contributor

Abacn commented Nov 20, 2023

The actual error should be emitted from the gcs delete operation. Currently it does not (just return an error code):

final_results.append((path, error_code))

The GCS migration work long struggled in proper handling errors in batch operation. CC: @shunping

@Abacn Abacn merged commit 2fac455 into apache:master Nov 21, 2023
@svetakvsundhar svetakvsundhar deleted the fileio_logging branch November 21, 2023 14:42
riteshghorse pushed a commit to riteshghorse/beam that referenced this pull request Nov 21, 2023
* Create HealthcareUtils file with shared resources

* revert

* improve logging

* improve logging

* better error msg

* improve logging
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants