Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-13250] Factorise gcsio.GcsIO() from gcsfilesystem #15977

Closed
wants to merge 1 commit into from
Closed

[BEAM-13250] Factorise gcsio.GcsIO() from gcsfilesystem #15977

wants to merge 1 commit into from

Conversation

mlhamel
Copy link

@mlhamel mlhamel commented Nov 15, 2021

Here is a proposition of change to fix https://issues.apache.org/jira/browse/BEAM-13250

Basically I wanted to make the demonstration of an easy of factorizing gcsio.GcsIO() from gcsfilesytem to make it easier to create some kind of replacement for it to inject custom parameters like a custom url for th gcs server.

R: @aaltay @charlesccychen


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

ValidatesRunner compliance status (on master branch)

Lang ULR Dataflow Flink Samza Spark Twister2
Go --- Build Status Build Status Build Status Build Status ---
Java Build Status Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Python --- Build Status
Build Status
Build Status
Build Status
Build Status
Build Status Build Status ---
XLang Build Status Build Status Build Status Build Status Build Status ---

Examples testing status on various runners

Lang ULR Dataflow Flink Samza Spark Twister2
Go --- --- --- --- --- --- ---
Java --- Build Status
Build Status
Build Status
--- --- --- --- ---
Python --- --- --- --- --- --- ---
XLang --- --- --- --- --- --- ---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go Java Python
Build Status Build Status Build Status
Build Status
Build Status

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website Whitespace Typescript
Non-portable Build Status
Build Status
Build Status
Build Status
Build Status
Build Status Build Status Build Status Build Status
Portable --- Build Status Build Status --- --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests

See CI.md for more information about GitHub Actions CI.

@codecov
Copy link

codecov bot commented Nov 15, 2021

Codecov Report

Merging #15977 (01c6ea9) into master (fd00fc9) will decrease coverage by 0.04%.
The diff coverage is 84.61%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #15977      +/-   ##
==========================================
- Coverage   83.65%   83.60%   -0.05%     
==========================================
  Files         445      445              
  Lines       61362    61365       +3     
==========================================
- Hits        51331    51306      -25     
- Misses      10031    10059      +28     
Impacted Files Coverage Δ
sdks/python/apache_beam/io/gcp/gcsfilesystem.py 90.22% <84.61%> (+0.22%) ⬆️
sdks/python/apache_beam/utils/interactive_utils.py 87.80% <0.00%> (-7.32%) ⬇️
...hon/apache_beam/runners/direct/test_stream_impl.py 94.02% <0.00%> (-2.24%) ⬇️
...ks/python/apache_beam/runners/worker/data_plane.py 87.50% <0.00%> (-1.71%) ⬇️
sdks/python/apache_beam/io/source_test_utils.py 88.01% <0.00%> (-1.39%) ⬇️
sdks/python/apache_beam/internal/metrics/metric.py 90.00% <0.00%> (-1.00%) ⬇️
...pache_beam/runners/interactive/interactive_beam.py 75.60% <0.00%> (-0.98%) ⬇️
...runners/interactive/display/pcoll_visualization.py 85.85% <0.00%> (-0.51%) ⬇️
...hon/apache_beam/runners/worker/bundle_processor.py 93.26% <0.00%> (-0.50%) ⬇️
...nners/portability/fn_api_runner/worker_handlers.py 79.44% <0.00%> (-0.35%) ⬇️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fd00fc9...01c6ea9. Read the comment docs.

@aaltay aaltay requested a review from ihji November 30, 2021 17:18
@aaltay
Copy link
Member

aaltay commented Dec 11, 2021

@ihji - a friendly ping to review this PR.

@mlhamel
Copy link
Author

mlhamel commented Jan 18, 2022

@aaltay @ihji - Friendly ping, what do you think of the implementation. Does it make sense ?

@aaltay
Copy link
Member

aaltay commented Jan 19, 2022

@chamikaramj @ihji - Who could help with reviewing this PR?

@ihji
Copy link
Contributor

ihji commented Jan 21, 2022

Not sure about making this change if it's mostly for easier testing. Do you have any other use cases in mind?

@mlhamel
Copy link
Author

mlhamel commented Feb 10, 2022

It's mostly for testing but if you feel it's not enought for introducing that type of pattern feel free to close that PR and the corresponding issue https://issues.apache.org/jira/browse/BEAM-13250.

I tend to prefer having a predefined of injecting those dependencies but I'm sure we can find alternatives for better testing our flow.

@aaltay
Copy link
Member

aaltay commented Feb 25, 2022

Closing based on the comment thread. Feel free to re-open if you disagree.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants