Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Task]: Reconsider the need to stage Beam SDK to GCS after removing runner v1 support. #26996

Closed
2 of 15 tasks
tvalentyn opened this issue Jun 2, 2023 · 3 comments · Fixed by #27436
Closed
2 of 15 tasks
Assignees
Labels
dataflow done & done Issue has been reviewed after it was closed for verification, followups, etc. P3 python task

Comments

@tvalentyn
Copy link
Contributor

What needs to happen?

SDK now uses Beam-provided containers as execution environment on Dataflow. It may no longer be necessary to stage the SDK to staging location, or at minimum, it should not be a requirement on runner v2 pipelines.

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@tvalentyn
Copy link
Contributor Author

we could actually remove staging for runner v2 now that we are using external contianers, instead of waiting for 2.50.0

@lostluck
Copy link
Contributor

lostluck commented Aug 2, 2023

2.50 release manager here.
This issue is currently tagged for the 2.50.0 release, which cuts in a week on August 9th.

Please complete work and get it into the main branch in that time, or move this issue to the 2.51 Milestone: https://github.com/apache/beam/milestone/15

@AnandInguva
Copy link
Contributor

2.50 release manager here. This issue is currently tagged for the 2.50.0 release, which cuts in a week on August 9th.

Please complete work and get it into the main branch in that time, or move this issue to the 2.51 Milestone: https://github.com/apache/beam/milestone/15

It is almost done. I will update the issue before August 9th.

@damccorm damccorm added the done & done Issue has been reviewed after it was closed for verification, followups, etc. label Aug 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dataflow done & done Issue has been reviewed after it was closed for verification, followups, etc. P3 python task
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants