New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Python Direct Runner doesn't support both streaming & non streaming sources #21103
Comments
Hello. We are currently experiencing this issue as well trying to use beam.Flatten() on a historical Pcol from bigquery and a streaming Pcol from pub/sub. Has anyone found a temporary workaround? |
@jamesandreou would an in-process Flink runner work for you?
|
any update on this? |
I don't think there has been significant work on Python streaming direct runner recently. |
We are running into this issue trying to implement a slowly changing side input as seen here: https://beam.apache.org/documentation/patterns/side-inputs/ Maybe a note on that page saying it's not working with DirectRunner? Unfortunately my pipeline is not working with Flink runner |
@damccorm is working on a fix for PeriodicImpulse transform that may help with that pattern. Not sure if it will work with DirectRunner though as it has other limitations. |
@BjornPrime - when you will document direct runner streaming limitations, incorporate #21103 (comment) |
any update on this? |
This is also affecting my pipeline, snippet below: with beam.Pipeline(argv=pipeline_args) as pipeline:
send_data = (pipeline | "Read Parquet" >> beam.io.ReadFromParquet(known_args.source)
| "Write to PubSub" >> beam.io.WriteToPubSub(topic=known_args.topic)
) |
Are there any workarounds for this? Using PeriodicImpulse for updating side inputs in the DirectRunner throws this error in my streaming pipeline. |
Please see Stack Overflow discussion:
https://stackoverflow.com/questions/68125864/transform-node-appliedptransform-was-not-replaced-as-expected-error-with-the-dir
When I create a GCS source & a Pub Source and try to flatten both, there is an error because of some incompatible transformation done by the direct runner.
Code example:
Error:
The direct runner corrupts the pipeline when it rewrites the transforms.
Imported from Jira BEAM-12586. Original Jira may contain additional context.
Reported by: rodriguezc.
The text was updated successfully, but these errors were encountered: