Releases: apache/beam
Beam 2.41.0 release
We are happy to present the new 2.41.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.
For more information on changes in 2.41.0, check out the detailed release notes.
I/Os
- Projection Pushdown optimizer is now on by default for streaming, matching the behavior of batch pipelines since 2.38.0. If you encounter a bug with the optimizer, please file an issue and disable the optimizer using pipeline option
--experiments=disable_projection_pushdown
.
New Features / Improvements
- Previously available in Java sdk, Python sdk now also supports logging level overrides per module. (#18222).
Breaking Changes
- Projection Pushdown optimizer may break Dataflow upgrade compatibility for optimized pipelines when it removes unused fields. If you need to upgrade and encounter a compatibility issue, disable the optimizer using pipeline option
--experiments=disable_projection_pushdown
.
Deprecations
- Support for Spark 2.4.x is deprecated and will be dropped with the release of Beam 2.44.0 or soon after (Spark runner) (#22094).
- The modules amazon-web-services and
kinesis for AWS Java SDK v1 are deprecated
in favor of amazon-web-services2
and will be eventually removed after a few Beam releases (Java) (#21249).
Bugfixes
- Fixed a condition where retrying queries would yield an incorrect cursor in the Java SDK Firestore Connector (#22089).
- Fixed plumbing allowed lateness in Go SDK. It was ignoring the user set value earlier and always used to set to 0. (#22474).
Known Issues
- See a full list of open issues that affect this version.
List of Contributors
According to git shortlog, the following people contributed to the 2.41.0 release. Thank you to all contributors!
Ahmed Abualsaud
Ahmet Altay
akashorabek
Alexey Inkin
Alexey Romanenko
Anand Inguva
andoni-guzman
Andrew Pilloud
Andrey
Andy Ye
Balázs Németh
Benjamin Gonzalez
BjornPrime
Brian Hulette
bulat safiullin
bullet03
Byron Ellis
Chamikara Jayalath
Damon Douglas
Daniel Oliveira
Daniel Thevessen
Danny McCormick
David Huntsperger
Dheeraj Gharde
Etienne Chauchot
Evan Galpin
Fernando Morales
Heejong Lee
Jack McCluskey
johnjcasey
Kenneth Knowles
Ke Wu
Kiley Sok
Liam Miller-Cushon
Lucas Nogueira
Luke Cwik
MakarkinSAkvelon
Manu Zhang
Minbo Bae
Moritz Mack
Naireen Hussain
Ning Kang
Oleh Borysevych
Pablo Estrada
pablo rodriguez defino
Pranav Bhandari
Rebecca Szper
Red Daly
Reuven Lax
Ritesh Ghorse
Robert Bradshaw
Robert Burke
Ryan Thompson
Sam Whittle
Steven Niemitz
Valentyn Tymofieiev
Vincent Marquez
Vitaly Terentyev
Vlad
Vladislav Chunikhin
Yichi Zhang
Yi Hu
yirutang
Yixiao Shen
Yu Feng
v2.40.0
We are happy to present the new 2.40.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this
release.
For more information on changes in 2.40.0 check out the detailed release notes.
Highlights
- Added RunInference API, a framework agnostic transform for inference. With this release, PyTorch and Scikit-learn are supported by the transform.
See also example at apache_beam/examples/inference/pytorch_image_classification.py
I/Os
- Upgraded to Hive 3.1.3 for HCatalogIO. Users can still provide their own version of Hive. (Java) (Issue-19554).
New Features / Improvements
- Go SDK users can now use generic registration functions to optimize their DoFn execution. (BEAM-14347)
- Go SDK users may now write self-checkpointing Splittable DoFns to read from streaming sources. (BEAM-11104)
- Go SDK textio Reads have been moved to Splittable DoFns exclusively. (BEAM-14489)
- Pipeline drain support added for Go SDK has now been tested. (BEAM-11106)
- Go SDK users can now see heap usage, sideinput cache stats, and active process bundle stats in Worker Status. (BEAM-13829)
- The serialization (pickling) library for Python is dill==0.3.1.1 (BEAM-11167)
Breaking Changes
- The Go Sdk now requires a minimum version of 1.18 in order to support generics (BEAM-14347).
- synthetic.SourceConfig field types have changed to int64 from int for better compatibility with Flink's use of Logical types in Schemas (Go) (BEAM-14173)
- Default coder updated to compress sources used with
BoundedSourceAsSDFWrapperFn
andUnboundedSourceAsSDFWrapper
.
Bugfixes
- Fixed X (Java/Python) (BEAM-X).
- Fixed Java expansion service to allow specific files to stage (BEAM-14160).
- Fixed Elasticsearch connection when using both ssl and username/password (Java) (BEAM-14000)
Detailed list of PRs
- [BEAM-14048] [CdapIO] Add ConfigWrapper for building CDAP PluginConfigs by @Amar3tto in #17051
- [BEAM-14196] add test verifying output watermark propagation in bundle by @je-ik in #17504
- Move master readme.md to 2.40.0 by @y1chi in #17552
- [BEAM-14173] Fix Go Loadtests on Dataflow & partial fix for Flink by @lostluck in #17554
- Upgrade python sdk container requirements. by @y1chi in #17549
- [BEAM-11205] Update Libraries BOM dependencies to version 25.2.0 by @benWize in #17497
- [BEAM-12603] Add retry on grpc data channel and remove retry from test. by @y1chi in #17537
- [BEAM-14303] Add a way to exclude output timestamp watermark holds by @reuvenlax in #17359
- [BEAM-14347] Allow users to optimize DoFn execution with a single generic registration function by @damccorm in #17429
- [BEAM-5878] Add (failing) kwonly-argument test by @TheNeuralBit in #17509
- [BEAM-14014] Add parameter for service account impersonation in GCP credentials by @kennknowles in #17394
- [BEAM-14370] [Website] Add new page about appache beam by @bullet03 in #17490
- [BEAM-1754] Adds experimental Typescript Beam SDK by @robertwb in #17341
- [BEAM-14059] Delete tags.go by @damccorm in #17541
- [BEAM-14332] Refactored cluster management for Flink on Dataproc by @KevinGG in #17402
- [BEAM-14146] Python Streaming job failing to drain with BigQueryIO write errors by @ihji in #17566
- [BEAM-13988] Update mtime to use time.UnixMilli() calls by @jrmccluskey in #17578
- Fixing patching error on missing dependencies by @pabloem in #17564
- [BEAM-14383] Improve "FailedRows" errors returned by beam.io.WriteToBigQuery by @Firlej in #17517
- Quote pip extra package names in quickstart by @kynx in #17450
- [BEAM-14374] Fix module import error in FullyQualifiedNamedTransform by @ihji in #17482
- [BEAM-14436] Adds code reviewers for GCP I/O connectors and KafkaIO to Beam OWNERS files by @chamikaramj in #17581
- [BEAM-13666] Stuck inventory jobs should be cancelled and rescheduled for next run by @elink21 in #17582
- [BEAM-14439] [BEAM-12673] Add extra details to PubSub matcher errors by @yeandy in #17586
- [BEAM-14423] Add exception injection tests for BigtableIO read in BigtableIOTest by @Abacn in #17559
- [BEAM-11104] Allow self-checkpointing SDFs to return without finishing their restriction by @jrmccluskey in #17558
- [BEAM-14415] Exception handling tests for BQIO streaming inserts in Python by @pabloem in #17544
- BEAM-14413 add Kafka exception test cases by @johnjcasey in #17565
- [BEAM-14417] Adding exception handling tests for JdbcIO.Write by @pabloem in #17555
- [BEAM-14433] Improve Go split error message. by @lostluck in #17575
- [BEAM-14429] Force java load test on dataflow runner v2 forceNumIniti… by @y1chi in #17576
- [BEAM-14435] Adding exception handling tests for SpannerIO write transform by @pabloem in #17577
- [BEAM-14347] Add generic registration functions for iters and emitters by @damccorm in #17574
- [BEAM-14169] Add Credentials rotation cron job for clusters by @elink21 in #17383
- [BEAM-14347] Add generic registration for Combiners by @damccorm in #17579
- [BEAM-12918] TPC-DS: add Jenkins jobs by @aromanenko-dev in #15679
- [BEAM-14448] add datastore test by @johnjcasey in #17592
- [BEAM-14423] Add test cases for BigtableIO.BigtableWriterFn fails due to writeRecord by @Abacn in #17593
- [BEAM-14429] Fix SyntheticUnboundedSource data duplication with SDF wrapper by @y1chi in #17600
- [BEAM-14447] Revert "Merge pull request #17517 from [BEAM-14383] Improve "FailedRo… by @pabloem in #17601
- [BEAM-14347] Rename registration package to register by @damccorm in #17603
- [BEAM-11104] Add self-checkpointing integration test by @jrmccluskey in #17590
- [BEAM-5492] Python Dataflow integration tests should export the pipeline console output to Jenkins Test Result section by @andoni-guzman in #17530
- [BEAM-14396] Bump httplib2 upper bound. by @tvalentyn in #17602
- [BEAM-11104] Add Go self-checkpointing to CHANGES.md by @jrmccluskey in #17612
- [BEAM-14081] [CdapIO] Add context classes for CDAP plugins by @Krasavinigor in #17104
- [BEAM-12526] Add Dependabot by @damccorm in #17563
- [BEAM-14096] bump junit-quickcheck to 1.0 by @masahitojp in #17519
- Remove python 3.6 postcommit from mass_comment.py by @y1chi in #17630
- [BEAM-14347] Add some benchmarks for generic registration by @damccorm in #17613
- [BEAM-12526] Correctly route go dependency changes to go label by @damccorm in #17632
- [BEAM-13695] Add jamm jvm options to Java 11 by @kileys in #17178
- [BEAM-14334] Fix leakage of SparkContext in Spark runner tests to remove forkEvery 1 by @mosche in #17406
- Typo & link update in typescript SDK readme by @lostluck in #17633
- [BEAM-12526] Trigger go precommits on go mod/sum changes by @damccorm in #17636
- Revert "[BEAM-14429] Force java load test on dataflow runner v2 forceNumIniti…" by @y1chi in #17609
- [BEAM-14442] Add GitHub issue templates by @damccorm in #17588
- [BEAM-14347] Add generic registration feature to CHANGES by @damccorm in #17643
- Better test assertion. by @robertwb in #17551
- Bump github.com/google/go-cmp from 0.5.7 to 0.5.8 in /sdks by @dependabot in https://github.com/apache/beam/...
Beam 2.39.0 release
We are happy to present the new 2.39.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this
release.
For more information on changes in 2.39.0 check out the detailed release notes.
I/Os
- JmsIO gains the ability to map any kind of input to any subclass of
javax.jms.Message
(Java) (BEAM-16308). - JmsIO introduces the ability to write to dynamic topics (Java) (BEAM-16308).
- A
topicNameMapper
must be set to extract the topic name from the input value. - A
valueMapper
must be set to convert the input value to JMS message.
- A
- Reduce number of threads spawned by BigqueryIO StreamingInserts (
BEAM-14283). - Implemented Apache PulsarIO (BEAM-8218).
New Features / Improvements
- Support for flink scala 2.12, because most of the libraries support version 2.12 onwards. (beam-14386)
- 'Manage Clusters' JupyterLab extension added for users to configure usage of Dataproc clusters managed by Interactive Beam (Python) (BEAM-14130).
- Pipeline drain support added for Go SDK (BEAM-11106). Note: this feature is not yet fully validated and should be treated as experimental in this release.
DataFrame.unstack()
,DataFrame.pivot()
andSeries.unstack()
implemented for DataFrame API (BEAM-13948, BEAM-13966).- Support for impersonation credentials added to dataflow runner in the Java and Python SDK (BEAM-14014).
- Implemented Jupyterlab extension for managing Dataproc clusters (BEAM-14130).
- ExternalPythonTransform API added for easily invoking Python transforms from
Java (BEAM-14143). - Added Add support for Elasticsearch 8.x (BEAM-14003).
- Shard aware Kinesis record aggregation (AWS Sdk v2), (BEAM-14104).
- Upgrade to ZetaSQL 2022.04.1 (BEAM-14348).
- Fixed ReadFromBigQuery cannot be used with the interactive runner (BEAM-14112).
Breaking Changes
- Unused functions
ShallowCloneParDoPayload()
,ShallowCloneSideInput()
, andShallowCloneFunctionSpec()
have been removed from the Go SDK's pipelinex package (BEAM-13739). - JmsIO requires an explicit
valueMapper
to be set (BEAM-16308). You can use theTextMessageMapper
to convertString
inputs to JMSTestMessage
s:
JmsIO.<String>write()
.withConnectionFactory(jmsConnectionFactory)
.withValueMapper(new TextMessageMapper());
- Coders in Python are expected to inherit from Coder. (BEAM-14351).
- New abstract method
metadata()
added to io.filesystem.FileSystem in the
Python SDK. (BEAM-14314)
Deprecations
- Flink 1.11 is no longer supported (BEAM-14139).
- Python 3.6 is no longer supported (BEAM-13657).
Bugfixes
- Fixed Java Spanner IO NPE when ProjectID not specified in template executions (Java) (BEAM-14405).
- Fixed potential NPE in BigQueryServicesImpl.getErrorInfo (Java) (BEAM-14133).
Known Issues
- See a full list of open issues that affect this version.
List of Contributors
According to git shortlog, the following people contributed to the 2.39.0 release. Thank you to all contributors!
Ahmed Abualsaud,
Ahmet Altay,
Aizhamal Nurmamat kyzy,
Alexander Zhuravlev,
Alexey Romanenko,
Anand Inguva,
Andrei Gurau,
Andrew Pilloud,
Andy Ye,
Arun Pandian,
Arwin Tio,
Aydar Farrakhov,
Aydar Zainutdinov,
AydarZaynutdinov,
Balázs Németh,
Benjamin Gonzalez,
Brian Hulette,
Buqian Zheng,
Chamikara Jayalath,
Chun Yang,
Daniel Oliveira,
Daniela Martín,
Danny McCormick,
David Huntsperger,
Deepak Nagaraj,
Denise Case,
Esun Kim,
Etienne Chauchot,
Evan Galpin,
Hector Miuler Malpica Gallegos,
Heejong Lee,
Hengfeng Li,
Ilango Rajagopal,
Ilion Beyst,
Israel Herraiz,
Jack McCluskey,
Kamil Bregula,
Kamil Breguła,
Ke Wu,
Kenneth Knowles,
KevinGG,
Kiley,
Kiley Sok,
Kyle Weaver,
Liam Miller-Cushon,
Luke Cwik,
Marco Robles,
Matt Casters,
Michael Li,
MiguelAnzoWizeline,
Milan Patel,
Minbo Bae,
Moritz Mack,
Nick Caballero,
Niel Markwick,
Ning Kang,
Oskar Firlej,
Pablo Estrada,
Pavel Avilov,
Reuven Lax,
Reza Rokni,
Ritesh Ghorse,
Robert Bradshaw,
Robert Burke,
Ryan Thompson,
Sam Whittle,
Steven Niemitz,
Thiago Nunes,
Tomo Suzuki,
Valentyn Tymofieiev,
Victor,
Yi Hu,
Yichi Zhang,
Yiru Tang,
ahmedabu98,
andoni-guzman,
brachipa,
bulat safiullin,
bullet03,
dannymartinm,
daria.malkova,
dpcollins-google,
egalpin,
emily,
fbeevikm,
johnjcasey,
kileys,
msbukal@google.com,
nguyennk92,
pablo rodriguez defino,
rszper,
rvballada,
sachinag,
tvalentyn,
vachan-shetty,
yirutang
Beam 2.38.0 release
We are happy to present the new 2.38.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.
For more information on changes in 2.38.0 check out the detailed release notes.
I/Os
- Introduce projection pushdown optimizer to the Java SDK (BEAM-12976). The optimizer currently only works on the BigQuery Storage API, but more I/Os will be added in future releases. If you encounter a bug with the optimizer, please file a JIRA and disable the optimizer using pipeline option
--experiments=disable_projection_pushdown
. - A new IO for Neo4j graph databases was added. (BEAM-1857) It has the ability to update nodes and relationships using UNWIND statements and to read data using cypher statements with parameters.
amazon-web-services2
has reached feature parity and is finally recommended over the earlieramazon-web-services
andkinesis
modules (Java). These will be deprecated in one of the next releases (BEAM-13174).- Long outstanding write support for
Kinesis
was added (BEAM-13175). - Configuration was simplified and made consistent across all IOs, including the usage of
AwsOptions
(BEAM-13563, BEAM-13663, BEAM-13587). - Additionally, there's a long list of recent improvements and fixes to
S3
Filesystem (BEAM-13245, BEAM-13246, BEAM-13441, BEAM-13445, BEAM-14011),
DynamoDB
IO (BEAM-13209, BEAM-13209),
SQS
IO (BEAM-13631, BEAM-13510) and others.
- Long outstanding write support for
New Features / Improvements
- Pipeline dependencies supplied through
--requirements_file
will now be staged to the runner using binary distributions (wheels) of the PyPI packages for linux_x86_64 platform (BEAM-4032). To restore the behavior to use source distributions, set pipeline option--requirements_cache_only_sources
. To skip staging the packages at submission time, set pipeline option--requirements_cache=skip
(Python). - The Flink runner now supports Flink 1.14.x (BEAM-13106).
- Interactive Beam now supports remotely executing Flink pipelines on Dataproc (Python) (BEAM-14071).
Breaking Changes
- (Python) Previously
DoFn.infer_output_types
was expected to returnIterable[element_type]
whereelement_type
is the PCollection elemnt type. It is now expected to returnelement_type
. Take care if you have overrideninfer_output_type
in aDoFn
(this is not common). See BEAM-13860. - (
amazon-web-services2
) The types ofawsRegion
/endpoint
inAwsOptions
changed from String toRegion
/URI
(BEAM-13563).
Deprecations
- Beam 2.38.0 will be the last minor release to support Flink 1.11.
- (
amazon-web-services2
) Client providers (withXYZClientProvider()
) as well as IO specificRetryConfiguration
s are deprecated, instead usewithClientConfiguration()
orAwsOptions
to configure AWS IOs / clients.
Custom implementations of client providers shall be replaced with a respectiveClientBuilderFactory
and configured throughAwsOptions
(BEAM-13563).
Bugfixes
- Fix S3 copy for large objects (Java) (BEAM-14011)
- Fix quadratic behavior of pipeline canonicalization (Go) (BEAM-14128)
- This caused unnecessarily long pre-processing times before job submission for large complex pipelines.
- Fix
pyarrow
version parsing (Python)(BEAM-14235)
Known Issues
- See a full list of open issues that affect this version.
List of Contributors
According to git shortlog, the following people contributed to the 2.38.0 release. Thank you to all contributors!
abhijeet-lele
Ahmet Altay
akustov
Alexander
Alexander Zhuravlev
Alexey Romanenko
AlikRodriguez
Anand Inguva
andoni-guzman
andreukus
Andy Ye
Ankur Goenka
ansh0l
Artur Khanin
Aydar Farrakhov
Aydar Zainutdinov
Benjamin Gonzalez
Brian Hulette
brucearctor
bulat safiullin
bullet03
Carl Mastrangelo
Chamikara Jayalath
Chun Yang
Daniela Martín
Daniel Oliveira
Danny McCormick
daria.malkova
David Cavazos
David Huntsperger
dmitryor
Dmytro Sadovnychyi
dpcollins-google
egalpin
Elias Segundo Antonio
emily
Etienne Chauchot
Hengfeng Li
Ismaël Mejía
Israel Herraiz
Jack McCluskey
Jakub Kukul
Janek Bevendorff
Jeff Klukas
Johan Sternby
Kamil Breguła
Kenneth Knowles
Ke Wu
Kiley
Kyle Weaver
laraschmidt
Lara Schmidt
LE QUELLEC Olivier
Luka Kalinovcic
Luke Cwik
Marcin Kuthan
masahitojp
Masato Nakamura
Matt Casters
Melissa Pashniak
Michael Li
Miguel Hernandez
Moritz Mack
mosche
nancyxu123
Nathan J Mehl
Niel Markwick
Ning Kang
Pablo Estrada
paul-tlh
Pavel Avilov
Rahul Iyer
Reuven Lax
Ritesh Ghorse
Robert Bradshaw
Robert Burke
Ryan Skraba
Ryan Thompson
Sam Whittle
Seth Vargo
sp029619
Steven Niemitz
Thiago Nunes
Udi Meiri
Valentyn Tymofieiev
Victor
vitaly.terentyev
Yichi Zhang
Yi Hu
yirutang
Zachary Houfek
Zoe
Beam 2.37.0 release
We are happy to present the new 2.37.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.
For more information on changes in 2.37.0 check out the detailed release notes.
Highlights
- Java 17 support for Dataflow (BEAM-12240).
- Users using Dataflow Runner V2 may see issues with state cache due to inaccurate object sizes (BEAM-13695).
- ZetaSql is currently unsupported (issue).
- Python 3.9 support in Apache Beam (BEAM-12000).
- Dataflow support for Python 3.9 is expected to be available with 2.37.0,
but may not be fully available yet when the release is announced (BEAM-13864). - Users of Dataflow Runner V2 can run Python 3.9 pipelines with 2.37.0 release right away.
- Dataflow support for Python 3.9 is expected to be available with 2.37.0,
I/Os
- Go SDK now has wrappers for the following Cross Language Transforms from Java, along with automatic expansion service startup for each.
- JDBCIO (BEAM-13293).
- Debezium (BEAM-13761).
- BeamSQL (BEAM-13683).
- BiqQuery (BEAM-13732).
- KafkaIO now also has automatic expansion service startup. (BEAM-13821).
New Features / Improvements
- DataFrame API now supports pandas 1.4.x (BEAM-13605).
- Go SDK DoFns can now observe trigger panes directly (BEAM-13757).
Known Issues
- See a full list of open issues that affect this version.
List of Contributors
According to git shortlog, the following people contributed to the 2.37.0 release. Thank you to all contributors!
Aizhamal Nurmamat kyzy
Alexander
Alexander Chermenin
Alexandr Zhuravlev
Alexey Romanenko
Anand Inguva
andoni-guzman
andreukus
Andy Ye
Artur Khanin
Aydar Farrakhov
Aydar Zainutdinov
AydarZaynutdinov
Benjamin Gonzalez
Brian Hulette
Chamikara Jayalath
Daniel Oliveira
Danny McCormick
daria-malkova
daria.malkova
darshan-sj
David Huntsperger
dprieto91
emily
Etienne Chauchot
Fernando Morales
Heejong Lee
Ismaël Mejía
Jack McCluskey
Jan Lukavský
johnjcasey
Kamil Breguła
kellen
Kenneth Knowles
kileys
Kyle Weaver
Luke Cwik
Marcin Kuthan
Marco Robles
Matt Rudary
Miguel Hernandez
Milena Bukal
Moritz Mack
Mostafa Aghajani
Ning Kang
Pablo Estrada
Pavel Avilov
Reuven Lax
Ritesh Ghorse
Robert Bradshaw
Robert Burke
Sam Whittle
Sandy Chapman
Sergey Kalinin
Thiago Nunes
thorbjorn444
Tim Robertson
Tomo Suzuki
Valentyn Tymofieiev
Victor
Victor Chen
Vitaly Ivanov
Yichi Zhang
Beam 2.36.0 release
We are happy to present the new 2.36.0 release of Apache Beam.
This release includes both improvements and new functionality.
See the download page for this release.
For more information on changes in 2.36.0, check out the detailed release
notes.
I/Os
- Support for stopReadTime on KafkaIO SDF (Java).(BEAM-13171).
New Features / Improvements
- Added support for cloudpickle as a pickling library for Python SDK (BEAM-8123). To use cloudpickle, set pipeline option: --pickler_lib=cloudpickle
- Added option to specify triggering frequency when streaming to BigQuery (Python) (BEAM-12865).
- Added option to enable caching uploaded artifacts across job runs for Python Dataflow jobs (BEAM-13459). To enable, set pipeline option: --enable_artifact_caching, this will be enabled by default in a future release.
Breaking Changes
- Updated the jedis from 3.x to 4.x to Java RedisIO. If you are using RedisIO and using jedis directly, please refer to this page to update it. (BEAM-12092).
- Datatype of timestamp fields in
SqsMessage
for AWS IOs for SDK v2 was changed fromString
tolong
, visibility of all fields was fixed frompackage private
topublic
BEAM-13638. - Properly check output timestamps on elements output from DoFns, timers, and onWindowExpiration in Java BEAM-12931.
- Fixed a bug with DeferredDataFrame.xs when used with a non-tuple key
(BEAM-13421).
Known Issues
- Users may encounter an unexpected java.lang.ArithmeticException when outputting a timestamp
for an element further than allowedSkew from an allowed DoFN skew set to a value more than
Integer.MAX_VALUE. - See a full list of open issues that affect this version.
List of Contributors
According to git shortlog, the following people contributed to the 2.36.0 release. Thank you to all contributors!
Ada Wong
Ahmet Altay
Alexander
Alexander Dahl
Alexandr Zhuravlev
Alexey Romanenko
AlikRodriguez
Anand Inguva
Andrew Pilloud
Andy Ye
Arkadiusz Gasiński
Artur Khanin
Arun Pandian
Aydar Farrakhov
Aydar Zainutdinov
AydarZaynutdinov
Benjamin Gonzalez
Brian Hulette
Chamikara Jayalath
Daniel Collins
Daniel Oliveira
Daniel Thevessen
Daniela Martín
David Hinkes
David Huntsperger
Emily Ye
Etienne Chauchot
Evan Galpin
Heejong Lee
Ilya
Ilya Kozyrev
In-Ho Yi
Jack McCluskey
Janek Bevendorff
Jarek Potiuk
Ke Wu
KevinGG
Kyle Hersey
Kyle Weaver
Luís Bianchin
Luke Cwik
Masato Nakamura
Matthias Baetens
Mehdi Drissi
Melissa Pashniak
Michel Davit
Miguel Hernandez
MiguelAnzoWizeline
Milena Bukal
Moritz Mack
Mostafa Aghajani
Nathan J Mehl
Niel Markwick
Ning Kang
Pablo Estrada
Pavel Avilov
Quentin Sommer
Reuben van Ammers
Reuven Lax
Ritesh Ghorse
Robert Bradshaw
Robert Burke
Ryan Thompson
Sam Whittle
Sayat
Sergei Lebedev
Sergey Kalinin
Steve Niemitz
Talat Uyarer
Thiago Nunes
Tianyang Hu
Tim Robertson
Valentyn Tymofieiev
Vitaly Ivanov
Yichi Zhang
Yiru Tang
Yu Feng
Yu ISHIKAWA
Zachary Houfek
blais
daria-malkova
daria.malkova
darshan-sj
dpcollins-google
emily
ewianda
johnjcasey
kileys
lam206
laraschmidt
mosche
msbukal@google.com
tvalentyn
Beam 2.35.0 release
We are happy to present the new 2.35.0 release of Apache Beam.
This release includes both improvements and new functionality.
See the download page for this release.
For more information on changes in 2.35.0, check out the detailed release
notes.
Highlights
- MultiMap side inputs are now supported by the Go SDK (BEAM-3293).
- Side inputs are supported within Splittable DoFns for Dataflow Runner V1 and Dataflow Runner V2. (BEAM-12522).
- Upgrades Log4j version used in test suites (Apache Beam testing environment only, not for end user consumption) to 2.17.0(BEAM-13434).
Note that Apache Beam versions do not depend on the Log4j 2 dependency (log4j-core) impacted by CVE-2021-44228.
However we urge users to update direct and indirect dependencies (if any) on Log4j 2 to the latest version by updating their build configuration and redeploying impacted pipelines.
I/Os
- We changed the data type for ranges in
JdbcIO.readWithPartitions
fromint
tolong
(BEAM-13149).
This is a relatively minor breaking change, which we're implementing to improve the usability of the transform without increasing cruft.
This transform is relatively new, so we may implement other breaking changes in the future to improve its usability. - Side inputs are supported within Splittable DoFns for Dataflow Runner V1 and Dataflow Runner V2. (BEAM-12522).
New Features / Improvements
- Added custom delimiters to Python TextIO reads (BEAM-12730).
- Added escapechar parameter to Python TextIO reads (BEAM-13189).
- Splittable reading is enabled by default while reading data with ParquetIO (BEAM-12070).
- DoFn Execution Time metrics added to Go (BEAM-13001).
- Cross-bundle side input caching is now available in the Go SDK for runners that support the feature by setting the EnableSideInputCache hook (BEAM-11097).
- Upgraded the GCP Libraries BOM version to 24.0.0 and associated dependencies (BEAM-11205). For Google Cloud client library versions set by this BOM,
see this table. - Removed avro-python3 dependency in AvroIO. Fastavro has already been our Avro library of choice on Python 3. Boolean use_fastavro is left for api compatibility, but will have no effect.(BEAM-13016).
- MultiMap side inputs are now supported by the Go SDK (BEAM-3293).
- Remote packages can now be downloaded from locations supported by apache_beam.io.filesystems. The files will be downloaded on Stager and uploaded to staging location. For more information, see BEAM-11275
Breaking Changes
- A new URN convention was adopted for cross-language transforms and existing URNs were updated. This may break advanced use-cases, for example, if a custom expansion service is used to connect diffrent Beam Java and Python versions. (BEAM-12047).
- The upgrade to Calcite 1.28.0 introduces a breaking change in the SUBSTRING function in SqlTransform, when used with the Calcite dialect (BEAM-13099, CALCITE-4427).
- ListShards (with DescribeStreamSummary) is used instead of DescribeStream to list shards in Kinesis streams (AWS SDK v2). Due to this change, as mentioned in AWS documentation, for fine-grained IAM policies it is required to update them to allow calls to ListShards and DescribeStreamSummary APIs. For more information, see Controlling Access to Amazon Kinesis Data Streams (BEAM-13233).
Deprecations
- Non-splittable reading is deprecated while reading data with ParquetIO (BEAM-12070).
Bugfixes
- Properly map main input windows to side input windows by default (Go)
(BEAM-11087). - Fixed data loss when writing to DynamoDB without setting deduplication key names (Java)
(BEAM-13009). - Go SDK Examples now have types and functions registered. (Go) (BEAM-5378)
Known Issues
- Users of beam-sdks-java-io-hcatalog (and beam-sdks-java-extensions-sql-hcatalog) must take care to override the transitive log4j dependency when they add a hive dependency (BEAM-13499).
List of Contributors
According to git shortlog, the following people contributed to the 2.35.0 release. Thank you to all contributors!
Ahmet Altay
Alexandr Zhuravlev
Alexey Romanenko
AlikRodriguez
Anand Inguva
Andrew Pilloud
Ankur Goenka
Anthony Sottile
Artur Khanin
Aydar Farrakhov
Aydar Zainutdinov
Benjamin Gonzalez
brachipa
Brian Hulette
Calvin Leung
Chamikara Jayalath
Chris Gray
Damon Douglas
Daniel Collins
Daniel Oliveira
daria.malkova
darshan-sj
David Huntsperger
Dmitrii Kuzin
dpcollins-google
dprieto
egalpin
Etienne Chauchot
Eugene Nikolaiev
Fernando Morales
Hector Lagos
Heejong Lee
Ilya Kozyrev
Iñigo San Jose Visiers
Jack McCluskey
Jiayang Wu
jrhy
Kenneth Knowles
KevinGG
kileys
klmilam
Kyle Weaver
Luís Bianchin
Luke Cwik
Melissa Pashniak
Michael Luckey
Miguel Hernandez
Milena Bukal
Minbo Bae
minherz
Moritz Mack
mosche
Natalie
Ning Kang
Pablo Estrada
Pavel Avilov
Reuven Lax
Ritesh Ghorse
Robert Bradshaw
Robert Burke
Rogan Morrow
Ruslan Altynnikov
Sam Whittle
Sergey Kalinin
Slava Chernyak
Svetak Sundhar
Tianyang Hu
Tim Robertson
Tomo Suzuki
tuorhador
Udi Meiri
vachan-shetty
Valentyn Tymofieiev
Yichi Zhang
zhoufek
Beam 2.34.0 release
We are happy to present the new 2.34.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.
For more information on changes in 2.34.0, check out the detailed release
notes.
Highlights
- The Beam Java API for Calcite SqlTransform is no longer experimental (BEAM-12680).
- Python's ParDo (Map, FlatMap, etc.) transforms now suport a
with_exception_handling
option for easily ignoring bad records and implementing the dead letter pattern.
I/Os
ReadFromBigQuery
andReadAllFromBigQuery
now run queries with BATCH priority by default. Thequery_priority
parameter is introduced to the same transforms to allow configuring the query priority (Python) (BEAM-12913).- [EXPERIMENTAL] Support for BigQuery Storage Read API added to
ReadFromBigQuery
. The newly introducedmethod
parameter can be set asDIRECT_READ
to use the Storage Read API. The default isEXPORT
which invokes a BigQuery export request. (Python) (BEAM-10917). - [EXPERIMENTAL] Added
use_native_datetime
parameter toReadFromBigQuery
to configure the return type of DATETIME fields when usingReadFromBigQuery
. This parameter can only be used whenmethod = DIRECT_READ
(Python) (BEAM-10917).
New Features / Improvements
- Upgrade to Calcite 1.26.0 (BEAM-9379).
- Added a new
dataframe
extra to the Python SDK that trackspandas
versions
we've verified compatibility with. We now recommend installing Beam withpip install apache-beam[dataframe]
when you intend to use the DataFrame API
(BEAM-12906). - Add an example of deploying Python Apache Beam job with Spark Cluster
Breaking Changes
- SQL Rows are no longer flattened (BEAM-5505).
- [Go SDK] beam.TryCrossLanguage's signature now matches beam.CrossLanguage. Like other Try functions it returns an error instead of panicking. (BEAM-9918).
- BEAM-12925 was fixed. It used to silently pass incorrect null data read from JdbcIO. Pipelines affected by this will now start throwing failures instead of silently passing incorrect data.
Bugfixes
- Fixed error while writing multiple DeferredFrames to csv (Python) (BEAM-12701).
- Fixed error when importing the DataFrame API with pandas 1.0.x installed (BEAM-12945).
- Fixed top.SmallestPerKey implementation in the Go SDK (BEAM-12946).
List of Contributors
According to git shortlog, the following people contributed to the 2.34.0 release. Thank you to all contributors!
Ahmet Altay,
Aizhamal Nurmamat kyzy,
Alex Amato,
Alexander Chermenin,
Alexey Romanenko,
AlikRodriguez,
Andrew Pilloud,
Andy Xu,
Ankur Goenka,
Aydar Farrakhov,
Aydar Zainutdinov,
Aydar Zaynutdinov,
AydarZaynutdinov,
Benjamin Gonzalez,
BenWhitehead,
Brachi Packter,
Brian Hulette,
Bu Sun Kim,
Chamikara Jayalath,
Chris Gray,
Chuck Yang,
Chun Yang,
Claire McGinty,
comet,
Daniel Collins,
Daniel Oliveira,
Daniel Thevessen,
daria.malkova,
David Cavazos,
David Huntsperger,
Dmytro Kozhevin,
dpcollins-google,
Eduardo Sánchez López,
Elias Djurfeldt,
emily,
Emily Ye,
Enis Sert,
Etienne Chauchot,
Fernando Morales,
Heejong Lee,
Ihor Indyk,
Ismaël Mejía,
Israel Herraiz,
Jack McCluskey,
Jonathan Hourany,
Judah Rand,
Kenneth Knowles,
KevinGG,
Ke Wu,
kileys,
Kyle Weaver,
Luke Cwik,
masahitojp,
MiguelAnzoWizeline,
Minbo Bae,
Niels Basjes,
Ning Kang,
Pablo Estrada,
pareshsarafmdb,
Paul Féraud,
Piotr Szczepanik,
Reuven Lax,
Ritesh Ghorse,
R. Miles McCain,
Robert Bradshaw,
Robert Burke,
Rogan Morrow,
Ruwan Lambrichts,
rvballada,
Ryan Thompson,
Sam Rohde,
Sam Whittle,
Ștefan Istrate,
Steve Niemitz,
Thomas Li Fredriksen,
Tomo Suzuki,
tvalentyn,
Udi Meiri,
Vachan,
Valentyn Tymofieiev,
Vincent Marquez,
WinsonT,
Yichi Zhang,
Yifan Mai,
Yilei "Dolee" Yang,
zhoufek
Beam 2.33.0 release
We are happy to present the new 2.33.0 release of Beam.
This release includes both improvements and new functionality.
See the download page for this release.
For more information on changes in 2.33.0, check out the detailed release
notes.
Highlights
- Go SDK is no longer experimental, and is officially part of the Beam release process.
- Matching Go SDK containers are published on release.
- Batch usage is well supported, and tested on Flink, Spark, and the Python Portable Runner.
- SDK Tests are also run against Google Cloud Dataflow, but this doesn't indicate reciprocal support.
- The SDK supports Splittable DoFns, Cross Language transforms, and most Beam Model basics.
- Go Modules are now used for dependency management.
- This is a breaking change, see Breaking Changes for resolution.
- Easier path to contribute to the Go SDK, no need to set up a GO_PATH.
- Minimum Go version is now Go v1.16
- See the announcement blogpost for full information once published.
New Features / Improvements
- Projection pushdown in SchemaIO (BEAM-12609).
- Upgrade Flink runner to Flink versions 1.13.2, 1.12.5 and 1.11.4 (BEAM-10955).
Breaking Changes
- Since release 2.30.0, "The AvroCoder changes for BEAM-2303 [changed] the reader/writer from the Avro ReflectDatum* classes to the SpecificDatum* classes" (Java). This default behavior change has been reverted in this release. Use the
useReflectApi
setting to control it (BEAM-12628).
Deprecations
- Python GBK will stop supporting unbounded PCollections that have global windowing and a default trigger in Beam 2.34. This can be overriden with
--allow_unsafe_triggers
. (BEAM-9487). - Python GBK will start requiring safe triggers or the
--allow_unsafe_triggers
flag starting with Beam 2.34. (BEAM-9487).
Bugfixes
- UnsupportedOperationException when reading from BigQuery tables and converting
TableRows to Beam Rows (Java)
(BEAM-12479). - SDFBoundedSourceReader behaves much slower compared with the original behavior
of BoundedSource (Python)
(BEAM-12781). - ORDER BY column not in SELECT crashes (ZetaSQL)
(BEAM-12759).
Known Issues
- Spark 2.x users will need to update Spark's Jackson runtime dependencies (
spark.jackson.version
) to at least version 2.9.2, due to Beam updating its dependencies. - See a full list of open issues that affect this version.
- Go SDK jobs may produce "Failed to deduce Step from MonitoringInfo" messages following successful job execution. The messages are benign and don't indicate job failure. These are due to not yet handling PCollection metrics.
List of Contributors
According to git shortlog, the following people contributed to the 2.33.0 release. Thank you to all contributors!
Ahmet Altay,
Alex Amato,
Alexey Romanenko,
Andreas Bergmeier,
Andres Rodriguez,
Andrew Pilloud,
Andy Xu,
Ankur Goenka,
anthonyqzhu,
Benjamin Gonzalez,
Bhupinder Sindhwani,
Chamikara Jayalath,
Claire McGinty,
Daniel Mateus Pires,
Daniel Oliveira,
David Huntsperger,
Dylan Hercher,
emily,
Emily Ye,
Etienne Chauchot,
Eugene Nikolaiev,
Heejong Lee,
iindyk,
Iñigo San Jose Visiers,
Ismaël Mejía,
Jack McCluskey,
Jan Lukavský,
Jeff Ruane,
Jeremy Lewi,
KevinGG,
Ke Wu,
Kyle Weaver,
lostluck,
Luke Cwik,
Marwan Tammam,
masahitojp,
Mehdi Drissi,
Minbo Bae,
Ning Kang,
Pablo Estrada,
Pascal Gillet,
Pawas Chhokra,
Reuven Lax,
Ritesh Ghorse,
Robert Bradshaw,
Robert Burke,
Rodrigo Benenson,
Ryan Thompson,
Saksham Gupta,
Sam Rohde,
Sam Whittle,
Sayat,
Sayat Satybaldiyev,
Siyuan Chen,
Slava Chernyak,
Steve Niemitz,
Steven Niemitz,
tvalentyn,
Tyson Hamilton,
Udi Meiri,
vachan-shetty,
Venkatramani Rajgopal,
Yichi Zhang,
zhoufek
Beam 2.32.0 release
We are happy to present the new 2.32.0 release of Apache Beam. This release includes both improvements and new functionality.
See the download page for this release.
For more information on changes in 2.32.0, check out the
Highlights
- The Beam DataFrame
API is no
longer experimental! We've spent the time since the 2.26.0 preview
announcement
implementing the most frequently used pandas operations
(BEAM-9547), improving
documentation
and error messages,
adding
examples,
integrating DataFrames with interactive
Beam,
and of course finding and fixing
bugs.
Leaving experimental just means that we now have high confidence in the API
and recommend its use for production workloads. We will continue to improve
the API, guided by your
feedback.
I/Os
- Support for X source added (Java/Python) (BEAM-X).
- Added ability to use JdbcIO.Write.withResults without statement and preparedStatementSetter. (BEAM-12511)
- Added ability to register URI schemes to use the S3 protocol via FileIO. (BEAM-12435).
- Respect number of shards set in SnowflakeWrite batch mode. (BEAM-12715)
- Java SDK: Update Google Cloud Healthcare IO connectors from using v1beta1 to using the GA version.
New Features / Improvements
- Add support to convert Beam Schema to Avro Schema for JDBC LogicalTypes:
VARCHAR
,NVARCHAR
,LONGVARCHAR
,LONGNVARCHAR
,DATE
,TIME
(Java)(BEAM-12385). - Reading from JDBC source by partitions (Java) (BEAM-12456).
- PubsubIO can now write to a dead-letter topic after a parsing error (Java)(BEAM-12474).
- New append-only option for Elasticsearch sink (Java) BEAM-12601
Breaking Changes
- ListShards (with DescribeStreamSummary) is used instead of DescribeStream to list shards in Kinesis streams. Due to this change, as mentioned in AWS documentation, for fine-grained IAM policies it is required to update them to allow calls to ListShards and DescribeStreamSummary APIs. For more information, see Controlling Access to Amazon Kinesis Data Streams (BEAM-12225).
Deprecations
Known Issues
- Fixed race condition in RabbitMqIO causing duplicate acks (Java) (BEAM-6516))
List of Contributors
According to git shortlog, the following people contributed to the 2.32.0 release. Thank you to all contributors!
Ahmet Altay, Ajo Thomas, Alex Amato, Alexey Romanenko, Alex Koay, allenpradeep, Anant Damle, Andrew Pilloud, Ankur Goenka, Ashwin Ramaswami, Benjamin Gonzalez, BenWhitehead, Blake Williams, Boyuan Zhang, Brian Hulette, Chamikara Jayalath, Daniel Oliveira, Daniel Thevessen, daria-malkova, David Cavazos, David Huntsperger, dennisylyung, Dennis Yung, dmkozh, egalpin, emily, Esun Kim, Gabriel Melo de Paula, Harch Vardhan, Heejong Lee, heidimhurst, hoshimura, Iñigo San Jose Visiers, Ismaël Mejía, Jack McCluskey, Jan Lukavský, Justin King, Kenneth Knowles, KevinGG, Ke Wu, kileys, Kyle Weaver, Luke Cwik, Maksym Skorupskyi, masahitojp, Matthew Ouyang, Matthias Baetens, Matt Rudary, MiguelAnzoWizeline, Miguel Hernandez, Nikita Petunin, Ning Ding, Ning Kang, odidev, Pablo Estrada, Pascal Gillet, rafal.ochyra, raphael.sanamyan, Reuven Lax, Robert Bradshaw, Robert Burke, roger-mike, Ryan McDowell, Sam Rohde, Sam Whittle, Siyuan Chen, Teng Qiu, Tianzi Cai, Tobias Hermann, Tomo Suzuki, tvalentyn, Tyson Hamilton, Udi Meiri, Valentyn Tymofieiev, Vitaly Terentyev, Yichi Zhang, Yifan Mai, yoshiki.obata, Yu Feng, YuqiHuai, yzhang559, Zachary Houfek, zhoufek