Minor: Add apt update after adding deadsnakes repository in dev env #16708
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tvalentyn
approved these changes
Feb 2, 2022
tushar19
pushed a commit
to twitter-forks/beam
that referenced
this pull request
Mar 18, 2022
Summary:
test commit
Merge branch 'apache:master' into master
Merge branch 'apache:master' into master
Update README with latest PreCommit Jobs
Update Postcommit jobs with latest jobs
Update Performace job tests in readme
update load job tests with latest updates
update other jobs test with latest updates
mismatch links fix
update trigger phrase for some postCommit jobs
correct trigger phrases in readme
[BEAM-12391] update avro sink to close the opened file handle, instead of getting it by reference from the writer. This ensures that when we load the python writer implementation we don't run into an attribute error
[BEAM-13482] Python fully qualified name external transforms.
Add Python expansion service entry point.
[BEAM-13509] Stop sharing SQS client between readers of same source.
[BEAM-13565][Playground] Add GetPreparationOutput method into the .proto files; Regenerate files;
update bom
[BEAM-13569] Change Spark dependencies to implementation.
They are needed to spin up a standalone Spark cluster.
remove redundant dependency
python sdk examples: Fixed typo in wordcount example.
[BEAM-13591] Bump log4j2 version to 2.17.1
[BEAM-13459] Update CHANGES.md, add note about artifact caching for python jobs
[BEAM-13587] Attempt to load AWS region from default provider chain in AwsOptions
Merge pull request #16405: [BEAM-13587] Attempt to load AWS region from default provider chain in AwsOptions
Merge pull request #16368: [BEAM-13509] Stop sharing SQS client between readers of same source.
Add Flink runner support for OrderedListState. This version reads the list entirely into memory, so will not work well for large lists.
Merge pull request #16404: [BEAM-13586] Fix NPE in DefaultS3ClientBuilderFactory
Fix sdk_container_builder too many values to unpack error
[BEAM-13480] Sickbay PubSubIntegrationTest.test_streaming_data_only on Dataflow (#16255)
remove redundant testImplementation dependencies
Merge pull request #16417 from y1chi/python_val
[BEAM-13459] Fix sdk_container_builder too many values to unpack error
[BEAM-13430] Swap to use "mainClass" instead of "main" since it was deprecated. (#16400)
See https://docs.gradle.org/7.3.2/dsl/org.gradle.api.tasks.JavaExec.html?_ga=2.124534154.1653102406.1640909413-346869268.1640099071#org.gradle.api.tasks.JavaExec:main for details
Fix remaining failing perf IT tests.
[BEAM-13430] Replace deprecated "appendix" with "archiveAppendix" (#16401)
See https://docs.gradle.org/7.3.2/dsl/org.gradle.api.tasks.bundling.AbstractArchiveTask.html?_ga=2.44668452.1653102406.1640909413-346869268.1640099071#org.gradle.api.tasks.bundling.AbstractArchiveTask:appendix for more details.
Merge pull request #16415 from deadwind4/BEAM-13591
[BEAM-13015] Add jamm as a java agent to the Java SDK harness container (#16412)
This allows for accurate object sizing for caching.
[BEAM-13430] Partially revert https://github.com/apache/beam/commit/eaacf709a861ea1c260e3856b720266348b1aecc (#16419)
Bump the version back to 2.37
Merge pull request #16246: [BEAM-12391] update avro sink to close the opened file handle directly
Merge pull request #16266 [BEAM-13482] Python fully qualified name external transforms.
[BEAM-13482] Python fully qualified name external transforms.
Merge pull request #15863 from [BEAM-13184] Autosharding for JdbcIO.write* transforms
* Supporting autosharding on JdbcIO.write transforms
* Making autosharding optional
* Adding validation
* integration test
* Reducing code duplication
* Adding a maximum bundle size to avoid overwhelming the memory
Merge pull request #15984: [BEAM-2791] OrderedListState for Flink
[BEAM-11936] Enable FloatingPointAssertionWithinEpsilon errorprone check (#16261)
[BEAM-11936] Enable LockNotBeforeTry errorprone check (#16259)
[BEAM-11936] Enable errorprone unused checks (#16262)
Add Nexmark Query 14 (#16337)
This change adds Query14.java for testing the performance of Reshuffle transform.
Co-authored-by: Arun Pandian <pandiana@google.com>
Merge pull request #16396 from ibzib/BEAM-13569
[BEAM-13569] Change Spark dependencies to implementation.
[BEAM-13015] Migrate all user state and side implementations to support caching. (#16263)
This change also ensures that prefetch can be invoked on the iterable to prevent the prefetch being lost once the iterator is discarded.
See https://s.apache.org/beam-fn-state-api-and-bundle-processing#heading=h.tms0ncgbzz6f
Merge pull request #16418 from y1chi/perf-it-2
Fix remaining failing perf IT tests.
[BEAM-13459] Update CHANGES.md, add note about artifact caching python jobs #16416
Upgrade python library versions in base_image_requirements.txt
[BEAM-13015] Use 20% of memory when the maximum has been configured. (#16420)
* [BEAM-13015] Use 20% of memory when the maximum has been configured.
The boot.go always sets the -Xmx value but we need a fallback incase used in different environments.
[BEAM-13567] Consolidate runner flag definition. (#16426)
Merge pull request #16424 from y1chi/bump_pubsub
Update python library versions in base_image_requirements.txt
[BEAM-13601] Don't cache Row types for a schema. (#16427)
This fixes apache_beam.tools.microbenchmarks_test.MicrobenchmarksTest.test_coders_microbenchmark.
Also separate out the named tuple and row cases as they have
little in common at this point.
[BEAM-13430] Re-enable checkerframework (#16429)
This fixes a bad merge conflicts that occurred on https://github.com/apache/beam/pull/16395
[BEAM-13430] Ensure that testRuntimeMigration depends on "default" configuration allowing us to get the main "jar" as a dependency. (#16430)
This better mirrors what the "shadow" and "shadowTest" configurations do and what "testRuntime" used to do with gradle 6.x and earlier.
Merge pull request #16277 from [BEAM-13124][Playground] Create readiness endpoint
* [BEAM-13124][Playground]
Implement handle func for /readiness address;
Add deleting all prepared files/folders for RunCode Api method in case of any error with the cache;
* [BEAM-13124][Playground]
renaming
* [BEAM-13124][Playground]
refactoring;
add check for numOfParallelJobs's value;
* [BEAM-13124][Playground]
merge with master;
fix staticcheck issues;
* [BEAM-13124][Playground]
rename tests
* [BEAM-13124][Playground]
update comments
Merge pull request #16314 from [BEAM-13260][Playground]Implement setup of CI test environment
* Add module to create GKE cluster via terraform, helm chart to deploy to GKE and run examples CI and GH action to run all that
* Update comments sections
Merge pull request #16383 from [BEAM-13566][Playground] Add logic of sending preparation's output on the backend side
* [BEAM-13565][Playground]
Add GetPreparationOutput method into the .proto files;
Regenerate files;
* [BEAM-13566][Playground]
Implement logic to save prepare step output
Merge pull request #16382 from [BEAM-13565][Playground] Add GetPreparationOutput API method to .proto file
[BEAM-13565][Playground] Add GetPreparationOutput API method to .proto file
Merge pull request #16365 from [BEAM-13559][Playground] Remove tag in examples CD
* [BEAM-13559][Playground]
Remove beam playground tag from examples for CI/CD steps;
Fix tests;
* [BEAM-13559][Playground]
Update test
Merge pull request #16360 from [BEAM-13546][Playground] Update nginx configuration to enable embedding iframes
* [Playground][BEAM-12941][Bugfix] Fix workflows for playground applications (#83)
* Update workflows for playground
* Attempt to fix tests
* Remove continue on error to catch errors
* Fix linter problem for backend dockerfile
* Update folder to run backend go linter
* Moved flutter test to execution via gradle tasks
* Revert "[Playground][BEAM-12941][Bugfix] Fix workflows for playground applications (#83)" (#88)
This reverts commit b73f5f70ac4184b56a0a03922731c5f2f69b9566.
* Added support for embedded iframe and also gzip compression of assets
* Add gzip_static config to nginx
Co-authored-by: Ilya <ilya.kozyrev@akvelon.com>
Co-authored-by: Aydar Zainutdinov <aydar.zaynutdinov@akvelon.com>
Co-authored-by: daria.malkova <daria.malkova@akvelon.com>
Co-authored-by: Pavel Avilov <pavel.avilov@akvelon.com>
Co-authored-by: Aydar Farrakhov <stranniknm@gmail.com>
Merge pull request #16192 from [BEAM-13395] [Playground] Tag katas
* Tag katas
* Update tags
* Update katas
* Update playground/categories.yaml
Co-authored-by: Ilya <ilya.kozyrev@akvelon.com>
* Remove the repeated lines in categories.yaml;
Edit tag;
Co-authored-by: daria.malkova <daria.malkova@akvelon.com>
Co-authored-by: Ilya <ilya.kozyrev@akvelon.com>
Merge pull request #16254 from [BEAM-13249][Playground] Security – Mock Network
* Added proxy server with allow-list using mitmproxy
* Added updated mock network dockerfile
* Updated the format of proxy env variables; added proxy for Python backend
* Added proxy for Go backend
* Add python certificat
* Stylefixes
* Remove redundant Dockerfile
* Added licenses
* Added GCS to allow-list
* Add allowed buckets
* Applied Python style and changed comments
* change name of lists
* change name of lists
* change name of the public bucket
Co-authored-by: Sergey Kalinin <sergey.kalinin@akvelon.com>
Co-authored-by: daria-malkova <daria.malkova@akvelon.com>
Merge pull request #16377: [BEAM-13576] update bom
[BEAM-12879] Prevented missing permission from failing GCS I/O
1. Errors caused by metrics gathering no longer fail the GCS I/O.
2. Added a debug level log about potential permission issue and missing project id label for reported metrics.
3. Added a test for the failing execution route.
Merge pull request #16347: fix: move connector to use v1 BigQuery Storage Write API
[BEAM-12879] Prevented missing permission from failing GCS I/O
[BEAM-13603] Fix bug in apache_beam.utils.Shared (#16437)
Co-authored-by: Ahmet Altay <aaltay@gmail.com>
[BEAM-10345] Add an import guard to support recent google-cloud-spanner versions. (#16434)
* Add an import guard to support recent google-cloud-python versions.
* Ignore when library is not available
[BEAM-13091] Generate missing staged names from hash for Dataflow runner
add test
[BEAM-13604] NPE while getting null from BigDecimal column
Fixed empty labels treated as wildcard when matching cache files (#16440)
1. Fixed a test watching nested dictionaries by watching the
concerned PCollections directly;
2. Fixed cache manager's "exists" implementations to avoid treating
empty labels as wildcard when matching cache files. This bug
is not a security issue but could cause unexpected behavior when
getting materialized values of PCollections that are not cached.
[BEAM-13570] Remove erroneous compileClasspath dependency. (#16438)
* [BEAM-13570] Remove erroneous compileClasspath dependency.
* Exclude unsupported test categories.
[BEAM-13015] Plumb through process wide and bundle cache through the FnApiStateAccessor. (#16423)
[BEAM-13015] Cache the state backed iterable used for large GBK results. (#16409)
Merge pull request #16442: [BEAM-13604] NPE while getting null from BigDecimal column
Fix formatting/alignment (#16443)
Merge pull request #16183 from [BEAM-13427] [Playground] show logs for precompiled objects
* [BEAM-13411][Playground]
Add method to get logs of precompiledObjects
* [BEAM-13427] show logs for precompiled objects
* [BEAM-13427] add support for dots in pipeline optons
* [BEAM-13427]: revert go file
* [BEAM-13427]: fix build
* [BEAM-13427] playground refactoring
* [BEAM-13427] playground print logs errors
Co-authored-by: AydarZaynutdinov <aydar.zaynutdinov@akvelon.com>
[BEAM-10277] re-write encoding position tests to declare schema protos explicitly (#16267)
Update local_env_tests.yml (#16444)
[BEAM-13574] Filesystem abstraction Rename support (#16428)
[BEAM-13597] Setup Go in github actions (#16446)
[BEAM-13602] Prevented metrics gathering from failing bigtable io
1. bigtable mutate row might return None instead of Status when running
into retryable errors. Handled the special case as DEADLINE_EXCEEDED
because bigtable must have exhausted the retry timeout or when there
is no retry policy inplace.
2. For all rows mutated, if the response status is not ok, that data
might have lost. Added a TODO item for that.
Merge pull request #15765 from ihji/BEAM-13091
[BEAM-13091] Generate missing staged names from hash for Dataflow runner
Merge pull request #16161 from [BEAM-12164] Add Spanner Partition Metadata DAOs
* [BEAM-12164] Add Spanner Partition Metadata DAOs
* fix: remove metrics table from DAO
* fix: fix compilation error
* chore: fix linting violations
* feat: add opencensus dependency
* deps: update OpenCensus API to 0.30.0
Merge pull request #16203 from [BEAM-12164] Add Spanner Change Stream Mappers
* [BEAM-12164] Add Spanner Partition Metadata DAOs
* fix: remove metrics table from DAO
* fix: fix compilation error
* chore: fix linting violations
* feat: add opencensus dependency
* [BEAM-12164] Add Spanner Change Stream Mappers
The mapper classes convert from Cloud Spanner Structs to the change
stream models used by the connector. There are two mappers implemented:
1. For mapping to partition metadata models.
2. For mapping to change stream records (one of heartbeat, data or child
partitions).
* deps: update OpenCensus API to 0.30.0
[adhoc] Fix BigTableIO description
Merge pull request #16453: [adhoc] Fix BigTableIO description
[BEAM-8727] Bump software.amazon.awssdk to 2.17.106
make the code more pythonic
[BEAM-13015] Remove dead code now that all instances have migrated to the state caching implementation. (#16447)
[BEAM-13386] Add RLock support for cloudpickle (#16250)
Fix overflow
Merge pull request #16433 from danthev/py_overflow_fix
[BEAM-13599] Fix overflow error in Python Datastore RampupThrottlingFn
don't close streams in finalize
Loosen typing extensions bound
Merge pull request #16456: [BEAM-8727] Bump software.amazon.awssdk to 2.17.106
[BEAM-4868] Bump com.amazonaws to 1.12.135
[BEAM-13243][BEAM-8374] Add support for missing PublishResponse fields in SnsIO.Write (AWS SDK v2)
Merge pull request #16454: [BEAM-4868] Bump com.amazonaws to 1.12.135
[BEAM-12092] Bump jedis to version 4.0.1 (#16287)
Merge pull request #16464: [BEAM-13617] don't close streams in finalize
[BEAM-13534] Add automated port polling to expansion service runner if port isn't provided (#16470)
Merge pull request #16344 from [BEAM-13536][Playground][Bugfix] CI step doesn't log case with empty category value
* [BEAM-13536][Playground]
Add processing of empty value in Beam tag validation method
* [BEAM-13536][Playground]
Add comments
Merge pull request #16359 from [BEAM-13545][Playground] Add GetValidationOutput API method to .proto file
* [BEAM-13545][Playground]
Add GetValidationOutput method into the .proto files;
Regenerate files;
* [BEAM-13545][Playground]
Regenerating proto files
Merge pull request #16384 from [BEAM-13308] [Playground] Getting baseFileFolder from environment
* Getting baseFileFolder from environment
* Replace 2 args to filepath.join on some methods
* Refactoring code
Merge pull request #16306 from [BEAM-13447] [Playground] Add filling of the chosen thumbs up/down button
* [BEAM-13428] add google analytics to the playground
* [BEAM-13428]: print analytics error on console
* BEAM-13252] Feedback Thumbs Up/Down Functionality
* [BEAM-13252] Deleted unnecessary comment
* [BEAM-13447]: playground - add filling of the chosen thumbs up/down button
* [BEAM-13447] playground - dont save feedback state on persitent storage
* [BEAM-13447] update pubspec
Co-authored-by: Alexander <alexander.zhuravlev@akvelon.com>
Merge pull request #16361 from [BEAM-13543][Playground] Add logic of sending validation's output on the backend side
* [BEAM-13545][Playground]
Add GetValidationOutput method into the .proto files;
Regenerate files;
* [BEAM-13543][Playground]
Implement logic to save validation output
* [BEAM-13543][Playground]
minor refactoring
* [BEAM-13543][Playground]
change back from string to []byte
* [BEAM-13545][Playground]
Regenerating proto files
[BEAM-12562] Dataframe pipe implementation (#16256)
* [BEAM-12565] Series implementation of compare
* [BEAM-12565] DataFrame implementation of compare
* [BEAM-12565] Fixed minor issues and error checks
* [BEAM-12562] Initial pipe implementation for DataFrame
* [BEAM-12562] Pipe implementation for Series
* [BEAM-12562] Pipe implementation for GroupBy
* [BEAM-12562] Added tests for Series and GroupBy pipe
* [BEAM-12562] Support for (callable, data_keyword) func in DataFrame, Series, and GroupBy
Co-authored-by: Brian Hulette <bhulette@google.com>
Merge pull request #16338 from [BEAM-13528][Playground] Add liveness check
* [BEAM-13528][Playground]
add liveness endpoint
* [BEAM-13528][Playground]
update comment
[BEAM-13626] Remap expanded outputs after merging. (#16471)
Merge pull request #16147 from [BEAM-13359] [Playground] Tag existing examples
* Tag examples
* Refactoring tags
* Add multifile to tag
* Updated categories of some examples
* Edit tags
* Edit tags
* Update tags
* fix spotless check
* Update tags
* MinimalWordCount.java and DistinctExample.java reads only one file
* Fix checks
* Change examples which read * from bucket to read one file
* Update tags;
Co-authored-by: Artur Khanin <artur.khanin@akvelon.com>
Co-authored-by: daria-malkova <daria.malkova@akvelon.com>
Co-authored-by: Pavel Avilov <pavel.avilov>
[BEAM-3221] Improve documentation in model pipeline protos (#16474)
* clarify state caching tokens
* fix documentation around side input types and valid request types.
[BEAM-13614] Add OnWindowExpiration support to the Java SDK harness and proto translation. (#16458)
This implementation adds a timer family spec in the event time domain and adds the field to the ParDoPayload mentioning which timer family spec represents the on window expiration callback.
Optional args and kwargs for named external transforms.
Merge pull request #16156 from [BEAM-13391] Fix temporary file format in WriteToBigQuery
* Fix temporary file format in WriteToBigQuery
* Change a desription
* Fix pylint issue
* Import BigQueryDisposition class
* Combine both tets together
* Fix lint issues
Co-authored-by: Sayat Satybaldiyev <sayat.satybaldiyev@getcruise.com>
Co-authored-by: Pablo <pabloem@users.noreply.github.com>
Loosen typing extensions bound #16466
[adhoc] Run spotlessApply on java examples to fix master
Merge pull request #16479: [adhoc] Run spotlessApply on java examples to fix master
[BEAM-8806] Integration test for SqsIO
[BEAM-13631] Add deterministic SQS message coder to fix reading from SQS in batch mode
Merge pull request #16128: [BEAM-13243][BEAM-8374] Add support for missing PublishResponse fields in SnsIO.Write (AWS SDK v2)
[BEAM-13628] Update SideInputCache to use full Transform and SideInputIDs as token information (#16483)
[BEAM-13602] Prevented metrics gathering from failing bigtable io
Merge pull request #16475 Optional args and kwargs for named external transforms.
[BEAM-13432] Skip ExpansionService creation in Job Server (#16222)
* [BEAM-13432] Update Java ExpansionService to be configurable of Environment
1. Update ExpansionService to use its own option instead of blindly inherit from pipeline when registering environment so it can be configurable.
2. Update JobServerDriver to skip creating expansion server when expansion port is negative.
* Revert ExpansionService change
* Add unit tests to test job server start up without expansion service.
* Apply spotless check
[BEAM-13616] Initial files for vendored gRPC 1.43.2 (#16460)
Co-authored-by: Lukasz Cwik <lcwik@google.com>
[BEAM-13638] Datatype of timestamp fields in SqsMessage for AWS IOs for SDK v2 was changed from String to long, visibility of all fields was fixed from package private to public
Merge pull request #16491: [BEAM-13638] Fix visibility of fields in SqsMessage (AWS Sdk v2)
[BEAM-13641][Playground] Add SCIO SDK support on the CI/CD step
Merge pull request #16469 from [BEAM-13623][Playground] [Bugfix] During unit tests failing there is no any output
* [BEAM-13623][Playground]
Add getting of error's output from `RunOutput` in case unit test is failed
* [BEAM-13623][Playground]
fix of tests
Merge pull request #16149 from [BEAM-13113] [Playground] playground frontend documentation
* [BEAM-13113] playground frontend documentation
* [BEAM-13113] playground frontend documentation structure
* [BEAM-13113] fix flutter command
* [BEAM-13113] remove trailing spaces
* [BEAM-13133] update beam playground frontend readme
Co-authored-by: Artur Khanin <artur.khanin@akvelon.com>
* [BEAM-13113] playground - fix typos on the docs
Co-authored-by: Artur Khanin <artur.khanin@akvelon.com>
Merge pull request #16363 from [BEAM-13557] [Playground] show code execution time
* [BEAM-13557] playground show code execution time
* [BEAM-13557] fix playground reset
* [BEAM-13557] playground - fix changing example
* [BEAM-13557] improve null checks
Merge pull request #16374 from [BEAM-13398][Playground] Split LifeCycle to DTO and business logic
* [BEAM-13398][Playground]
Add LifeCycleDto to separate DTO and business logic.
* [BEAM-13398][Playground]
Change LifeCycle structure
* [BEAM-13398][Playground]
Merge with master
* [BEAM-13398][Playground]
Merge with master
Fix using `executable_files` value
Fix comments
* [BEAM-13398][Playground]
renaming
[BEAM-13616][BEAM-13646] Update vendored calcite 1.28.0 with protobuf 3.19.2 (#16473)
Adds several example multi-language Python pipelines
Merge pull request #16325 from [BEAM-13471] [Playground] Tag existing unit-tests
* Tag java unit-test
* Tag go unit-test
* Tag python unit-test
* Update tags
* Fix checks
* Edit tags
* Update tags
Merge pull request #16488: [BEAM-13637] Adds several example multi-language Python pipelines
[BEAM-13399] Move service liveness polling to Runner type (#16487)
Split builder into several builder for each step of pipeline execution
[BEAM-8806] Integration test for SqsIO using Localstack
Provide API to check whether a hint is known.
[BEAM-13480] Increase pipeline timeout for PubSubIntegrationTest.test_streaming_data_only (#16496)
Stronger typing inference for CoGBK. (#16465)
Also fix empty union consistency check.
[BEAM-12464] Change ProtoSchemaTranslator beam schema creation to match the order for protobufs containing Oneof fields (#14974)
* ProtoSchemaTranslator now orders oneof fields in the resultant beam schema in accordance with their location in the protobuf definition
* add reverse order protobuf
* add noncontiguous oneof and some renaming
* Comments and variable renaming
* add reversed row tests
* add noncontiguous tests
* remove redundant null check
* minor test comment update
* update
* add reversedonof test
* add noncontiguous oneof test
Co-authored-by: Reuben van Ammers <reuben.vanammers@eliiza.com.au>
Introduce the notion of a JoinIndex for fewer shuffles. (#16101)
This allow joining (aka zipping) operations to execute
without requiring a global repartitioning as long as the
operands have a common, unchanged ancestor index.
Also add counter and tests to ensure expected fusion.
Merge pull request #16467 from [BEAM-12164]: SpannerIO DetectNewPartitions SDF
* [BEAM-12164]: SpannerIO DetectNewPartitions SDF
Adds the DetectNewPartitions SDF. This component will be responsible
for:
- Emitting a watermark based on the min of all unfinished partitions
in the metadata table.
- Querying all partitions in the CREATED state.
- Updating the created partitions to SCHEDULED state.
- Emitting the scheduled partitions to the PCollection.
This SDF will run periodically as based on the configured resume
interval (default is 100ms, best effort).
* chore: fix linting violations
Co-authored-by: Hengfeng Li <hengfeng@google.com>
[BEAM-12558] Fix doc typo.
Merge pull request #16385 from [BEAM-13535] [Playground] add cancel execution button
* [BEAM-13535] playground - add cancel execution button
* [BEAM-13535] fix merging cancel button
* [BEAM-13535] fix hash code to recommended
* [BEAM-13535] cancel pipeline execution log
* [BEAM-13535] playground - fix merge conflicts
Merge pull request #16485 from [BEAM-13486] [Playground] For unit tests (java) if one of tests fails the output goes to stdOutput
* [BEAM-13486] Added errorOutput to output tab
* [BEAM-13486] Fixed tests for code repository
* [BEAM-13486] Fixed runOutputError printing
Merge pull request #16413 from blais/master
[BEAM-13455] Remove duplicated artifacts when using multiple environments with Dataflow Java
[BEAM-12164]: Add SDF for reading change stream records
Adds ReadChangeStreamPartitionDoFn, which is an SDF to read partitions
from change streams and process them accordingly. This component
receives a change stream name, a partition, a start time and an end time
to query. It then initiates a change stream query with the received
parameters.
Within a change stream, 3 types of records can be received:
1. A Data record
2. A Heartbeat record
3. A Child partitions record
Upon receiving #1, the function updates the watermark with the record's
commit timestamp and emits the record into the output PCollection.
Upon receiving #2, the function updates the watermark with the record's
timestamp, but it does not emit any record into the PCollection.
Upon receiving #3, the function updates the watermark with the record's
timestamp and writes the new child partitions into the metadata table.
These partitions will be later scheduled by the DetectNewPartitions
component.
Once the change stream query for the element partition finishes, it
marks the partition as finished in the metadata table and terminates.
[BEAM-13577] Beam Select's uniquifyNames function loses nullability of Complex types while inferring schema
[BEAM-13400] JDBC IO does not support UUID and JSONB PostgreSQL types and OTHER JDBC types in general
* Support BLOB, CLOB, OTHER, JAVA_OBJECT JDBC types
* Add native support of UUID type
Merge pull request #16225 from ihji/BEAM-13455
[BEAM-13455] Remove duplicated artifacts when using multiple environm…
[BEAM-12572] Run java examples on multiple runners (#16450)
[BEAM-10206] Resolve go vet errors in protox package
Merge pull request #16284: [BEAM-13400] JdbcIO should support UUID and JSONB PostgreSQL types and OTHER JDBC types in general
[BEAM-13656] Provide an API to check whether a hint is known. #16508
Merge pull request #16477 from [BEAM-13560][Playground] Split builder into several builders for each step of the pipeline execution
[BEAM-13560][Playground] Split builder into several builders for each step of the pipeline execution
Merge pull request #16482 from [BEAM-13429][Playground] Add builder for preparers
* Split builder into several builder for each step of pipeline execution
* Add preparers builder
* Rename preparators -> preparers
* Small renaming at builders
* Forgotten renaming
[BEAM-13590] Fix abc imports from collections (#15850)
Fix staticcheck errors in transforms directory
Remove unnecessary fmt.Sprintf() in partition.go
Replace bytes.Compare() with bytes.Equal() in test cases
Replace string(buf.Bytes()) with buf.String() in coder_test.go
Remove unnecessary blank identifier assignment in harness.go
fix capitalized error strings in expansionx
Clean up string cast of bytes in vet.go and corresponding tests
Remove unnecessary fmt call in universal.go
Remove tab from source.
Redirecting cross-language transforms content (#16504)
* redirected xlang content
* updated redirect and added blurb for Java multi-lang quickstart
* removing unnecessary alias from programming guide
* improved opening sentence of Java multi-language pipelines page
Merge pull request #16528 Remove tab from source.
Merge pull request #16520: [BEAM-10206] Resolve go vet errors in protox package
[BEAM-10206] Resolve go vet errors in protox package
doc tweaks (#16498)
[BEAM-12621] - Update Jenkins VMs to modern Ubuntu version (#16457)
Co-authored-by: Giomar Osorio <95301399+GiomarWize@users.noreply.github.com>
Merge pull request #16524: [BEAM-10206] Clean up some simple staticcheck warnings in Go SDK
[BEAM-10206] Clean up some simple staticcheck warnings in Go SDK
Bump beam container version.
[BEAM-13664] Fix Primitives hashing benchmark (#16523)
* [BEAM-13628] Fix Primitives hashing benchmark
* staticcheck cleanup
Merge pull request #16507: [BEAM-13137] Fixes ES utest size flakiness with _flush api and index.store.stats_refresh_interval=0
* [BEAM-13137] Fixes ES utest size flakiness with _flush api and index.store.stats_refresh_interval=0
Merge pull request #16468: [BEAM-8806] Integration test for SqsIO using Localstack (AWS Sdk v2)
Remove jcenter repositories from gradle configuration. (#16532)
JCenter has in read-only mode and appears to have degraded availability.
https://blog.gradle.org/jcenter-shutdown
Update GH Actions to use proper variables names and proper triggers
Merge pull request #16492 from [BEAM-13641][Playground] Add SCIO SDK support on the CI/CD step
[BEAM-13641][Playground] Add SCIO SDK support on the CI/CD step
[BEAM-13430] Remove jcenter which will no longer contain any updates. (#16536)
See https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/
There are already issues with spurious timeouts/failures.
This removes the remaining references left after https://github.com/apache/beam/pull/16532
[BEAM-13616] Update com.google.cloud:libraries-bom to 24.2.0 (#16509)
[BEAM-13680] Fixed code_repository (added pipelineUuid to RunCodeResult when status is "Finished")
Merge pull request #16480: [BEAM-13631] Add deterministic SQS message coder to fix reading from SQS in batch mode (AWS Sdk v1)
Merge pull request #16481: [BEAM-8806] Integration test for SqsIO using Localstack (AWS Sdk v1)
Also bump FnAPI container.
[BEAM-13616][BEAM-13645] Switch to vendored grpc 1.43.2 (#16543)
Merge pull request #16529 Bump dataflow worker container versions.
[BEAM-13616][BEAM-13646] Upgrade vendored calcite to 1.28.0:0.2 (#16544)
Merge pull request #16545 from [BEAM-13680] [Playground] [Bugfix] Fix tests on frontend
[BEAM-13680] [Playground] [Bugfix] Fix tests on frontend
Merge pull request #16486 from [BEAM-13544][Playground] Add logs to examples CI/CD to see the progress
* [BEAM-13544][Playground]
Add logs for the CI/CD steps
Fix minor issues
* [BEAM-13544][Playground]
update logs' messages
* [BEAM-13544][Playground]
Update using of `tqdm`
Add `tqdm` to requirements.txt
* [BEAM-13544][Playground]
Add logic to replace `\t` with spaces for tag
* [BEAM-13544][Playground]
Change using `logger` to `logging`
* [BEAM-13544][Playground]
Fix according to `linter` and `yapf`
[BEAM-13683] Correct SQL transform schema, fix expansion address override bug (#16551)
* Update expansion service address after overrides
* Make options field unexported to match SQL transform schema
Update walkthrough.md (#16512)
Fixed naming error that leads to a ParseException. "appId" is the name of the field, but then in the query it erroneously said id in the WHERE.
[BEAM-11808][BEAM-9879] Support aggregate functions with two arguments (#16200)
* [BEAM-11808] Enable two params in aggregate functions, add string_agg with delimiter
* [BEAM-11808] Fix checkstyle warning
* [BEAM-11808] Fix spotlessApply
* [BEAM-11808] Fix checkstyle warning
* [BEAM-11808] Fix spotlessApply
* [BEAM-11808] Change initialization StringAgg and minor fixes
* [BEAM-11808] Fix checkstyle warnings
* [BEAM-11808] Add test cases for array_agg and timestamp null max,min
* [BEAM-11808] Remove leftover file
* [BEAM-11808] Enable resolved_literal as firts arg
* [BEAM-11808] Remove tests, validate RESOLVED_LITERAL as second argument
* [BEAM-11808] Add unsupportedException for delimiter as ResolvedParam
* use zetasql exception
* update test
Co-authored-by: Kyle Weaver <kcweaver@google.com>
Merge pull request #16506 from [BEAM-13652][Playground] Send examples' links to the frontend
* [BEAM-13652][Playground]
Add link to the PrecompiledObject
* [BEAM-13652][Playground]
Add a comment about `link` field
Merge pull request #16322 from [BEAM-13407] [Playground] Preload fonts for the web application
* [BEAM-13407] Added google fonts & licences to project
* [BEAM-13407] Fixed license name in main.dart
* [BEAM-13407] Removed unused font files
* [BEAM-13407] Added licenses to google-fonts
* [BEAM-13407] Changes in font licenses
* [BEAM-13407] Fixed _getPipelineResult method of code_repository
[BEAM-13665] Make SpannerIO projectID optional again (#16547)
Fixes regression introduced by PR #15493 which inadvertently caused
an NPE when the projectID was not specified for a SpannerIO read or
write.
Adds unit test for reading/writing both with and without projectID
[BEAM-13015] Add state caching capability to be used as hint for runners to not duplicate cached data if the SDK can do it for user state and side inputs. (#16525)
Merge pull request #16309: [BEAM-13503] Set a default value to throwWriteErrors in BulkIO constructor
[BEAM-13503] Set a default value to throwWriteErrors in BulkIO constructor
[BEAM-13015] Provide caching statistics in the status client. (#16495)
* [BEAM-13015] Provide caching statistics in the status client.
* Address PR comments.
* Address PR comments.
* Address PR comments by weighing the key.
[BEAM-13611] Skip test_xlang_jdbc_write (#16554)
Merge pull request #16370 from [BEAM-13556] playground - color and scroll tabs with new content
* [BEAM-13556] playground - color and scroll tabs with new content
* [BEAM-13556] playground - add enter after the processing has started log
* [BEAM-13556] fix tests
Merge pull request #16531 from [BEAM-13567] [playground] Handle run code validation and preparation errors
Merge pull request #16533 from [BEAM-13548] [Playground] Add example description popover
Merge pull request #16519 from [BEAM-13639] [Playground] Add notification to Log/Output tabs about cached example
* [BEAM-13639] Added log about cached results for precompiled examples
* [BEAM-13639] Fixed _getPipelineResult method of code_repository
Merge pull request #16518 from [BEAM-13619] [Playground] Add loading animation to the catalog
* [BEAM-13619] Added loading animation to the catalog
* [BEAM-13619] Refactored loading indicator component
* [BEAM-13619] Added license to loading_indicator component
* [BEAM-13619] Fixed _getPipelineResult method of code_repository
edited README and comments in Python multi-lang pipes examples
Merge pull request #16243 from darshan-sj/feature/support-priority-spannerio - Making rpcPriority a ValueProvider in SpannerConfig
* Making rpcPriority a ValueProvider in SpannerConfig
* correcting build failure
* correcting tests
* formatting changes
Allow Java 17 to be used in SDK
add testing support
Add more testing support for java 17
[BEAM-13683] Make cross-language SQL example pipeline (#16567)
[BEAM-13688] fixed type in BPG 4.5.3 window section (#16560)
Remove obsolete commands from Inventory job. (#16564)
workaround for jamm
Revert "Merge pull request #15863 from [BEAM-13184] Autosharding for JdbcIO.write* transforms"
This reverts commit 818428ff092d41e408af38832c86538d15c95033.
BEAM-13611 reactivating jdbcio xlang test
Disable logging for memoization test. (#16556)
Test loggers may cache logged items, extending their lifetime thus
violating this test's expectations about when things will be gc'd.
[BEAM-13689] Output token elements when BQ batch writes complete.
Merge pull request #16472: [BEAM-13697] Add SchemaFieldNumber annotation
Merge pull request #16373 from [BEAM-13515] [Playground] Hiding lines in an example that are not necessary
* [BEAM-13515] Implemented structure of initial scroll in CodeField
* [BEAM-13515] Changed scroll mechanics
* [BEAM-13515] Moved values to consts, changed quotes to single
* [BEAM-13515] Changed RegExp for Java, moved RegExp strings to consts
* [BEAM-13515] Clarified variable declaration
[BEAM-13093] Enable JavaUsingPython CrossLanguageValidateRunner test for dataflow runner v2
Merge pull request #16569 from Revert "Merge pull request #15863 from [BEAM-13184] Autosharding for …
Revert "Merge pull request #15863 from [BEAM-13184] Autosharding for …
Merge pull request #16371 from [BEAM-13518][Playground] Beam Playground quickstart page on the Beam website
* [WIP] Beam Playground quickstart page
* Add Beam Playground quickstart page in Get Started section
* Removed trailing whitespace
* Updated the content of Apache Beam Playground quickstart page
* Updated Beam Playground quickstart page
Update Java FnAPI beam master (#16572)
[BEAM-13699] Replace fnv with maphash. (#16573)
[BEAM-13693] Bump beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming timeout to 12 hours (#16576)
Merge pull request #16380 from talatuyarer/BEAM-13577-select-uniquifyNames-nullablity
[BEAM-13577] Beam Select's uniquifyNames function loses nullability of Complex types while inferring schema
Merge pull request #16561 from [BEAM-13689] Output TableDestination elements when BQ batch writes complete.
[BEAM-13689] Output TableDestination elements when BQ batch writes complete.
[BEAM-10206] Remove Fatalf calls in non-test goroutines for tests/benchmarks (#16575)
[BEAM-13687] Improved Spanner IO request count metrics
Moved the creation of metrics counter to startbundle.
[BEAM-13430] Re-add provided configuration (#16552)
Merge pull request #16566: edited README and comments in Python multi-lang pipes examples
Merge pull request #16514 from [BEAM-12164]: Add SDF for reading change stream records
[BEAM-12164]: Add SDF for reading change stream records
Merge pull request #16540 from [BEAM-13678][Playground]Update Github Action To Deploy Examples
* Fix deploy of examples
* Update license
Merge pull request #16539 from [BEAM-13677][Playground]Update GitHub Actions to Build Playground Containers In Proper Way
[BEAM-13677][Playground]Update GitHub Actions to Build Playground Containers In Proper Way
Merge pull request #16546 from [BEAM-13661] [BEAM-13704] [Playground] Update tags for examples/katas/unit-tests
* Add default_example field to the tags;
Remove empty optional fields from tags;
* Add quickstart category
* Add new line in the categories.yaml
Merge pull request #16369 from [BEAM-13558] [Playground] Hide the Graph tab and SCIO from SDK options
* [BEAM-13558] Hid the Graph tab from output area & SCIO from SDK options
* [BEAM-13515] Added clarifying comments
[BEAM-10206] Add key for fields in wrapper (#16583)
Merge pull request #16530 from Adding JSON support in SpannerIO and Spanner schema parser
* Adding JSON support in SpannerIO and Spanner schema parser
* formatting changes
[BEAM-13687] Improved Spanner IO request count metrics
[BEAM-13685] Enable users to specify cache directory under Interactive Beam options
Add Jenkins test for Java 17
Fix jvm hex and skip errorprone
Fix display data for anonymous classes
fix jpms tests
[BEAM-13716] Clear before creating a new virtual environment in setupVirtualenv
[BEAM-13653] Make SnsIO.write topicArn optional. If provided, validate it and force usage on the Sns publish request.
Merge pull request #16503: [BEAM-13653] Fix usage of SnsIO.write topicArn (AWS Sdk v2)
Privacy policy update regarding Apache Beam Playground
[BEAM-10897] Update the fastavro lower bound due to an issue on Windows (#16553)
[BEAM-13605] Update pandas_doctests_test denylists in preparation for pandas 1.4.0 (#16571)
Merge pull request #16538 from [BEAM-13676][Playground][Bugfix]Build Of Playground Applications Is Broken
* Update build gradle files to fix build of applications without double dockerfile issues
* Test fix
Co-authored-by: Ilya <ilya.kozyrev@akvelon.com>
Co-authored-by: Artur Khanin <artur.khanin@akvelon.com>
Merge pull request #16582 from [BEAM-13711] [Playground] [Bugfix] Add Cloud Logging API in allow list
* Add cloud logging api in allow list
* Add api to allowed_list
Merge pull request #16515 from [BEAM-13636] [Playground] Checking the default field on CI/CD step
* Checking the default field on CI/CD step
* Support an optional fields
* Update proto;
* Refactoring code
* Change default value for default_example field
* Refactoring code
[BEAM-13275] Removed the explicit selenium dependency from setup
The dependency can be transitively installed through needle, no need to
fix it in setup.
[BEAM-13275] Removed the explicit selenium dependency from setup
[BEAM-13321] Fix exception with BigQuery StreamWriter TraceID.
When running a BigQuery IO transform with a query and query location set, was given the exception: "java.lang.IllegalArgumentException: TraceId must follow the format of A:B. Actual:Dataflow". Documentation shows an example of using it with the format "Dataflow:job_id" which is what I changed this to: https://cloud.google.com/java/docs/reference/google-cloud-bigquerystorage/latest/com.google.cloud.bigquery.storage.v1.StreamWriter.Builder#com_google_cloud_bigquery_storage_v1_StreamWriter_Builder_setTraceId_java_lang_String_
[BEAM-10206] Deprecate unused shallow cloning functions (#16600)
Bump Dataflow container versions (#16602)
Merge pull request #16591 from ihji/BEAM-13716
[BEAM-13716] Clear before creating a new virtual environment in setupVirtualenv
Improved multi-language pipelines section of the programming guide (#16587)
* improved multi-language pipelines section of the programming guide
* made changes to multi-lang pipeline content, in response to feedback
* updating Beam version number for supporting no-code Java xlang transforms
* updating one more Beam version number
skip zetasql
Get rid of unnessecary logs for BigQuery streaming writes in auto-sharding mode.
[BEAM-13510] Don't retry on invalid SQS receipt handles.
Support SCIO SDK via sbt projects
Merge pull request #16478 from mosche/BEAM-13510-InvalidRetries
Merge pull request #16478: [BEAM-13510] Don't retry on invalid SQS receipt handles (AWS Sdk v2)
[BEAM-8807] Add integration test for SnsIO.write (Sdk v1 & v2)
[BEAM-13736] Make lifting cache exact. (#16603)
Merge branch 'master' of github.com:MarcoRob/beam into BEAM-12650
Merge pull request #16565 from [BEAM-13692][Playground] Implement method to receive Graph
* [BEAM-13692][Playground]
Add `GRAPH` subKey
Add `GetGraph` api method
Update `GetValue` cache method to correctly receive graph from the cache
* [BEAM-13692][Playground]
fixes according to `staticcheck`
* [BEAM-13692][Playground]
fix test
* [BEAM-13692][Playground]
Fix error with `RunError` output
* [BEAM-13692][Playground]
Change graph from []byte to string
* [BEAM-13692][Playground]
Edit comment
* [BEAM-13692][Playground]
Update comments
Co-authored-by: Ilya <ilya.kozyrev@akvelon.com>
Merge pull request #16502 from [BEAM-13650][Playground] Add link for examples
* [BEAM-13650][Playground]
Add link for examples.
* [BEAM-13650][Playground]
Change using string concatenation
Fix according to `linter` and `yapf`
* [BEAM-13650][Playground]
Change the `link` field
add pending jobs to readme
Update README.md
removed comment
[BEAM-13310] remove call to get offset consumer config, which was rep… (#16588)
* [BEAM-13310] remove call to get offset consumer config, which was replacing the intended group id with a randomly generated group id for commits
* [BEAM-13310] run spotless commit
* [BEAM-13310] fix checkstyle error
Merge pull request #16599: [BEAM-13321] Fix exception with BigQuery StreamWriter TraceID.
Merge pull request #16594 from [BEAM-13710][Playground] Update the Apache Beam privacy policy with Playground note
[BEAM-13710][Playground] Update the Apache Beam privacy policy with Playground note
[BEAM-11648] Share thread pool across RetryManager instances.
RetryManager's have short lifetime (for example processElement). This
means that many different threads are created instead of effectively
reusing them. This is adds expense but also can result in many threads
because the each individual thread lasts for up to 60 seconds and
until the GC runs.
Merge pull request #16595З: [BEAM-8807] Add integration test for SnsIO.write (Sdk v1 & v2)
[BEAM-13737][Playground] Update logic of receiving precompiled objects; Increase timeout;
added GitHub example references to Python multilang quickstart
[BEAM-13746] Fix deserialization of SSECustomerKey for AWS Sdk v2
[BEAM-7928] Allow users to specify worker disk type for Dataflow runner (#16622)
Merge pull request #16623: [BEAM-13746] Fix deserialization of SSECustomerKey for AWS Sdk v2
Exclude per-key order tests on Twister2 runner
Merge pull request #16534 from [BEAM-13671][Playground] Add backend contribute guide to the project
* [BEAM-13671][Playground]
Add CONTRIBUTE.md file
* [BEAM-13671][Playground]
Update CONTRIBUTE.md
[BEAM-13271] Bump errorprone to 2.10.0 (#16231)
* Bump errorprone to 2.10.0
* Bump autoservice to 1.0.1
Works around https://github.com/google/error-prone/issues/2745 for some
Beam projects
* Address new check failures (non-controversial)
* Address new check failures (potentially controversial)
* Disable some new checks
* Disable a few more checks
* Address check failures in test code
* Add license reference for jgit
* Explicitly ignore some return values in CoGbkResultTest
* Update sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Regex.java
Co-authored-by: kileys <kileysok@gmail.com>
* Update SnsIOTest
Co-authored-by: kileys <kileysok@gmail.com>
[BEAM-13595] Don't load main session when cloudpickle is used. (#16589)
Merge pull request #15767 from ihji/BEAM-13093
[BEAM-13093] Enable JavaUsingPython CrossLanguageValidateRunner test …
Update readme for XVR tests
Merge pull request #16626 from ihji/update_readme
Update readme for XVR tests
[adhoc] Test S3Options and AwsOptions for Sdk v2
[BEAM-13537] Fix NPE in kafkatopubsub example (#16625)
Merge pull request #16628: [adhoc] Test S3Options and AwsOptions for Sdk v2
[BEAM-13740] update java_tests.yml to remove setup-go, which is misconfigured and blocking the build (#16608)
Fix google3 import error
Merge pull request #16604: [BEAM-13714] Get rid of unnecessary logs for BigQuery streaming writes
[BEAM-12976] Implement Java projection pushdown optimizer. (#16513)
* finish basic pushdown optimizer implementation
* add docstrings
* add tests for non pbegin inputs
* support pushdown on producers with multiple outputs; ignore producers with inputs.
* make test code private
* fix null checking
* use beam null check instead of java util
* vendored guava
* nit: dont need to use string format
* address review comments
Make num-stages counter into an internal counter.
This unbreaks tests that (arguably to brittly) reject extra counters.
Merge pull request #16635 from ihji/fix_google3_import
Fix google3 import error due to missing Python gradle build file
[BEAM-13751] Don't block on gcloud when attempting to get default GCP region.
Merge branch 'master' of github.com:apache/beam into java7tests
[BEAM-13751] Parameterize wait timeout so test doesn't waste 2s.
[BEAM-13751] Add comment explaining sleep.
Merge pull request #16621: added GitHub example references to Python multilang quickstart
Merge pull request #16579 from Revert "Revert "Merge pull request #15863 from [BEAM-13184] Autoshard…
* Revert "Revert "Merge pull request #15863 from [BEAM-13184] Autosharding for JdbcIO.write* transforms""
This reverts commit 421bc8068fc561a358cfbf6c9842408672872120.
* Using batchSize to define element batch size
* Handle corner case for null list
Merge pull request #16606 from [BEAM-13247] [Playground] Embedding iframe
* [BEAM-13247] Building embedded iframe with run functionality
* [BEAM-13247] Implemented Run button & log section to the embedded iframe
* [BEAM-13247] Refactored components
* [BEAM-13247] Restricted editing in embedded iframe
* [BEAM-13247] Implemented editable/not editable versions of iframe, added Output/Log TabBar
* [BEAM-13247] Redesigned page providers for EmbeddedPlaygroundPage, changed default ratio of the EmbeddedSplitView
* [BEAM-13247] Fixes after merge
* [BEAM-13247] Added possibilities to change height & use any text in iFrame, removed codeScrolling. Added new iFrame on the try-beam-playground page
* [BEAM-13247] Changed iFrame src link
* [BEAM-13247] Fixed PR remarks
Update Python SDK beam-master tags (#16630)
Merge pull request #16592 from [BEAM-13722][Playground] Add precompiling of the graph into examples CI/CD
* [BEAM-13722][Playground]
Add receiving graph value from backend to send it as a metadata field
* [BEAM-13722][Playground]
Add license
* [BEAM-13722][Playground]
Regenerate proto files
* [BEAM-13722][Playground]
Remove graph from meta file;
Add sending of graph as a separate file;
* [BEAM-13722][Playground]
add condition for SDK
* [BEAM-13722][Playground]
Regenerate proto files
* [BEAM-13722][Playground]
Regenerate proto files
* [BEAM-13722][Playground]
small fix
* [BEAM-13722][Playground]
add try-except to receive graph
Merge pull request #16505 from [BEAM-13527] [Playground] Pipeline options dialog
* [BEAM-13527] pipeline options dropdown
* [BEAM-13527] playground - parse pipeline error message
* [BEAM-13527] playground - fix parse options
* [BEAM-13527] playground - move pipelines options lines count to const
* [BEAM-13527] playground fix tests
* fix merge
* [BEAM-13527] pipeline options fix review comments
* [BEAM-13527] pipeline options fix review comments
spotless
spotless
Generalize S3FileSystem to support multiple URI schemes.
This ports https://github.com/apache/beam/pull/15036 to aws2.
[BEAM-13768] Fix NullPointerException in BigQueryStorageSourceBase
Fix Java SDK container image name for load-tests and nexmark
Merge pull request #16639 from ibzib/BEAM-13751
[BEAM-13751] Don't block on gcloud when attempting to get default GCP…
[BEAM-13293] XLang Jdbc IO for Go SDK (#16111)
[BEAM-10206] Add Go Vet to Github Actions (#16612)
Merge pull request #16607: [BEAM-13245] Generalize S3FileSystem to support multiple URI schemes.
Change executable name fo go tests
Fix java test
[BEAM-13769] Skip test_main_session_not_staged_when_using_cloudpickle (#16651)
Merge pull request #16663 from [BEAM-13773] [Playground] [Bugfix] Error during executing of tests on Java SDK
[BEAM-13773] [Playground] [Bugfix] Error during executing of tests on Java SDK
Merge pull request #16662 from [BEAM-13772][Playground][Bugfix] Change executable name for go tests
[BEAM-13772][Playground][Bugfix] Change executable name for go tests
[BEAM-6744] Support implicitly setting project id in Go Dataflow runner (#16643)
Merge pull request #16493 from [BEAM-13632][Playground] Save catalog data to the cache
* Added saving of precompiled objects catalog to cache at the server startup
* Added caching of the catalog only in case of unspecified SDK
* Update regarding comments
* Update regarding comments
* Simplified logging regarding comment
* Updates regarding comments
* Update for environment_service_test.go
* Docstring update for GetPrecompiledObjects
Exclude jul-to-slf4j from Spark runner in quickstart POM templates (#16656)
[BEAM-13734] Support cache directories that use GCS buckets
[BEAM-11936] Enable a few errorprone checks that were broken by pinned guava in annotationProcessor (#16647)
* Don't pin old guava for annotationProcessor configurations
* Enable broken checks
* Fix CassandraIOTest assertion
[BEAM-13780] Add CONTRIBUTING.md pointing to main guide (#16666)
Co-authored-by: Danny McCormick <damccorm@github.com>
Merge pull request #16624 from [BEAM-13749] Exclude per-key order tests on Twister2 runner
[BEAM-13749] Exclude per-key order tests on Twister2 runner
[BEAM-13777] Accept cache capacity as input parameter instead of default max integer (#16652)
* Change cache capacity to suggested default value instead of max integer to prevent out of memory error
* Receive cache capacity as input prarmeter
* Spotless fixes
* [BEAM-13777] keep .of methods compatibility and use suggested default value for cache capacity by confluent
[BEAM-13051][A] Enable pylint warnings (function-redefined/bad-super-call) (#16521)
Merge pull request #16563 from [BEAM-13701][Playground] Support SCIO SDK via sbt projects
[BEAM-13701][Playground] Support SCIO SDK via sbt projects
[BEAM-13779] Fix pr labeling (#16665)
Co-authored-by: Danny McCormick <damccorm@github.com>
Merge pull request #16581 from [BEAM-12164]: Add SpannerIO.readChangeStreams
* feat: add experimental spanner readChangeStreams
Adds the SpannerIO.readChangeStreams feature that will enable users to
consume a change stream from Cloud Spanner.
This feature is under preview now, and can only be used for allowlisted
customers.
When reading a change stream the users will be able to operate on a
PCollection of DataChangeRecords, containing the modifications made to
the database as well as the type of operation.
* fix: remove public api exposure of opencensus
Do not expose the Opencensus TraceSampler in the
SpannerIO.readChangeStreams.
This is done so that we can upgrade the opencensus library without
having to concern ourselves with which version customers are using.
This commit also removes the deserializer option since it is not used.
Co-authored-by: Zoe Cai <zoc@google.com>
Fix labeler trigger (#16674)
Merge pull request #16619: [BEAM-11648] Share thread pool across RetryManager instances.
Merge pull request #16645 from ihji/fix_nexmark_dataflow_v2
Fix Java SDK container image name for load-tests and nexmark
[BEAM-13781] Exclude grpc-netty-shaded from gax-grpc's dependency (#16672)
[BEAM-13051] Fixed pylint warnings : raising-non-exception (E0710), super-on-old-class (E1002) (#16541)
* [BEAM-BEAM-13051] Fixed pylint warnings unexpeted-keyword-arg (E1123), undefined-variable (E0602)
* Delete raising-non-exception and super-on-old-class from .pylintrc
* Disable checks that broke tests
* Delete extra blanck lines
[BEAM-13740] Correctly install go before running tests (#16673)
[BEAM-12830] Update local Docker env Go version. (#16670)
[BEAM-13051][B] Enable pylint warnings (import-error/invalid-unary-operand-type) (#16522)
* [BEAM-13051] Pylint invalid-unary-operand-type warning enabled
* [BEAM-13051] Added unary negative operator to Timestamp
* [BEAM-13051] Disabled invalid-unary-operand-type warning in specific cases
[BEAM-13430] Revert Spark libraries in Spark runner to provided (#16675)
[BEAM-12240] Add Java 17 support (#16568)
Merge branch 'master' of github.com:apache/beam into java7tests
[BEAM-13760] Add random component to default python dataflow job name (#16641)
* Add random component to default python dataflow job name
* Correct formattiing of multiline statements
* Remove uuid from random string implementation.
Fix trigger
Merge pull request #16655 from [BEAM-12164]: Add retry protection to DetectNewPartitions
* feat: add experimental spanner readChangeStreams
Adds the SpannerIO.readChangeStreams feature that will enable users to
consume a change stream from Cloud Spanner.
This feature is under preview now, and can only be used for allowlisted
customers.
When reading a change stream the users will be able to operate on a
PCollection of DataChangeRecords, containing the modifications made to
the database as well as the type of operation.
* fix: remove public api exposure of opencensus
Do not expose the Opencensus TraceSampler in the
SpannerIO.readChangeStreams.
This is done so that we can upgrade the opencensus library without
having to concern ourselves with which version customers are using.
This commit also removes the deserializer option since it is not used.
* [BEAM-12164]: Add retry protection to DetectNewPartitions
The original algorithm of the DetectNewPartitions is susceptible to
failures, because it produces side effects on every try which is not
idempotent. Specifically, it marks the partitions as SCHEDULED in the
Spanner database and outputs them. If there is a bundle commit failure,
during retry, the already SCHEDULED partitions will not be picked up
again.
We change the algorithm in this PR to always schedule partitions that
have a created at timestamp greater than the one saved in the
DetectNewPartitions restriction. When scheduling the partitions, this
SDF will also claim the created at of such partitions, advancing the
timestamp saved. If there is a bundle commit failure, the restriction
timestamp won't be saved, thus the partitions in the bundle will be
picked up again regardless of their state.
Co-authored-by: Zoe Cai <zoc@google.com>
Merge pull request #16586 from [BEAM-13731] FhirIO: Add support for BATCH bundle errors.
* FhirIO: Add support for BATCH bundle errors.
* SpotlessApply
skip checker framework
fix app name
[BEAM-13011] Adds a link to Multi-language Pipelines Tips wiki page (#16649)
* Adds a link to Multi-language Pipelines Tips wiki page
* Address reviewer comment
* Add anchor
[BEAM-13734] Support cache directories that use GCS buckets
remove duplicate property check
[BEAM-12572] Run python examples on multiple runners (#16154)
[BEAM-13574] Large Wordcount (#16455)
Avoid packaging avro in the java harness jar.
Unfortunately we can't simply shade it, as it's used in the public API
(AvroCoder) of sdks:java:core. However, outside that use, it should not
be needed when running in portable mode. Users will have to provide
their own avro (typically by depending on sdks:java:core, but possibly
choosing another version explicitly or from elsewhere).
[BEAM-13293] Refactor JDBC IO Go Wrapper (#16686)
* [BEAM-13293] refactor jdbcio
* [BEAM-13293] resolve comments
Edit license script for Java, add manual licenses for xz (#16692)
[BEAM-13563] Restructure Kinesis Source for Aws 2 internally to prepare for ClientBuilderFactory
Merge pull request #16657: [BEAM-13563] Restructure Kinesis Source for AWS 2
[BEAM-4665] Allow joining a running dataflow pipeline without throwing (#16689)
[BEAM-13801] Add standard coder tests for state backed iterable. (#16696)
Add coverage for Java and Python. Skip testing for Go. Go change a future follow-up.
Merge branch 'master' of github.com:apache/beam into java7tests
Merge pull request #16644: [BEAM-13768] Fix NullPointerException in BigQueryStorageSourceBase
[BEAM-13430] Fix provided configuration by removing extendsFrom for implementation (#16688)
[BEAM-12830] Print clearer go version fail message (#16693)
Add reference to Books to Learning Resources in website
Use ThreadLocal for DESERIALIZATION_CONTEXT (#16680)
Co-authored-by: Kellen Dye <kellend@spotify.com>
Merge pull request #16682: Add reference to Books to Learning Resources in website
Minor: Add apt update after adding deadsnakes repository in dev env (#16708)
Merge pull request #16638 from robertwb/internal-counters
Make num-stages counter into an internal counter.
Merge pull request #16694 [BEAM-13800] Avoid packaging avro in the java harness jar.
[BEAM-13807] Regenerate container images to get TF 2.8.0 (#16707)
* Regenerate container images to get TF 2.8.0
* Add tf-estimator-nightly license URL
[BEAM-13399, BEAM-13683] Eagerly materialize artifacts for automated expansion service, add feature to SQL transform (#16671)
Merge pull request #16617 from [BEAM-13743] [Playground] Add context line for examples
* Add context line for examples
* Subtract number of tag lines from context_line
* Update proto
* Update licenses;
Merge pull request #16618 from [BEAM-13744] [Playground] Add context line field to the tags
* Add context_line to tags
* Edit context_line`s in the tags
* Edit info about default_example
Merge pull request #16698 from [BEAM-13802][Playground] [Bugfix] Clean all build directroies, clean kubernetes env
[BEAM-13293][BEAM-13806] Pipe a SchemaIO flag through Go integration test script. (#16705)
This is needed for integration tests of Schema IO based xlang transforms, like JDBC or BigQuery.
Merge pull request #16620 from [BEAM-13737][Playground] Local retrieving of example objects may fail with context deadline
[BEAM-13737][Playground] Local retrieving of example objects may fail with context deadline
[BEAM-13605] Modify groupby.apply implementation in preparation for pandas 1.4.0 (#16706)
* Modify groupby.apply implementation in preparation for pandas 1.4.0
* fixup! Modify groupby.apply implementation in preparation for pandas 1.4.0
* Address review comments
Merge pull request #16436 from [BEAM-1330] - DatastoreIO Writes should flush early when duplicate keys arrive
* Switching to a HashSet to flush batch when duplicate is found
* Keeping track of duplicates in a separate HashSet
* clearing HashSet when flushing the data
* Adding test for the duplicated entries scenario
* Changing test implementation to use TestPipeline
* Applying spotless
* Executing tests with Mutations
* Forcing the processing of the List of mutation in the same order the mutations were added to the original List
* Moving the batchSize.update call to within the flushBatch method
* Bumping to 7 since batchSize.update is now being called from within flushBatch
[BEAM-13813] Add support for URL artifact to extractStagingToPath
[BEAM-13663] Remove unused duplicate option for AWS client configuration (AWS Sdk v1).
Remove Python SQL Test example from catalog
[BEAM-13246] Add support for S3 Bucket Key at the object level (AWS Sdk v2).
Fix timer consistency in direct runner
[BEAM-13757] adds pane observation in DoFn (#16629)
Merge pull request #16650:[BEAM-11971] Fix directrunner timer consistency
Change links to Books from Amazon to Publisher
[BEAM-13605] Add support for pandas 1.4.0 (#16590)
* Addding new functions to / fixing doctests
* Add _rename and value_counts()
* Move import statement
* Add if DataFrame has value_counts attr
* Fix typo
* Update precommit script and setup.py to 1.4
* Add backwards compatability for rename and replace
* Add docstring and simplify kwargs
* Skip DataFrame construction with series
* Add change to CHANGES.md
* Skip failing pyarrow test
* Add pandas 1.4 to tox.ini
Merge pull request #16718: [website] Change links to Books from Amazon to Publisher
[BEAM-13761] adds Debezium IO wrapper for Go SDK (#16642)
[BEAM-12976] Log projection pushdown optimizations.
Merge pull request #16640 from kileys/java7tests
[BEAM-12240] Add Jenkins tests for Java 17
[BEAM-13024] Unify PipelineOptions behavior (#16719)
Update sdks/go/pkg/beam/artifact/materialize_test.go
Co-authored-by: Robert Burke <lostluck@users.noreply.github.com>
Merge pull request #16710 from ihji/BEAM-13813
[BEAM-13813] Add support for URL artifact to extractStagingToPath
Merge pull request #16713 from [BEAM-13815] [Playground] [Bugfix] Remove Python SQL Test example from catalog
[BEAM-13815] [Playground] [Bugfix] Remove Python SQL Test example from catalog
Merge pull request #16605 from [BEAM-13634][Playground] Create a separate Dockerfile for the routing service
* Added saving of precompiled objects catalog to cache at the server startup
* Added caching of the catalog only in case of unspecified SDK
* Update regarding comments
* Update regarding comments
* Simplified logging regarding comment
* Updates regarding comments
* Update for environment_service_test.go
* Docstring update for GetPrecompiledObjects
* Add router services as separate container
* too stage container builder
* add empty string to build_playground_backend.yml
* fix comments
Co-authored-by: Artur Khanin <artur.khanin@akvelon.com>
Co-authored-by: akustov <akustov@diasoft.ru>
Co-authored-by: Ilya <ilya.kozyrev@akvelon.com>
Merge pull request #16593 from [BEAM-13725][Playground] Add graph to the precompiled objects
* [BEAM-13725][Playground]
Add graph field to `PrecompiledObject`
* [BEAM-13725][Playground]
Fix test
* [BEAM-137235][Playground]
Remove `graph` from PrecompiledObject
Add a new methods to receive graph of the PrecompiledObject.
* [BEAM-13725][Playground]
Update comments;
Remove unnecessary field;
* [BEAM-13725][Playground]
Fix tests
* [BEAM-13725][Playground]
Regenerate proto files
* [BEAM-13725][Playground]
renaming
* [BEAM-13725][Playground]
Regenerate files
* [BEAM-13725][Playground]
Regenerate proto files
Merge pull request #16699 from [BEAM-13789][Playground] Change logic of keeping examples to the bucket on CD side
* [BEAM-13789][Playground]
Change examples' folders structure
Fix by `yapf` and `pylint`
* [BEAM-13789][Playground]
Change examples' folders structure
Fix by `yapf` and `pylint`
Fixed CSS for Case study page
Merge pull request #16730: Fixed CSS for Case study page
[BEAM-13203] Deprecate SnsIO.writeAsync for AWS Sdk v2 due to risk of data loss.
Merge pull request #16712: [BEAM-13663] Remove unused duplicate option for AWS client configuration (AWS Sdk v1)
Merge pull request #16711: [BEAM-13203] Deprecate SnsIO.writeAsync for AWS Sdk v2 due to risk of data loss.
[BEAM-13828] Fix stale bot (#16734)
Merge pull request #16364 from [BEAM-13182] Add diagrams to backend readme
* Add diagrams to backend README.md
* Fix comments
* Fix comments
[BEAM-13811] Fix save_main_session arg in tests examples (#16709)
…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I ran into 404s from deadsnakes when running
./start-build-env.shfor #16707. Running anapt updateafter adding the deadsnakes repository resolved it for me.ValidatesRunnercompliance status (on master branch)Examples testing status on various runners
Post-Commit SDK/Transform Integration Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.