Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BEAM-12164]: fix the autoscaling backlog estimation for Spanner Change Streams Connector #17200

Merged
merged 5 commits into from
Apr 14, 2022
Merged

[BEAM-12164]: fix the autoscaling backlog estimation for Spanner Change Streams Connector #17200

merged 5 commits into from
Apr 14, 2022

Conversation

hengfengli
Copy link
Contributor

No description provided.

thiagotnunes
thiagotnunes reviewed Mar 29, 2022
View reviewed changes
...in/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/action/QueryChangeStreamAction.java Outdated Show resolved Hide resolved
...ava/org/apache/beam/sdk/io/gcp/spanner/changestreams/dofn/ReadChangeStreamPartitionDoFn.java Show resolved Hide resolved
...ava/org/apache/beam/sdk/io/gcp/spanner/changestreams/dofn/ReadChangeStreamPartitionDoFn.java Outdated Show resolved Hide resolved
...n/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/restriction/ThroughputEstimator.java Outdated Show resolved Hide resolved
...va/org/apache/beam/sdk/io/gcp/spanner/changestreams/restriction/ThroughputEstimatorTest.java Show resolved Hide resolved
.../src/test/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/util/TimestampGenerator.java Outdated Show resolved Hide resolved
...n/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/restriction/ThroughputEstimator.java Show resolved Hide resolved
@hengfengli hengfengli force-pushed the update-restriction-tracker branch 4 times, most recently from d9031fe to 06fd55c Compare March 31, 2022 06:05
@codecov
Copy link

codecov bot commented Mar 31, 2022 •
edited
Loading

Codecov Report

Merging #17200 (8af69d7) into master (0086a2b) will increase coverage by 0.02%.
The diff coverage is n/a.

❗ Current head 8af69d7 differs from pull request most recent head 8c990db. Consider uploading reports for the commit 8c990db to get more accurate results

@@            Coverage Diff             @@
##           master   #17200      +/-   ##
==========================================
+ Coverage   73.94%   73.97%   +0.02%     
==========================================
  Files         684      672      -12     
  Lines       89519    88460    -1059     
==========================================
- Hits        66194    65437     -757     
+ Misses      22165    21902     -263     
+ Partials     1160     1121      -39
Flag Coverage Δ
go 49.56% <0.00%> (-0.21%) ⬇️
python 83.66% <0.00%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
sdks/go/pkg/beam/core/runtime/graphx/cogbk.go 0.00% <0.00%> (-72.73%) ⬇️
...beam/runners/portability/local_job_service_main.py 0.00% <0.00%> (-14.44%) ⬇️
sdks/go/pkg/beam/core/runtime/graphx/translate.go 28.81% <0.00%> (-14.12%) ⬇️
sdks/python/apache_beam/runners/job/utils.py 69.23% <0.00%> (-7.70%) ⬇️
sdks/go/pkg/beam/runners/dataflow/dataflow.go 50.00% <0.00%> (-3.03%) ⬇️
sdks/python/apache_beam/io/gcp/gcsio.py 90.12% <0.00%> (-2.50%) ⬇️
...ache_beam/runners/portability/local_job_service.py 80.00% <0.00%> (-1.58%) ⬇️
sdks/go/pkg/beam/core/graph/coder/map.go 63.72% <0.00%> (-1.28%) ⬇️
sdks/go/pkg/beam/core/runtime/exec/translate.go 5.81% <0.00%> (-1.16%) ⬇️
sdks/python/apache_beam/internal/metrics/metric.py 91.00% <0.00%> (-1.00%) ⬇️
... and 125 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0086a2b...8c990db. Read the comment docs.

@hengfengli
Copy link
Contributor Author

Run Java_Examples_Dataflow PreCommit

@hengfengli hengfengli force-pushed the update-restriction-tracker branch 3 times, most recently from 68fa94e to fd1e90a Compare April 4, 2022 07:18
thiagotnunes
thiagotnunes approved these changes Apr 7, 2022
View reviewed changes
Copy link
Contributor

@thiagotnunes thiagotnunes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

...va/org/apache/beam/sdk/io/gcp/spanner/changestreams/restriction/ThroughputEstimatorTest.java Outdated Show resolved Hide resolved
@hengfengli hengfengli force-pushed the update-restriction-tracker branch 2 times, most recently from 17a2d10 to 2afbc65 Compare April 10, 2022 23:49
@hengfengli
Copy link
Contributor Author

Run Java PreCommit

@thiagotnunes thiagotnunes mentioned this pull request Apr 12, 2022
Merged
@pabloem
Copy link
Member

pabloem commented Apr 12, 2022

taking a look at this...

@thiagotnunes
Copy link
Contributor

Run Java PostCommit

@hengfengli
Copy link
Contributor Author

Run Java PreCommit

@thiagotnunes
Copy link
Contributor

Run Java PostCommit

@pabloem
Copy link
Member

pabloem commented Apr 14, 2022

this LGTM. Thanks y'all!

@pabloem pabloem merged commit 8a33871 into apache:master Apr 14, 2022
@TheNeuralBit TheNeuralBit mentioned this pull request Oct 6, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@thiagotnunes thiagotnunes thiagotnunes approved these changes

@pabloem pabloem pabloem approved these changes

Assignees
No one assigned
Labels
dataflow gcp io java runners
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@hengfengli @pabloem @thiagotnunes