Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-9547] Thorough testing for global aggregations #14893

Merged
merged 5 commits into from Jun 7, 2021

Conversation

TheNeuralBit
Copy link
Member

In the same spirit as #14850 I added some thorough parameterized tests for global aggregations, separated into their own test suite AggregationTests. Not surprisingly this knocked loose a few unhandled edge cases, also fixed in this PR:

  • We don't need a specialized implementation of nunique in DataFrame, it can be done with _agg_method. We can add a distributed implementation in Series though, based on drop_duplicates, and it will be used through DataFrame.agg.
  • Added logic to direct DataFrame.agg to specialized implementations for corr, cov, quantile and size.
  • Direct Series.agg to specialized implementations for corr, cov.

ValidatesRunner compliance status (on master branch)

Lang ULR Dataflow Flink Samza Spark Twister2
Go --- Build Status Build Status --- Build Status ---
Java Build Status Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Python --- Build Status
Build Status
Build Status
Build Status
Build Status
--- Build Status ---
XLang Build Status Build Status Build Status --- Build Status ---

Examples testing status on various runners

Lang ULR Dataflow Flink Samza Spark Twister2
Go --- --- --- --- --- --- ---
Java --- Build Status
Build Status
Build Status
--- --- --- --- ---
Python --- --- --- --- --- --- ---
XLang --- --- --- --- --- --- ---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go Java Python
Build Status Build Status Build Status
Build Status
Build Status

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website Whitespace Typescript
Non-portable Build Status
Build Status
Build Status
Build Status
Build Status
Build Status Build Status Build Status Build Status
Portable --- Build Status Build Status --- --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests

See CI.md for more information about GitHub Actions CI.

@TheNeuralBit
Copy link
Member Author

R: @rohdesamuel

@codecov
Copy link

codecov bot commented May 28, 2021

Codecov Report

Merging #14893 (e41b1f4) into master (f207ad1) will increase coverage by 0.03%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #14893      +/-   ##
==========================================
+ Coverage   83.76%   83.79%   +0.03%     
==========================================
  Files         435      435              
  Lines       58365    58434      +69     
==========================================
+ Hits        48887    48966      +79     
+ Misses       9478     9468      -10     
Impacted Files Coverage Δ
...e_beam/portability/api/beam_interactive_api_pb2.py
..._beam/testing/benchmarks/nexmark/queries/query8.py
..._beam/testing/benchmarks/nexmark/queries/query4.py
...runners/interactive/display/pcoll_visualization.py
..._beam/testing/benchmarks/nexmark/queries/query3.py
...apache_beam/typehints/native_type_compatibility.py
...am/testing/benchmarks/chicago_taxi/trainer/taxi.py
...ld/srcs/sdks/python/apache_beam/utils/processes.py
...n/apache_beam/examples/cookbook/bigquery_schema.py
...ks/python/apache_beam/runners/worker/statecache.py
... and 860 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f207ad1...e41b1f4. Read the comment docs.

@TheNeuralBit TheNeuralBit merged commit b6aad41 into apache:master Jun 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants