Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-13867] Drop NaNs returned by nlargest in flight_delays example pipeline #16801

Merged
merged 1 commit into from Feb 10, 2022

Conversation

TheNeuralBit
Copy link
Member

@TheNeuralBit TheNeuralBit commented Feb 9, 2022

pandas 1.4.0 slightly changed the behavior of nlargest for pandas-dev/pandas#28984 - now it can return NaNs in some specific cases. This PR modifies our flight_delays example to explicitly drop those NaNs.

ValidatesRunner compliance status (on master branch)

Lang ULR Dataflow Flink Samza Spark Twister2
Go --- Build Status Build Status Build Status Build Status ---
Java Build Status Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Python --- Build Status
Build Status
Build Status
Build Status
Build Status
Build Status Build Status ---
XLang Build Status Build Status
Build Status
Build Status
Build Status Build Status Build Status ---

Examples testing status on various runners

Lang ULR Dataflow Flink Samza Spark Twister2
Go --- --- --- --- --- --- ---
Java --- Build Status
Build Status
Build Status
--- --- --- --- ---
Python --- --- --- --- --- --- ---
XLang --- --- --- --- --- --- ---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go Java Python
Build Status Build Status Build Status
Build Status
Build Status

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website Whitespace Typescript
Non-portable Build Status
Build Status
Build Status
Build Status
Build Status
Build Status Build Status Build Status Build Status
Portable --- Build Status Build Status --- --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests

See CI.md for more information about GitHub Actions CI.

@TheNeuralBit TheNeuralBit changed the title [WIP] Drop NaNs returned by nlargest in flight_delays example pipeline [BEAM-13867] Drop NaNs returned by nlargest in flight_delays example pipeline Feb 9, 2022
@TheNeuralBit
Copy link
Member Author

Run Python 3.8 PostCommit

@codecov
Copy link

codecov bot commented Feb 9, 2022

Codecov Report

Merging #16801 (b315955) into master (16355ca) will increase coverage by 37.23%.
The diff coverage is 0.00%.

Impacted file tree graph

@@             Coverage Diff             @@
##           master   #16801       +/-   ##
===========================================
+ Coverage   46.38%   83.62%   +37.23%     
===========================================
  Files         202      452      +250     
  Lines       19828    62252    +42424     
===========================================
+ Hits         9198    52059    +42861     
- Misses       9643    10193      +550     
+ Partials      987        0      -987     
Impacted Files Coverage Δ
...on/apache_beam/examples/dataframe/flight_delays.py 27.90% <0.00%> (ø)
sdks/go/pkg/beam/io/databaseio/database.go
sdks/go/pkg/beam/io/synthetic/step.go
sdks/go/pkg/beam/io/filesystem/filesystem.go
sdks/go/pkg/beam/core/runtime/graphx/serialize.go
sdks/go/pkg/beam/transforms/stats/quantiles.go
sdks/go/pkg/beam/core/runtime/exec/plan.go
.../pkg/beam/runners/universal/extworker/extworker.go
sdks/go/pkg/beam/core/util/protox/base64.go
sdks/go/pkg/beam/core/runtime/harness/datamgr.go
... and 645 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 16355ca...b315955. Read the comment docs.

@tvalentyn
Copy link
Contributor

Run Python Examples_Direct

@TheNeuralBit TheNeuralBit merged commit 633169c into apache:master Feb 10, 2022
Naireen pushed a commit to Naireen/beam that referenced this pull request Feb 18, 2022
Naireen pushed a commit to Naireen/beam that referenced this pull request Feb 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants