Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BEAM-10961] enable strict dependency checking for sdks/java/io/parquet #14062

Merged
merged 3 commits into from Feb 25, 2021
Merged

[BEAM-10961] enable strict dependency checking for sdks/java/io/parquet #14062

merged 3 commits into from Feb 25, 2021

Conversation

shehzaadn-vd
Copy link
Contributor

Split from #13688

R: @ibzib

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang SDK Dataflow Flink Samza Spark Twister2
Go Build Status --- Build Status --- Build Status ---
Java Build Status Build Status
Build Status
Build Status		 Build Status
Build Status
Build Status
Build Status		 Build Status Build Status
Build Status
Build Status		 Build Status
Python Build Status
Build Status
Build Status		 Build Status
Build Status
Build Status		 Build Status
Build Status		 --- Build Status ---
XLang Build Status Build Status Build Status --- Build Status ---

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website Whitespace Typescript
Non-portable Build Status Build Status
Build Status
Build Status
Build Status		 Build Status Build Status Build Status Build Status
Portable --- Build Status --- --- --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests

See CI.md for more information about GitHub Actions CI.

ibzib
ibzib requested changes Feb 24, 2021
View reviewed changes
sdks/java/io/parquet/build.gradle Outdated
@@ -46,11 +47,10 @@ dependencies {
compile "org.apache.parquet:parquet-common:$parquet_version"
compile "org.apache.parquet:parquet-hadoop:$parquet_version"
compile library.java.avro
compile "org.apache.parquet:parquet-column:1.10.0"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, please move this line so its with the rest of the parquet dependencies.

Suggested change
compile "org.apache.parquet:parquet-column:1.10.0"
compile "org.apache.parquet:parquet-column:$parquet_version"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops. I didn't notice the $parquet_version variable.

ibzib
ibzib approved these changes Feb 25, 2021
View reviewed changes
Copy link
Contributor

@ibzib ibzib left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ibzib ibzib merged commit 1a4d9a2 into apache:master Feb 25, 2021
kennknowles added a commit to kennknowles/beam that referenced this pull request Mar 4, 2021
@kennknowles
Merge remote-tracking branch 'github/master' into py-continuation-tri…
44fba3a 
…gger

* github/master: (123 commits)
  [BEAM-11899] Bump commons-pool to 2.8.1 and bump commons-dbcp to 2.8.0, Because there is a library dependency
  Update pillars.yaml (apache#14142)
  [BEAM-10632] Checkerframework nullness cleanups (apache#14107)
  [BEAM-11213] Instantiate SparkListenerApplicationStart in a Spark 3 compatible way
  Fix typos for excluding testMergingCustomWindowsWithoutCustomWindowTypes
  Specify the time resolution for TestStreamPayload.
  [BEAM-10961] Enable strict depdency checking for sdks/java/extensions/euphoria (second attempt)
  [BEAM-11848] Store Docker images in a variable for consistency.
  Splitting old Go Precommit and new ULR integration test precommit.
  Moving runner imports out of ptest.
  Add the TO_STRING capability to Java and Python
  [BEAM-11848] Fix Docker images list.
  jdbc python supported Dataflow runner (apache#13960)
  Adding a warning to use multi-workers on FnApiRunner
  Fix legend for Python Directrunner microbenchmarks
  [BEAM-11740] Estimate PCollection byte size
  [BEAM-10961] enable strict dependency checking for sdks/java/extensions/zetasketch (apache#14093)
  Map Dataflow JOB_STATE_CANCELLING to Beam RUNNING state
  [BEAM-11833] Fix reported watermark after restriction split in UnboundedSourceAsSDFRestrictionTracker
  [BEAM-10761] add reference to BEAM-11761
  [BEAM-10961] enable strict dependency checking for flink/job-server
  Exclude MapState example integration tests from Dataflow runner v2 suite
  Remove InvalidWindows from Java SDK, instead track "already merged" bit
  Fix checkstyle in watermark latency benchmark
  Fix compile breakage in WindmillStateInternals
  Improve test, error on ALREADY_MERGED.
  [BEAM-10961] Strict dependency checking for sdks/java/io/gcp (apache#13791)
  Initial watermark latency benchmark
  Attempting improvements on DirectRunner Python dash
  [BEAM-10961] enable strict dependency checking for sdks/java/extensions/google-cloud-platform-core (apache#14084)
  Merge pull request apache#13802: [BEAM-1474]. Adding MapState and SetState support for the Dataflow runner
  Remove some false positives
  Remove nullness warning suppression
  [BEAM-11861] Add methods to explicitly provide coder for ParquetIO's Parse and ParseFiles (apache#14078)
  [BEAM-11531] Use pandas 1.2 for python>=3.7 (apache#14099)
  [BEAM-10961] add reference to BEAM-11761
  [BEAM-10961] add explicit compile for auto_value_annotations in sdks/extensions/ml/build.gradle
  Attempting improvements on DirectRunner Python dash
  Recognize JOB_STATE_PENDING from Dataflow and map to RUNNING
  never run checkerframework on tests
  Puts more expensive BQ empty table check to the right of the 'and' condition (apache#14094)
  Use the windowing strategy of the input, not output, PCollection of GBK.
  Do not stage dataflow worker jar when use runner_v2.
  [BEAM-11870] Re-raise underlying exception for InvocationTargetException (apache#14098)
  [BEAM-11778] Create a wrapper for ZetaSQL catalog and refactor accordingly. (apache#13934)
  [BEAM-9378] Add ignored tests which fail in various ways when querying nested structures (apache#14077)
  Merge Fn API and runner v2 configurations for DataflowRunner
  Fix up! formatting
  Add validate runner test for testing custom merging windows fn without custom window types
  Revert "Revert "[BEAM-2914] Add portable merging window support to Python. (apache#12995)""
  [BEAM-10961] fix stray reordering of lines
  [BEAM-10961] enable strict dependency checking for sdks/java/extensions/sorter
  [BEAM-10961] enable strict dependency checking for sdks/java/extensions/sketching
  [BEAM-10961] enable strict dependency checking for sdks/java/extensions/schemaio-expansion-service
  [BEAM-10961] enable strict dependency checking for sdks/java/extensions/protobuf
  [BEAM-10961] enable strict dependency checking for sdks/java/extensions/ml
  [BEAM-10961] enable strict dependency checking for sdks/java/extensions/kyro
  [BEAM-10961] enable strict dependency checking for sdks/java/extensions/join-library
  [BEAM-10961] undo line moves (originally intended for alphabeticization)
  [BEAM-10961] enable strict dependency checking for sdks/java/extensions/jackson
  [BEAM-10961] Enable strict dependency checking on sdks/java/extensions/sql (apache#13830)
  [BEAM-10961] enable strict dependency checking for sdks/java/extensions/euphoria
  [BEAM-10961] enable strict dependency checking for sdks/java/io/parquet (apache#14062)
  [BEAM-10961] enable strict dependency checking for sdks/java/io/thrift (apache#14066)
  Refactor ZetaSqlDialectSpecTest and add some passing tests. (apache#14080)
  [BEAM-11864] Use objects.equals instead of raw comparison
  [BEAM-11707] Change WindmillStateCache cache invalidation to be based upon reference invalidation instead of expensive set management. Reduce operations of shared cache by caching per-key object sets locally and flushing as groups to shared cache. Remove byte tracking which could be racy based upon background evictions in favor of just iterating for rendering the status page. This also lets us capture more stats.
  [BEAM-11730] Reduce context switching overhead for appliance reads by issuing reads directly from calling threads in the case that there is no reads being queued.
  Fix preview
  Show string from Dataflow service when job terminates in unrecognized state
  Log a warning when Dataflow returns an unrecognized state
  Merge pull request apache#14033 from [BEAM-11408] Integrate Python BigQuery sink with GroupIntoBatches
  Remove SYNCHRONIZED_PROCESSING_TIME from model proto
  Remove use of model SYNCHRONIZED_PROCESSING_TIME
  Merge redundant model feature columns in capability matrix
  Remove MapReduce runner from capability matrix, because it is on a branch and unreleased
  Remove JStorm runner from capability matrix, because it is on a branch and unreleased
  Remove retractions from capability matrix, because they do not exist yet
  Remove metadata-driven triggers from capability matrix, because they do not exist
  [BEAM-10937] Add Tour of Beam page (apache#13747)
  [BEAM-11344] Apply "Become a Committer" changes from Website Revamp (apache#14036)
  Merge pull request apache#14046 from [BEAM-11791] Adding a microbenchmark for TestStream
  Returning successful writes in FhirIO.Write.Result (apache#14034)
  Fixup
  [BEAM-10961] enable strict dependency checking for sdks/java/io/file-based-io-tests (apache#14052)
  [BEAM-10961] enable strict dependency checking for sdks/java/io/contextualtextio (apache#14049)
  [BEAM-10961] enable strict dependency checking for sdks/java/io/kinesis (apache#14058)
  [BEAM-10961] enable strict dependency checking for sdks/java/io/bigquery-io-perf-tests (apache#14048)
  [BEAM-10961] enable strict dependency checking for sdks/java/io/elasticsearch (apache#14050)
  [BEAM-10961] enable strict dependency checking for sdks/java/io/expansion-service (apache#14051)
  [BEAM-10961] enable strict dependency checking for sdks/java/io/jdbc (apache#14055)
  [BEAM-10961] enable strict dependency checking for sdks/java/io/jms (apache#14056)
  [BEAM-10961] enable strict dependency checking for sdks/java/io/kafka (apache#14057)
  [BEAM-10961] enable strict dependency checking for sdks/java/io/hcatalog (apache#14053)
  [BEAM-11859] Fixed bug in python S3 IO
  [BEAM-10114] Fix PerSubscriptionPartitionSdf to not rely on the presence of BundleFinalizer
  [BEAM-10114] Fix PerSubscriptionPartitionSdf to not rely on the presence of BundleFinalizer
  [BEAM-10961] fix spacing
  [BEAM-10961] enable strict dependency checking for sdks/java/io/xml
  [BEAM-10961] enable strict dependency checking for sdks/java/io/tika
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ibzib ibzib ibzib approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@shehzaadn-vd @ibzib