[HUDI-68] Pom cleanup & demo automation #846
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@thesuperzapper this is now up, with full demo automation.. FYI |
vinothchandar
changed the title
vinothchandar
commented
Aug 22, 2019
| @@ -127,6 +127,8 @@ | |||
| <pattern>parquet.schema</pattern> | |||
| <shadedPattern>org.apache.hudi.parquet.schema</shadedPattern> | |||
| </relocation> | |||
| <!-- TODO: Revisit GH ISSUE #533 & PR#633--> | |||
There was a problem hiding this comment.
@bvaradar this needs your special attention.. Had to do this to get hive sync as a part of spark datasource working.. some background in those issues and prs
There was a problem hiding this comment.
Ack. We would have to check with other certified combinations of Hive and Spark
vinothchandar
changed the title
|
CI has passed 4 times in a row now. I think its stable enough to merge |
bvaradar
approved these changes
Aug 22, 2019
| @@ -127,6 +127,8 @@ | |||
| <pattern>parquet.schema</pattern> | |||
| <shadedPattern>org.apache.hudi.parquet.schema</shadedPattern> | |||
| </relocation> | |||
| <!-- TODO: Revisit GH ISSUE #533 & PR#633--> | |||
There was a problem hiding this comment.
Ack. We would have to check with other certified combinations of Hive and Spark
- Fix ordering of dependencies in poms, to enable better resolution - Idea is to place more specific ones at the top - And place dependencies which use them below them
- Move hive queries from hive cli to beeline - Standardize on taking query input from text command files - Deltastreamer ingest, also does hive sync in a single step - Spark Incremental Query materialized as a derived Hive table using datasource - Fix flakiness in HDFS spin up and output comparison - Code cleanup around streamlining and loc reduction - Also fixed pom to not shade some hive classs in spark, to enable hive sync
vinothchandar
force-pushed
the
pom-cleanup-demo-automation
branch
from
0c103db
to
e23112a
Compare
bvaradar
pushed a commit
to bvaradar/hudi
that referenced
this pull request
Sep 19, 2019
- [HUDI-172] Cleanup Maven POM/Classpath - Fix ordering of dependencies in poms, to enable better resolution - Idea is to place more specific ones at the top - And place dependencies which use them below them - [HUDI-68] : Automate demo steps on docker setup - Move hive queries from hive cli to beeline - Standardize on taking query input from text command files - Deltastreamer ingest, also does hive sync in a single step - Spark Incremental Query materialized as a derived Hive table using datasource - Fix flakiness in HDFS spin up and output comparison - Code cleanup around streamlining and loc reduction - Also fixed pom to not shade some hive classs in spark, to enable hive sync
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.