Skip to content

[FSTORE-789] append - fix deltastreamer config #980

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 38 commits into from
Apr 5, 2023
Merged

[FSTORE-789] append - fix deltastreamer config #980

Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
2329257
remove unused import
davitbzh Jul 13, 2022
e38ca9c
Merge remote-tracking branch 'upstream/master'
davitbzh Jul 13, 2022
10ad8b9
Merge remote-tracking branch 'upstream/master'
davitbzh Jul 13, 2022
16b2e52
Merge remote-tracking branch 'upstream/master'
davitbzh Jul 15, 2022
63b812b
Merge remote-tracking branch 'upstream/master'
davitbzh Jul 18, 2022
9fdd925
Merge remote-tracking branch 'upstream/master'
davitbzh Aug 4, 2022
3be3d58
Merge remote-tracking branch 'upstream/master'
davitbzh Sep 7, 2022
6873412
Merge remote-tracking branch 'upstream/master'
davitbzh Sep 14, 2022
c033dad
Merge remote-tracking branch 'upstream/master'
davitbzh Sep 15, 2022
1b3d076
Merge remote-tracking branch 'upstream/master'
davitbzh Sep 15, 2022
738ce21
Merge remote-tracking branch 'upstream/master'
davitbzh Sep 19, 2022
afe2459
Merge remote-tracking branch 'upstream/master'
davitbzh Sep 20, 2022
f972c0d
Merge remote-tracking branch 'upstream/master'
davitbzh Sep 22, 2022
3145a76
Merge remote-tracking branch 'upstream/master'
davitbzh Sep 26, 2022
fc39ad2
Merge remote-tracking branch 'upstream/master'
davitbzh Sep 28, 2022
b750dd4
Merge remote-tracking branch 'upstream/master'
davitbzh Oct 3, 2022
941d0e3
Merge remote-tracking branch 'upstream/master'
davitbzh Oct 3, 2022
3fbbe22
Merge remote-tracking branch 'upstream/master'
davitbzh Oct 11, 2022
75b9d89
Merge remote-tracking branch 'upstream/master'
davitbzh Oct 25, 2022
ac281ff
Merge remote-tracking branch 'upstream/master'
davitbzh Oct 26, 2022
2a493ea
Merge remote-tracking branch 'upstream/master'
davitbzh Nov 2, 2022
5a70a55
Merge remote-tracking branch 'upstream/master'
davitbzh Nov 6, 2022
20812f4
Merge remote-tracking branch 'upstream/master'
davitbzh Nov 8, 2022
435faec
Merge remote-tracking branch 'upstream/master'
davitbzh Nov 8, 2022
5d378a1
Merge remote-tracking branch 'upstream/master'
davitbzh Nov 8, 2022
dad216f
Merge remote-tracking branch 'upstream/master'
davitbzh Nov 14, 2022
866d575
Merge remote-tracking branch 'upstream/master'
davitbzh Nov 16, 2022
ca44cda
Merge remote-tracking branch 'upstream/master'
davitbzh Nov 25, 2022
42c8771
Merge remote-tracking branch 'upstream/master'
davitbzh Nov 30, 2022
3b4466b
Merge remote-tracking branch 'upstream/master'
davitbzh Dec 4, 2022
1b7245b
Merge remote-tracking branch 'upstream/master'
davitbzh Dec 5, 2022
eba5166
Merge remote-tracking branch 'upstream/master'
davitbzh Dec 13, 2022
5ba28f6
Merge remote-tracking branch 'upstream/master'
davitbzh Jan 9, 2023
04f0cc8
Merge remote-tracking branch 'upstream/master'
davitbzh Jan 24, 2023
55646b7
Merge remote-tracking branch 'upstream/master'
davitbzh Feb 28, 2023
8d91ad9
Merge remote-tracking branch 'upstream/master'
davitbzh Mar 29, 2023
25b78e3
Merge remote-tracking branch 'upstream/master'
davitbzh Apr 5, 2023
7e4eb4c
fix delta streamer config
davitbzh Apr 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions java/spark/src/main/java/com/logicalclocks/hsfs/spark/engine/hudi/HudiEngine.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,16 +112,16 @@ public class HudiEngine {
protected static final String COMMIT_METADATA_KEYPREFIX_OPT_KEY = "hoodie.datasource.write.commitmeta.key.prefix";
protected static final String DELTASTREAMER_CHECKPOINT_KEY = "deltastreamer.checkpoint.key";
protected static final String INITIAL_CHECKPOINT_STRING = "initialCheckPointString";
protected static final String FEATURE_GROUP_SCHEMA = "StreamFeatureGroup.avroSchema";
protected static final String FEATURE_GROUP_SCHEMA = "com.logicalclocks.hsfs.spark.StreamFeatureGroup.avroSchema";
Copy link
Contributor

@moritzmeister moritzmeister Apr 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't quite understand why these ones have to be prefixed, aren't these just keys to get our settings from the config?

Copy link
Contributor Author

@davitbzh davitbzh Apr 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK this one doesn't need to be prefixed to have full path. it doesn't harm either. should I remove it?

protected static final String FEATURE_GROUP_ENCODED_SCHEMA =
"StreamFeatureGroup.encodedAvroSchema";
"com.logicalclocks.hsfs.spark.StreamFeatureGroup.encodedAvroSchema";
protected static final String FEATURE_GROUP_COMPLEX_FEATURES =
"StreamFeatureGroup.complexFeatures";
protected static final String KAFKA_SOURCE = "DeltaStreamerKafkaSource";
"com.logicalclocks.hsfs.spark.StreamFeatureGroup.complexFeatures";
protected static final String KAFKA_SOURCE = "com.logicalclocks.hsfs.spark.engine.hudi.DeltaStreamerKafkaSource";
protected static final String SCHEMA_PROVIDER =
"DeltaStreamerSchemaProvider";
"com.logicalclocks.hsfs.spark.engine.hudi.DeltaStreamerSchemaProvider";
protected static final String DELTA_STREAMER_TRANSFORMER =
"DeltaStreamerTransformer";
"com.logicalclocks.hsfs.spark.engine.hudi.DeltaStreamerTransformer";
protected static final String DELTA_SOURCE_ORDERING_FIELD_OPT_KEY = "sourceOrderingField";

protected static final String MIN_SYNC_INTERVAL_SECONDS = "minSyncIntervalSeconds";
Expand Down