You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hive does not support CDC sink, so we could give a option to regard all changelog data as insert-only data. So we can insert into hive from canal-json,rowkind as a hive schema field.
e.g.
CREATE TABLE mysql_inlong_0505 (
`id` BIGINT,
`name` STRING,
`op_type` STRING METADATA FROM 'meta.op-type',
`log_ts` TIMESTAMP_LTZ(3) METADATA FROM 'meta.op_ts' VIRTUAL,
PRIMARY KEY(`id`) NOT ENFORCED
) with (
'connector' = 'mysql-cdc-inlong',
'hostname' = 'localhost',
'username' = 'root',
'password' = '123456',
'database-name' = 'test',
'table-name' = 'mysql_inlong_0505'
);
CREATE TABLE hive_table (
`id` BIGINT,
`name` STRING,
`op_type` STRING,
) PARTITIONED BY (dt string, hr string, me string) STORED AS parquet;
INSERT INTO hive_table
SELECT user_id, order_amount, op_type, DATE_FORMAT(log_ts, 'yyyy-MM-dd'), DATE_FORMAT(log_ts, 'HH'), DATE_FORMAT(log_ts, 'mm')
FROM kafka_table;
healchow
changed the title
[Improve][Sort] option to produce insert-only stream in canal json format
[Improve][Sort] Support to produce insert-only stream in canal JSON format
Apr 29, 2022
healchow
changed the title
[Improve][Sort] Support to produce insert-only stream in canal JSON format
[Improve][Sort] Import data in Canal JSON format from Kafka into Hive in insert-only mode
Apr 29, 2022
thexiay
changed the title
[Improve][Sort] Import data in Canal JSON format from Kafka into Hive in insert-only mode
[Improve][Sort] Import all changelog mode data ingest into Hive
May 6, 2022
Description
Hive does not support CDC sink, so we could give a option to regard all changelog data as insert-only data. So we can insert into hive from canal-json,rowkind as a hive schema field.
e.g.
InLong Component
InLong Sort
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: