Skip to content
This repository has been archived by the owner on Oct 18, 2021. It is now read-only.

import data inside kafka value to nebula #157

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

sworduo
Copy link

@sworduo sworduo commented Sep 28, 2021

This is the resolution for issue https://github.com/vesoft-inc/nebula-spark-utils/issues/130 (import data from Kafka to Nebula). In this update, it is supported by nebula-exchange to parse data from the value field of Kafka and import which to Nebula. It's worth noting that other fields included in Kafka like offset, key,etc are abandoned. Meanwhile, since Kafka is streaming data, it's impossible to switch data source once Kafka is chosen, which means the tag/edge defined in configuration can only be parsed from Kafka. Hence, the Kafka config is defined independently instead of indicated inside the tag/edge config. In this case, all tag/edge share the same Kafka config. More details can be found in the accompanying README-CN.md.

@CLAassistant
Copy link

CLAassistant commented Sep 28, 2021

CLA assistant check
All committers have signed the CLA.

@wey-gu
Copy link
Contributor

wey-gu commented Sep 28, 2021

Thank you so much @sworduo, this PR makes real-world Kafka streaming source Usability to the next level.

@Nicole00 🎉

@Nicole00
Copy link
Contributor

Thanks for your pr to support the parsing for kafka‘s value.
This pr changes the architecture of Exchange showed in doc https://docs.nebula-graph.com.cn/2.5.1/nebula-exchange/about-exchange/ex-ug-what-is-exchange/, can we just modify the StreamingReader to parse the kafka's value to DataFrame?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants