Skip to content

[AURON #1849] Introduce native json deserializer#2112

Merged
Tartarus0zm merged 3 commits intoapache:masterfrom
Tartarus0zm:flink-json-deserializer
Mar 24, 2026
Merged

[AURON #1849] Introduce native json deserializer#2112
Tartarus0zm merged 3 commits intoapache:masterfrom
Tartarus0zm:flink-json-deserializer

Conversation

@Tartarus0zm
Copy link
Contributor

Which issue does this PR close?

Closes #1849

Rationale for this change

  • auron flink kafka connector support json

What changes are included in this PR?

  • add json_deserializer to deserialize JSON data from Kafka
  • modify kafka_scan_exec to supports selecting different deserializers based on the data format

Are there any user-facing changes?

  • No

How was this patch tested?

  • No kafka environment, test via rust UT for json_deserializer

@Tartarus0zm Tartarus0zm linked an issue Mar 23, 2026 that may be closed by this pull request
@Tartarus0zm Tartarus0zm requested a review from richox March 23, 2026 13:14
output_arrays.push(new_null_array(field.data_type(), array_ref.len()));
} else {
output_arrays.push(
datafusion_ext_commons::arrow::cast::cast(&array_ref, field.data_type())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use ? instead of .except(..)

pub fn new(
output_schema: SchemaRef,
nested_msg_mapping: &HashMap<String, String>,
) -> datafusion::error::Result<Self> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can be simplified with use datafusion::error::Result

@Tartarus0zm Tartarus0zm force-pushed the flink-json-deserializer branch from cb49471 to 2a4ded9 Compare March 24, 2026 02:54
@Tartarus0zm
Copy link
Contributor Author

@richox I've fixed the issue with the comments. Please take another look.

@Tartarus0zm Tartarus0zm merged commit f0953e1 into apache:master Mar 24, 2026
123 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Introduce native json deserializer

2 participants