[FLINK-16627][json]Support ignore null fields when serializing into JSON #24430

RubyChou · 2024-03-04T04:25:16Z

What is the purpose of the change

This pull request introduces json.encode.ignore-null-fields for JSON formats which indicates whether to ignore null fields when serializing into JSON.

Brief change log

Add this option in JsonOptions.
Let JsonFormatFactorys and JsonRowDataSerializationSchemas be aware of this option.

Verifying this change

This change added tests and can be verified as follows:

Added test in JsonRowDataSerDeSchemaTest to check whether the result of serialization is expected.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (yes)
If yes, how is the feature documented? (docs)

flinkbot · 2024-03-04T04:31:35Z

CI report:

bed55cf Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

libenchao

@RubyChou generally looks good, I only left some minor comments about the styles and tests.

libenchao · 2024-03-05T07:11:48Z

docs/content.zh/docs/connectors/table/formats/debezium.md

+      <td>选填</td>
+      <td style="word-wrap: break-word;">false</td>
+      <td>Boolean</td>
+      <td>仅序列化非Null的列，默认情况下，会序列化所有列无论是否为Null。</td>


In Chinese document, we usually add spaces between english words, see more here: https://cwiki.apache.org/confluence/display/FLINK/Flink+Translation+Specifications

Suggested change

<td>仅序列化非Null的列，默认情况下，会序列化所有列无论是否为Null。</td>

<td>仅序列化非 Null 的列，默认情况下，会序列化所有列无论是否为 Null。</td>

libenchao · 2024-03-05T07:12:23Z

docs/content.zh/docs/connectors/table/formats/maxwell.md

+      <td>optional</td>
+      <td style="word-wrap: break-word;">false</td>
+      <td>Boolean</td>
+      <td>Encode only non-null fields. By default, all fields will be written.</td>


Suggested change

<td>Encode only non-null fields. By default, all fields will be written.</td>

<td>Encode only non-null fields. By default, all fields will be included.</td>

libenchao · 2024-03-05T07:15:12Z

...rmats/flink-json/src/test/java/org/apache/flink/formats/json/JsonRowDataSerDeSchemaTest.java

@@ -626,6 +635,62 @@ void testSerializationDecimalEncode() throws Exception {
        assertThat(scientificDecimalResult).isEqualTo(scientificDecimalJson);
    }

+    @TestTemplate
+    public void testSerDeMultiRowsWithNullValuesIgnored() throws Exception {


Since we have migrated to Junit5, public is no longer needed, see more here: https://docs.google.com/document/d/1514Wa_aNB9bJUen4xm5uiuXOooOJTtXqS_Jqk9KJitU/edit#heading=h.4iilyfqpoiw2

Suggested change

public void testSerDeMultiRowsWithNullValuesIgnored() throws Exception {

void testSerDeMultiRowsWithNullValuesIgnored() throws Exception {

libenchao · 2024-03-05T07:20:42Z

...flink-json/src/test/java/org/apache/flink/formats/json/canal/CanalJsonFormatFactoryTest.java

@@ -89,6 +90,7 @@ void testUserDefinedOptions() {
        options.put("canal-json.map-null-key.mode", "LITERAL");
        options.put("canal-json.map-null-key.literal", "nullKey");
        options.put("canal-json.encode.decimal-as-plain-number", "true");
+        options.put("canal-json.encode.ignore-null-fields", "true");


Why are you enable Canal json format to test "encode.ignore-null-fields" while leave others (debezium, maxwell, ogg) out?

RubyChou · 2024-03-05T09:29:46Z

hi @libenchao, I've resolved all comments above. Please help reviewing. Thanks.

libenchao

+1, merging.

…JSON Close apache#24430

libenchao · 2024-03-05T11:54:54Z

docs/content.zh/docs/connectors/table/formats/json.md

+    <tr>
+      <td><h5>json.encode.ignore-null-fields</h5></td>
+      <td>选填</td>
+      <td style="word-wrap: break-word;">false</td>


I think this option can be forwarded.

RubyChou added 3 commits March 4, 2024 12:36

[FLINK-16627][json]Support ignore null fields when serializing into JSON

712862e

Add docs for the feature.

736b0f4

Fix typo for ogg-json.timestamp-format.standard.

43667dc

RubyChou force-pushed the support-json-encode-ignore-null branch from 1df5347 to 43667dc Compare March 4, 2024 04:36

libenchao reviewed Mar 5, 2024

View reviewed changes

Resolve docs format issue and comments about tests.

2d3bba8

Add option tests for ogg and original json format.

e4fe979

libenchao approved these changes Mar 5, 2024

View reviewed changes

libenchao pushed a commit to libenchao/flink that referenced this pull request Mar 5, 2024

[FLINK-16627][json] Support ignore null fields when serializing into …

d422da4

…JSON Close apache#24430

libenchao reviewed Mar 5, 2024

View reviewed changes

Forward this option.

bed55cf

libenchao closed this in 93555a7 Mar 5, 2024

flinkbot added component=Formats component=TableSQL/Planner labels Apr 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-16627][json]Support ignore null fields when serializing into JSON #24430

[FLINK-16627][json]Support ignore null fields when serializing into JSON #24430

RubyChou commented Mar 4, 2024

flinkbot commented Mar 4, 2024 •

edited

libenchao left a comment

libenchao Mar 5, 2024

libenchao Mar 5, 2024

libenchao Mar 5, 2024

libenchao Mar 5, 2024

RubyChou commented Mar 5, 2024

libenchao left a comment

libenchao Mar 5, 2024

	<td>仅序列化非Null的列，默认情况下，会序列化所有列无论是否为Null。</td>
	<td>仅序列化非 Null 的列，默认情况下，会序列化所有列无论是否为 Null。</td>

	<td>Encode only non-null fields. By default, all fields will be written.</td>
	<td>Encode only non-null fields. By default, all fields will be included.</td>

	public void testSerDeMultiRowsWithNullValuesIgnored() throws Exception {
	void testSerDeMultiRowsWithNullValuesIgnored() throws Exception {

[FLINK-16627][json]Support ignore null fields when serializing into JSON #24430

[FLINK-16627][json]Support ignore null fields when serializing into JSON #24430

Conversation

RubyChou commented Mar 4, 2024

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

flinkbot commented Mar 4, 2024 • edited

CI report:

libenchao left a comment

Choose a reason for hiding this comment

libenchao Mar 5, 2024

Choose a reason for hiding this comment

libenchao Mar 5, 2024

Choose a reason for hiding this comment

libenchao Mar 5, 2024

Choose a reason for hiding this comment

libenchao Mar 5, 2024

Choose a reason for hiding this comment

RubyChou commented Mar 5, 2024

libenchao left a comment

Choose a reason for hiding this comment

libenchao Mar 5, 2024

Choose a reason for hiding this comment

flinkbot commented Mar 4, 2024 •

edited