Skip to content

[ISSUE-11199] Pipe: fix NPE when casting from null values in InsertRowNode deserialization#11200

Merged
SteveYurongSu merged 2 commits intoapache:masterfrom
VGalaxies:fix-cast-null
Sep 24, 2023
Merged

[ISSUE-11199] Pipe: fix NPE when casting from null values in InsertRowNode deserialization#11200
SteveYurongSu merged 2 commits intoapache:masterfrom
VGalaxies:fix-cast-null

Conversation

@VGalaxies
Copy link
Copy Markdown
Contributor

@VGalaxies VGalaxies commented Sep 22, 2023

Description

Fix #11199, allow casting from null when deserializing InsertRowNode.


For this insert statement:

insert into root.sg.d1(time, s0, s1) values (3, null, 25.34);

The source-side pipe stream processing engine will convert it into the following RPC sent to the target-side:

TPipeTransferReq(version:1, type:6, body:[0, 0, 0, 0, 0, 0, 0, 1, 0, 14, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 10, 114, 111, 111, 116, 46, 115, 103, 46, 100, 49, 0, 0, 0, 2, 1, 0, 0, 0, 2, 115, 48, 3, 8, 7, 0, 0, 0, 0, 0, 0, 0, 2, 115, 49, 3, 8, 7, 0, 0, 0, 0, -2, 3, 65, -54, -72, 82, 0, 1, 0, 0, 0, 1, 49, 0, 0, 0, 0, 0, 0, 0, 0])

The focus here is on the deserialization logic of its body.

The deserialization logic is in org.apache.iotdb.db.pipe.connector.payload.evolvable.request.PipeTransferTabletBatchReq#fromTPipeTransferReq.

binary explanation remark
0, 0, 0, 0 binary reqs size
0, 0, 0, 1 insert node reqs size
0, 14 plan node type InsertRowNode
0, 0, 0, 0, 0, 0, 0, 3 time
0, 0, 0, 10, 114, 111, 111, 116, 46, 115, 103, 46, 100, 49 device path root.sg.d1
0, 0, 0, 2 measurement size
1 has schema
0, 0, 0, 2, 115, 48 first measurement s0
3 measurement type float
8 measurement encoding gorilla
7 measurement compressor lz4
0, 0, 0, 0 measurement props size
0, 0, 0, 2, 115, 49 second measurement s1
3 measurement type float
8 measurement encoding gorilla
7 measurement compressor lz4
0, 0, 0, 0 measurement props size
-2 first value type null
3 second value type float
65, -54, -72, 82 second value 25.34
0 is need infer type false
1 is aligned true
0, 0, 0, 1, 49 plan node id 1
0, 0, 0, 0 child count
0, 0, 0, 0 tablet reqs size

So, it can be observed that after deserialization, the InsertRowStatement object has dataTypes member with values of [null, TSDataType.FLOAT]. This leads to a NullPointerException in org.apache.iotdb.db.utils.CommonUtils#checkCanCastType. Similarly, the values member has values of [null, 25.34], also leads to NPE in org.apache.iotdb.db.utils.CommonUtils#castValue.

One natural idea is to check in checkCanCastType and castValue whether the source type or source value is null. If so, allow further deserialization (i.e., pass type checking or return a null value).

Since the utility methods checkCanCastType and castValue appear to be used only by InsertRowStatement and InsertTabletStatement, the logic changes resulting from the above modification are manageable.


This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods.
  • added or updated version, license, or notice information
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
    for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR

Copy link
Copy Markdown

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, this is your first pull request in IoTDB project. Thanks for your contribution! IoTDB will be better because of you.

@SteveYurongSu SteveYurongSu self-assigned this Sep 22, 2023
Copy link
Copy Markdown
Contributor

@MiniSho MiniSho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

YOU MAKE PIPE GREAT AGAIN ! 🚀

@VGalaxies VGalaxies requested a review from MiniSho September 23, 2023 02:44
@SteveYurongSu SteveYurongSu changed the title Pipe: fix NPE when casting from null values in InsertRowNode deserialization [ISSUE-11199] Pipe: fix NPE when casting from null values in InsertRowNode deserialization Sep 24, 2023
Copy link
Copy Markdown
Member

@SteveYurongSu SteveYurongSu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job. Please cheery pick it to rel/1.2 :)

@SteveYurongSu SteveYurongSu merged commit a9093c2 into apache:master Sep 24, 2023
VGalaxies added a commit to VGalaxies/iotdb that referenced this pull request Sep 24, 2023
…wNode deserialization (apache#11200)

* fix: allow casting from null when deserializing InsertRowNode
VGalaxies added a commit to VGalaxies/iotdb that referenced this pull request Sep 24, 2023
…s in InsertRowNode deserialization (apache#11200)

* fix: allow casting from null when deserializing InsertRowNode
VGalaxies added a commit to VGalaxies/iotdb that referenced this pull request Sep 24, 2023
… in InsertRowNode deserialization (apache#11200)

* fix: allow casting from null when deserializing InsertRowNode
SteveYurongSu pushed a commit that referenced this pull request Sep 24, 2023
… in InsertRowNode deserialization (#11200) (#11205)

* fix: allow casting from null when deserializing InsertRowNode
@VGalaxies VGalaxies deleted the fix-cast-null branch September 25, 2023 03:37
HTHou added a commit that referenced this pull request Sep 26, 2023
…l values in InsertRowNode deserialization (#11200) (#11205)"

This reverts commit 23e889c.
SteveYurongSu added a commit that referenced this pull request Sep 27, 2023
…l values in InsertRowNode deserialization (#11200) (#11205)"

This reverts commit 23e889c.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] NPE when inserting null values for pipe data synchronization

3 participants