Skip to content

NIFI-7410 Fix CLOB would be unreadable code in ExecuteSql or QueryDataBaseTable#4243

Closed
zhangchengk wants to merge 3 commits intoapache:masterfrom
zhangchengk:NIFI-7410
Closed

NIFI-7410 Fix CLOB would be unreadable code in ExecuteSql or QueryDataBaseTable#4243
zhangchengk wants to merge 3 commits intoapache:masterfrom
zhangchengk:NIFI-7410

Conversation

@zhangchengk
Copy link

https://issues.apache.org/jira/browse/NIFI-7410

Description of PR

fixes bug NIFI-7410

when ExecuteSql or QueryDataBaseTable processor trigger below code in JdbcCommon.java ,clob would be unreadable code(with Chinese character),because it used
InputStream is = clob.getAsciiStream(); and rec.put(i - 1, new String(buffer));

Update JdbcCommon.java when javaSqlType is CLOB or NCLOB in convertToAvroStream method, use the DataTypeUtils.toString(clob/nClob,(String) null, StandardCharsets.UTF_8)) method to get the String

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

  • [ √] Is there a JIRA ticket associated with this PR? Is it referenced
    in the commit message?

  • [ √] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

  • [ √] Has your PR been rebased against the latest commit within the target branch (typically master)?

  • [√ ] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

  • [ √] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
  • [ √] Have you written or updated unit tests to verify your changes?
  • Have you verified that the full build is successful on both JDK 8 and JDK 11?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
  • If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
  • If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

  • Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

1. Update JdbcCommon.java when javaSqlType is CLOB or NCLOB in convertToAvroStream method, use the DataTypeUtils.toString(clob/nClob,(String) null, StandardCharsets.UTF_8)) method to get the String
@zhangchengk zhangchengk changed the title NIFI-7410 NIFI-7410 fix CLOB would be unreadable code in ExecuteSql or QueryDataBaseTable May 1, 2020
@zhangchengk zhangchengk changed the title NIFI-7410 fix CLOB would be unreadable code in ExecuteSql or QueryDataBaseTable NIFI-7410 Fix CLOB would be unreadable code in ExecuteSql or QueryDataBaseTable May 1, 2020
Update JdbcCommon.java when javaSqlType is CLOB or NCLOB in convertToAvroStream method, use the CharacterStream rto read the value of CLOB
@mattyb149
Copy link
Contributor

Is there a way to add a unit test for this? Maybe inserting Chinese characters into a Derby table with a CLOB column and reading the results?

@zhangchengk
Copy link
Author

Is there a way to add a unit test for this? Maybe inserting Chinese characters into a Derby table with a CLOB column and reading the results?

Thanks for reply. I will add a unit test for this situation.

Add a unit test. validate if it's unreadable when the clob value is Chinese, Japanese, and Korean.
@zhangchengk
Copy link
Author

Is there a way to add a unit test for this? Maybe inserting Chinese characters into a Derby table with a CLOB column and reading the results?

Hi @mattyb149 I have added one unit test.Clould you help me to check the PR of the NIFI-7403 - PutSQL's transactions support #4239 if you have time? Thanks.

@pvillard31
Copy link
Contributor

Hey @mattyb149 - does it look good to you?

@pvillard31
Copy link
Contributor

Manually fixed the checkstyle violations and added the license to make the RAT check happy. Merging to main. Thanks @KuKuDeCheng (next time, please make sure to add the header and check the build with -Pcontrib-check)

@asfgit asfgit closed this in e18b4f0 Jul 17, 2020
Wastack pushed a commit to Wastack/nifi that referenced this pull request Jul 20, 2020
… convertToAvroStream method, use the DataTypeUtils.toString(clob/nClob,(String) null, StandardCharsets.UTF_8)) method to get the String

NIFI-7410 Update JdbcCommon.java when javaSqlType is CLOB or NCLOB in convertToAvroStream method, use the CharacterStream rto read the value of CLOB
NIFI-7410 Add a unit test. validate if it's unreadable when the clob value is Chinese, Japanese, and Korean.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes apache#4243.
ets pushed a commit to ets/nifi that referenced this pull request Oct 2, 2020
… convertToAvroStream method, use the DataTypeUtils.toString(clob/nClob,(String) null, StandardCharsets.UTF_8)) method to get the String

NIFI-7410 Update JdbcCommon.java when javaSqlType is CLOB or NCLOB in convertToAvroStream method, use the CharacterStream rto read the value of CLOB
NIFI-7410 Add a unit test. validate if it's unreadable when the clob value is Chinese, Japanese, and Korean.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes apache#4243.
driesva pushed a commit to driesva/nifi that referenced this pull request Mar 19, 2021
… convertToAvroStream method, use the DataTypeUtils.toString(clob/nClob,(String) null, StandardCharsets.UTF_8)) method to get the String

NIFI-7410 Update JdbcCommon.java when javaSqlType is CLOB or NCLOB in convertToAvroStream method, use the CharacterStream rto read the value of CLOB
NIFI-7410 Add a unit test. validate if it's unreadable when the clob value is Chinese, Japanese, and Korean.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes apache#4243.
krisztina-zsihovszki pushed a commit to krisztina-zsihovszki/nifi that referenced this pull request Jun 28, 2022
… convertToAvroStream method, use the DataTypeUtils.toString(clob/nClob,(String) null, StandardCharsets.UTF_8)) method to get the String

NIFI-7410 Update JdbcCommon.java when javaSqlType is CLOB or NCLOB in convertToAvroStream method, use the CharacterStream rto read the value of CLOB
NIFI-7410 Add a unit test. validate if it's unreadable when the clob value is Chinese, Japanese, and Korean.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes apache#4243.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants