Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove extra space for string value for postgresql datasource #65

Merged
merged 1 commit into from
Jan 30, 2022

Conversation

Nicole00
Copy link
Contributor

For Postgresql datasource, if source value's data type is CHAR(10) but the value length is less than 10, then data read by spark will be fill with spaces.
Example: data in postgres is "tom" and it's data type is CHAR(10), then after importing, the data in nebula will be "tom ", which is wrong. So we need to remove the extra spaces.

@Nicole00
Copy link
Contributor Author

Nicole00 commented Jan 29, 2022

This pr has been tested for client import mode and sst import mode for spark2.2, spark2.4 and spark3.0
postgresql:

create table exchange(id INT PRIMARY KEY, col1 CHAR(10), col2 CHAR(10), col3 CHAR(10), col4 CHAR(20), col5 CHAR(30), col6 CHAR(30), col7 CHAR(10));
insert into exchange values (2, 'jina', 'jina', '1000', '2021-12-12', '342786543', 'true', '1.2');

nebula:

create tag player(col1 string, col2 fixed_string(20), col3 int32, col4 date, col5 timestamp, col6 bool, col7 double);
create edge friend(col1 string, col2 fixed_string(20), col3 int32, col4 date, col5 timestamp, col6 bool, col7 double);

@codecov-commenter
Copy link

Codecov Report

Merging #65 (242d559) into master (f69bb65) will not change coverage.
The diff coverage is 84.00%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master      #65   +/-   ##
=========================================
  Coverage     55.14%   55.14%           
  Complexity       76       76           
=========================================
  Files            17       17           
  Lines          1342     1342           
  Branches        254      254           
=========================================
  Hits            740      740           
  Misses          473      473           
  Partials        129      129           
Impacted Files Coverage Δ
...m/vesoft/exchange/common/processor/Processor.scala 66.41% <84.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f69bb65...242d559. Read the comment docs.

@Aiee Aiee merged commit d003fad into vesoft-inc:master Jan 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants