Skip to content

[HOP-Coalesce] - Fixes bug in Coalesce plugin#1635

Merged
hansva merged 6 commits intoapache:masterfrom
weiqingping:master
Aug 17, 2022
Merged

[HOP-Coalesce] - Fixes bug in Coalesce plugin#1635
hansva merged 6 commits intoapache:masterfrom
weiqingping:master

Conversation

@weiqingping
Copy link

I found that the result always takes the first column regardless of whether the first column is null or not

@weiqingping weiqingping changed the title [HOP-Coalesce] - Fixes bug in SessionManager` [HOP-Coalesce] - Fixes bug in Coalesce plugin Aug 16, 2022
@hansva
Copy link
Contributor

hansva commented Aug 16, 2022

Can you give a sample of what this should fix?
We have a test on this and as far as I can see the result is correct.
image

@weiqingping
Copy link
Author

the data's input is cvs. but the log show that the target field is null

result

@weiqingping
Copy link
Author

data

first

second

thrid

transform-flow

result

@hansva
Copy link
Contributor

hansva commented Aug 16, 2022

Confirmed, so the problem actually is the binary-string types.
Thank you @weiqingping !

@hansva
Copy link
Contributor

hansva commented Aug 16, 2022

Just wanted to add that this breaks every other case except binary-string, but you noticed it too

@weiqingping
Copy link
Author

Confirmed, so the problem actually is the binary-string types. Thank you @weiqingping !

I get it. I modified it again

@hansva
Copy link
Contributor

hansva commented Aug 16, 2022

@weiqingping,

You will have to revert the Mysql change though, the IValueMeta.TYPE_DATE is a Java Date and contains date + Time.
This then translates to a DateTime in Mysql.

The difference between IValueMeta.TYPE_DATE and IValueMeta.TYPE_TIMESTAMP is the objects behind it
Java Date vs Java Time

@weiqingping
Copy link
Author

@weiqingping,

You will have to revert the Mysql change though, the IValueMeta.TYPE_DATE is a Java Date and contains date + Time. This then translates to a DateTime in Mysql.

The difference between IValueMeta.TYPE_DATE and IValueMeta.TYPE_TIMESTAMP is the objects behind it Java Date vs Java Time

the problem is that the db's field type is Date but it show DateTime in the hop's ui.
1660644458373

@hansva
Copy link
Contributor

hansva commented Aug 16, 2022

The generated DDL does not show the actual DDL of the table, it shows how we would create the table.
And because we have no date-only type we suggest DateTime indeed.

A Hop "Date" is DateTime and if we select a MySQL Date from a table we have to convert it to a Hop Date and will contain time again (00:00:00 timestamp in the backend). I know it's a bit annoying but we will have more errors if we suggest the Date type because people will wonder where their time has gone.

Edit:
But I would honestly never suggest creating tables from the UI, your DDL should be managed outside of Hop

@weiqingping
Copy link
Author

The generated DDL does not show the actual DDL of the table, it shows how we would create the table. And because we have no date-only type we suggest DateTime indeed.

A Hop "Date" is DateTime and if we select a MySQL Date from a table we have to convert it to a Hop Date and will contain time again (00:00:00 timestamp in the backend). I know it's a bit annoying but we will have more errors if we suggest the Date type because people will wonder where their time has gone.

Edit: But I would honestly never suggest creating tables from the UI, your DDL should be managed outside of Hop

OK ,I understand. thank you @hansva

@hansva hansva merged commit 2f67f90 into apache:master Aug 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants