New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FLINK-31663][table] Add ARRAY_EXCEPT function #22588
Conversation
cbbf0c1
to
5e9e678
Compare
} | ||
|
||
List<Object> list = new ArrayList(); | ||
Set<Object> arrayOneSet = new HashSet<>(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we shouldn't depend on java objects equality., but rather use a generated comparator:
equalityEvaluator =
context.createEvaluator(
$("element1").isEqual($("element2")),
DataTypes.BOOLEAN(),
DataTypes.FIELD("element1", dataType.notNull().toInternal()),
DataTypes.FIELD("element2", dataType.notNull().toInternal()));
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated to use arrayContains
comparator
if ((arrayTwo == null && !seen.contains(element)) | ||
|| (element == null && !isNullPresent) | ||
|| (element != null | ||
&& !seen.contains(element) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we should not depend on java equals
and hashcode
What is the purpose of the change
This is an implementation of ARRAY_EXCEPT
Brief change log
ARRAY_EXCEPT for Table API and SQL
See also https://spark.apache.org/docs/latest/api/sql/index.html#array_except
Verifying this change
Does this pull request potentially affect one of the following parts:
@Public(Evolving)
: (yes)Documentation