[SYSTEMDS-439] Builtins for set operations on vectors

Illustration of the implementation:

X = matrix("1   2
            3.1 4", rows=2, cols=2)
Y = matrix("3.1 4
            5   6", rows=2, cols=2)

union(X, Y): Union of the sets X and Y

  1   2
  3.1 4
  5   6

setdiff(X, Y): Set difference between X, and Y, with elements in X
but not in Y.

  1 2

symmetricDifference(X,Y): Set difference between X, and Y, with elements
in X and Y but not in both.

  1 2
  5 6

unique(X): Unique elements of the set X

  1 2
  3.1 4

Future work:
also to support string elements. These operations are helpful for bridging
the gap between the relational and linear algebra.

Resolves SYSTEMDS-440, SYSTEMDS-441, SYSTEMDS-442, SYSTEMDS-3183.
Closes apache#1479.

Co-authored-by: David Fleischhacker <david.fleischhacker@student.tugraz.at>
Co-authored-by: Joachim Dunkel <dunkel@student.tugraz.at>

Author: 2 people

docs/site/builtins-reference.md

@@ -71,15 +71,19 @@ limitations under the License.
     * [`outlier`-Function](#outlier-function)
     * [`pnmf`-Function](#pnmf-function)
     * [`scale`-Function](#scale-function)
+    * [`setdiff`-Function](#setdiff-function)
     * [`sherlock`-Function](#sherlock-function)
     * [`sherlockPredict`-Function](#sherlockPredict-function)
     * [`sigmoid`-Function](#sigmoid-function)
     * [`slicefinder`-Function](#slicefinder-function)
     * [`smote`-Function](#smote-function)
     * [`steplm`-Function](#steplm-function)
+    * [`symmetricDifference`-Function](#symmetricdifference-function)
     * [`tomekLink`-Function](#tomekLink-function)
     * [`toOneHot`-Function](#toOneHOt-function)
     * [`tSNE`-Function](#tSNE-function)
+    * [`union`-Function](#union-function)
+    * [`unique`-Function](#unique-function)
     * [`winsorize`-Function](#winsorize-function)
     * [`xgboost`-Function](#xgboost-function)
 
@@ -1823,6 +1827,36 @@ scale=TRUE;
 Y= scale(X,center,scale)
 ```
 
+## `setdiff`-Function
+
+The `setdiff`-function returns the values of X that are not in Y.
+
+### Usage
+
+```r
+setdiff(X, Y)
+```
+
+### Arguments
+
+| Name | Type   | Default  | Description |
+| :--- | :----- | -------- | :---------- |
+| X    | Matrix[Double] | required | input vector|
+| Y    | Matrix[Double] | required | input vector|
+
+### Returns
+
+| Type   | Description |
+| :----- | :---------- |
+| Matrix[Double] | values of X that are not in Y.|
+
+### Example
+
+```r
+X = matrix("1 2 3 4", rows = 4, cols = 1)
+Y = matrix("2 3", rows = 2, cols = 1)
+R = setdiff(X = X, Y = Y)
+```
 
 ## `sherlock`-Function
 
@@ -2107,6 +2141,37 @@ y = X %*% rand(rows = ncol(X), cols = 1)
 [C, S] = steplm(X = X, y = y, icpt = 1);
 ```
 
+## `symmetricDifference`-Function
+
+The `symmetricDifference`-function returns the symmetric difference of the two input vectors.
+This is done by calculating the `setdiff` (nonsymmetric) between `union` and `intersect` of the two input vectors.
+
+### Usage
+
+```r
+symmetricDifference(X, Y)
+```
+
+### Arguments
+
+| Name | Type   | Default  | Description |
+| :--- | :----- | -------- | :---------- |
+| X    | Matrix[Double] | required | input vector|
+| Y    | Matrix[Double] | required | input vector|
+
+### Returns
+
+| Type   | Description |
+| :----- | :---------- |
+| Matrix[Double] | symmetric difference of the input vectors |
+
+### Example
+
+```r
+X = matrix("1 2 3.1", rows = 3, cols = 1)
+Y = matrix("3.1 4", rows = 2, cols = 1)
+R = symmetricDifference(X = X, Y = Y)
+```
 
 ## `tomekLink`-Function
 
@@ -2212,6 +2277,66 @@ X = rand(rows = 100, cols = 10, min = -10, max = 10))
 Y = tSNE(X)
 ```
 
+## `union`-Function
+
+The `union`-function combines all rows from both input vectors and removes all duplicate rows by calling `unique` on the resulting vector.
+
+### Usage
+
+```r
+union(X, Y)
+```
+
+### Arguments
+
+| Name | Type   | Default  | Description |
+| :--- | :----- | -------- | :---------- |
+| X    | Matrix[Double] | required | input vector|
+| Y    | Matrix[Double] | required | input vector|
+
+### Returns
+
+| Type   | Description |
+| :----- | :---------- |
+| Matrix[Double] | the union of both input vectors.|
+
+### Example
+
+```r
+X = matrix("1 2 3 4", rows = 4, cols = 1)
+Y = matrix("3 4 5 6", rows = 4, cols = 1)
+R = union(X = X, Y = Y)
+```
+
+## `unique`-Function
+
+The `unique`-function returns a set of unique rows from a given input vector.
+
+### Usage
+
+```r
+unique(X)
+```
+
+### Arguments
+
+| Name | Type   | Default  | Description |
+| :--- | :----- | -------- | :---------- |
+| X    | Matrix[Double] | required | input vector|
+
+### Returns
+
+| Type   | Description |
+| :----- | :---------- |
+| Matrix[Double] | a set of unique values from the input vector |
+
+### Example
+
+```r
+X = matrix("1 3.4 7 3.4 -0.9 8 1", rows = 7, cols = 1)
+R = unique(X = X)
+```
+
 ## `winsorize`-Function
 
 The `winsorize`-function removes outliers from the data. It does so by computing upper and lower quartile range

scripts/builtin/intersect.dml

@@ -39,12 +39,14 @@
 m_intersect = function(Matrix[Double] X, Matrix[Double] Y)
   return(Matrix[Double] R)
 {
-  # compute indicator vector of intersection output
-  X = (table(X, 1) != 0) 
-  Y = (table(Y, 1) != 0)
-  n = min(nrow(X), nrow(Y))
-  I = X[1:n,] * Y[1:n,]
+    X = unique(X);
+    Y = unique(Y);
 
-  # reconstruct integer values and create output
-  R = removeEmpty(target=seq(1,n), margin="rows", select=I)
+    combined = rbind(X, Y);
+
+    combined = order(target=combined, by=1, decreasing=FALSE, index.return=FALSE);
+    temp = combined[1:nrow(combined)-1,] != combined[2:nrow(combined),];
+    mask = rbind(matrix(1, rows = 1, cols = 1), rowSums(temp));
+
+    R = removeEmpty(target = combined, margin = "rows", select = !mask);
 }

scripts/builtin/setdiff.dml

@@ -0,0 +1,53 @@
+#-------------------------------------------------------------
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#-------------------------------------------------------------
+
+# Builtin function that implements difference operation on vectors
+
+# INPUT PARAMETERS:
+# ---------------------------------------------------------------------------------------------
+# NAME            TYPE    DEFAULT     MEANING
+# ---------------------------------------------------------------------------------------------
+# X               Matrix  ---         input vector
+# ---------------------------------------------------------------------------------------------
+# Y               Matrix  ---         input vector
+# ---------------------------------------------------------------------------------------------
+
+# Output(s)
+# ---------------------------------------------------------------------------------------------
+# NAME            TYPE    DEFAULT      MEANING
+# ---------------------------------------------------------------------------------------------
+# R               Matrix  ---         vector with all elements that are present in X but not in Y
+
+
+setdiff = function(Matrix[double] X, Matrix[double] Y)
+    return (matrix[double] R)
+{
+    common = intersect(X, Y);
+    X = unique(X);
+    combined = rbind(X, common);
+    combined = order(target=combined, by=1, decreasing=FALSE, index.return=FALSE);
+    temp = combined[1:nrow(combined)-1,] != combined[2:nrow(combined),];
+    mask1 = rbind(rowSums(temp), matrix(1, rows=1, cols=1));
+    mask2 = rbind(matrix(1, rows = 1, cols = 1), rowSums(temp));
+
+    mask = mask1 & mask2;
+    R = removeEmpty(target = combined, margin = "rows", select = mask);
+}

scripts/builtin/symmetricDifference.dml

@@ -0,0 +1,44 @@
+#-------------------------------------------------------------
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#-------------------------------------------------------------
+
+# Builtin function that implements symmetric difference set-operation on vectors
+
+# INPUT PARAMETERS:
+# ---------------------------------------------------------------------------------------------
+# NAME            TYPE    DEFAULT     MEANING
+# ---------------------------------------------------------------------------------------------
+# X               Matrix  ---         input vector
+# ---------------------------------------------------------------------------------------------
+# Y               Matrix  ---         input vector
+# ---------------------------------------------------------------------------------------------
+
+# Output(s)
+# ---------------------------------------------------------------------------------------------
+# NAME            TYPE    DEFAULT      MEANING
+# ---------------------------------------------------------------------------------------------
+# R               Matrix  ---         vector with all elements in X and Y but not in both
+
+
+symmetricDifference = function(Matrix[Double] X, Matrix[Double] Y)
+    return (matrix[double] R)
+{
+    R = setdiff(union(X,Y), intersect(X,Y))
+}

scripts/builtin/union.dml

@@ -0,0 +1,45 @@
+#-------------------------------------------------------------
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#-------------------------------------------------------------
+
+# Builtin function that implements union operation on vectors
+
+# INPUT PARAMETERS:
+# ---------------------------------------------------------------------------------------------
+# NAME            TYPE    DEFAULT     MEANING
+# ---------------------------------------------------------------------------------------------
+# X               Matrix  ---         input vector
+# ---------------------------------------------------------------------------------------------
+# Y               Matrix  ---         input vector
+# ---------------------------------------------------------------------------------------------
+
+# Output(s)
+# ---------------------------------------------------------------------------------------------
+# NAME            TYPE    DEFAULT      MEANING
+# ---------------------------------------------------------------------------------------------
+# R               Matrix  ---         matrix with all unique rows existing in X and Y
+
+
+union = function(Matrix[Double] X, Matrix[Double] Y)
+    return (matrix[double] R)
+{
+    combined = rbind(X,Y);
+    R = unique(combined);
+}

scripts/builtin/unique.dml

@@ -0,0 +1,48 @@
+#-------------------------------------------------------------
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#-------------------------------------------------------------
+
+# Builtin function that implements unique operation on vectors
+
+# INPUT PARAMETERS:
+# ---------------------------------------------------------------------------------------------
+# NAME            TYPE    DEFAULT     MEANING
+# ---------------------------------------------------------------------------------------------
+# X               Matrix  ---         input vector
+# ---------------------------------------------------------------------------------------------
+
+# Output(s)
+# ---------------------------------------------------------------------------------------------
+# NAME            TYPE    DEFAULT      MEANING
+# ---------------------------------------------------------------------------------------------
+# R               Matrix  ---         matrix with only unique rows
+
+unique = function(matrix[double] X)
+    return (matrix[double] R) {
+    if(nrow(X) > 1) {
+        X_sorted = order(target=X, by=1, decreasing=FALSE, index.return=FALSE);
+        temp = X_sorted[1:nrow(X_sorted)-1,] != X_sorted[2:nrow(X_sorted),];
+        mask = rbind(matrix(1, rows = 1, cols = 1), rowSums(temp));
+        R = removeEmpty(target = X_sorted, margin = "rows", select = mask);
+    }
+    else {
+        R = X
+    }
+}