Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-11313][SQL] implement cogroup on DataSets #9279

Closed
wants to merge 1 commit into from
Closed

[SPARK-11313][SQL] implement cogroup on DataSets #9279

wants to merge 1 commit into from

Conversation

cloud-fan
Copy link
Contributor

No description provided.

marmbrus
marmbrus reviewed Oct 26, 2015
View reviewed changes
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala
@@ -513,3 +513,16 @@ case class MapGroups[K, T, U](
override def missingInput: AttributeSet = AttributeSet.empty
}

case class CoGroup2[K, T, U, V](
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding a factory like in the other cases to simplify implicit passing.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also I'd probably just call it cogroup as we'll probably use a single variadic operator if we decide to do more than 2.

@marmbrus
Copy link
Contributor

Looking pretty good so far! Let's wait to add cogroup for more than 2 datasets.

@SparkQA
Copy link

SparkQA commented Oct 26, 2015

Test build #44351 has finished for PR 9279 at commit d4b6920.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * case class CoGroup2[K, T, U, V](\n * case class CoGroup2[K, T, U, V](\n

@cloud-fan cloud-fan force-pushed the cogroup branch 2 times, most recently from 1c7f4c0 to 2f1cef0 Compare October 27, 2015 11:12
@SparkQA
Copy link

SparkQA commented Oct 27, 2015

Test build #44424 has finished for PR 9279 at commit 1c7f4c0.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Oct 27, 2015

Test build #44425 has finished for PR 9279 at commit 2f1cef0.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * case class CoGroup[K, R](\n * case class CoGroup[K, T, U, R](\n

@cloud-fan cloud-fan changed the title [SPARK-11313][SQL][WIP] implement cogroup on DataSets [SPARK-11313][SQL] implement cogroup on DataSets Oct 27, 2015
@SparkQA
Copy link

SparkQA commented Oct 27, 2015

Test build #44426 has finished for PR 9279 at commit 2fb01ec.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * case class CoGroup[K, R](\n * case class CoGroup[K, T, U, R](\n

@cloud-fan
Copy link
Contributor Author

cc @marmbrus

@cloud-fan
Copy link
Contributor Author

retest this please.

@SparkQA
Copy link

SparkQA commented Oct 27, 2015

Test build #44442 has finished for PR 9279 at commit 2fb01ec.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * case class CoGroup[K, R](\n * case class CoGroup[K, T, U, R](\n

@cloud-fan cloud-fan mentioned this pull request Oct 28, 2015
Closed
@cloud-fan
Copy link
Contributor Author

will open it again when we need to support cogroup on more than 2 datasets.

@cloud-fan cloud-fan closed this Oct 28, 2015
ghost pushed a commit to dbtsai/spark that referenced this pull request Oct 28, 2015
@cloud-fan @marmbrus
[SPARK-11313][SQL] implement cogroup on DataSets (support 2 datasets)
075ce49 
A simpler version of apache#9279, only support 2 datasets.

Author: Wenchen Fan <wenchen@databricks.com>

Closes apache#9324 from cloud-fan/cogroup2.
kiszk pushed a commit to kiszk/spark-gpu that referenced this pull request Dec 26, 2015
@cloud-fan @kiszk
[SPARK-11313][SQL] implement cogroup on DataSets (support 2 datasets)
684d5fa 
A simpler version of apache/spark#9279, only support 2 datasets.

Author: Wenchen Fan <wenchen@databricks.com>

Closes #9324 from cloud-fan/cogroup2.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@cloud-fan @marmbrus @SparkQA