Skip to content

[WIP]Support minPartitions for batch and Kafka source v1 #25436

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
zsxwing wants to merge 4 commits into from
Closed

[WIP]Support minPartitions for batch and Kafka source v1 #25436

zsxwing wants to merge 4 commits into from

Conversation

@zsxwing
Copy link
Member

@zsxwing zsxwing commented Aug 13, 2019 •
edited
Loading

What changes were proposed in this pull request?

Support minPartitions for batch and Kafka source v1. Still WIP

How was this patch tested?

Jenkins

@zsxwing zsxwing changed the title [SC-21182]Remove Structured Streaming Kafka 0.8 connector (6.x) [WIP]Support minPartitions for batch and Kafka source v1 Aug 13, 2019
@SparkQA
Copy link

SparkQA commented Aug 13, 2019

Test build #109052 has finished for PR 25436 at commit e35128c.

  • This patch fails Spark unit tests.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 16, 2019

Test build #109163 has finished for PR 25436 at commit d340d7d.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • abstract class FileCommitProtocol extends Logging
  • class BindingParquetOutputCommitter(
  • class PathOutputCommitProtocol(
  • class DecisionTreeParams(Params):
  • case class UnresolvedTable(v1Table: CatalogTable) extends Table
  • implicit class IdentifierHelper(identifier: TableIdentifier)
  • case class Milliseconds(child: Expression, timeZoneId: Option[String] = None)
  • case class Microseconds(child: Expression, timeZoneId: Option[String] = None)
  • case class IsoYear(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
  • case class Epoch(child: Expression, timeZoneId: Option[String] = None)
  • case class DeleteFromTable(
  • case class DeleteFromStatement(
  • case class DeleteFromTableExec(

@gaborgsomogyi
Copy link
Contributor

gaborgsomogyi commented Sep 24, 2019

@zsxwing are you working on this or shall I offer a helping hand?

@HeartSaVioR HeartSaVioR mentioned this pull request Dec 11, 2019
Closed
dongjoon-hyun pushed a commit that referenced this pull request Dec 11, 2019
@HeartSaVioR @dongjoon-hyun
[MINOR][SS][DOC] Fix the ss-kafka doc for availability of 'minPartiti…
e39bb4c 
…ons' option

### What changes were proposed in this pull request?

This patch fixes the availability of `minPartitions` option for Kafka source, as it is only supported by micro-batch for now. There's a WIP PR for batch (#25436) as well but there's no progress on the PR so far, so safer to fix the doc first, and let it be added later when we address it with batch case as well.

### Why are the changes needed?

The doc is wrong and misleading.

### Does this PR introduce any user-facing change?

No.

### How was this patch tested?

Just a doc change.

Closes #26849 from HeartSaVioR/MINOR-FIX-minPartition-availability-doc.

Authored-by: Jungtaek Lim (HeartSaVioR) <kabhwan.opensource@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
dongjoon-hyun pushed a commit that referenced this pull request Dec 12, 2019
@HeartSaVioR @dongjoon-hyun
[MINOR][SS][DOC] Fix the ss-kafka doc for availability of 'minPartiti…
f4e95ca 
…ons' option

### What changes were proposed in this pull request?

This patch fixes the availability of `minPartitions` option for Kafka source, as it is only supported by micro-batch for now. There's a WIP PR for batch (#25436) as well but there's no progress on the PR so far, so safer to fix the doc first, and let it be added later when we address it with batch case as well.

### Why are the changes needed?

The doc is wrong and misleading.

### Does this PR introduce any user-facing change?

No.

### How was this patch tested?

Just a doc change.

Closes #26849 from HeartSaVioR/MINOR-FIX-minPartition-availability-doc.

Authored-by: Jungtaek Lim (HeartSaVioR) <kabhwan.opensource@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
(cherry picked from commit e39bb4c)
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
@github-actions
Copy link

github-actions bot commented Jan 3, 2020

We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just
a way of keeping the PR queue manageable.

If you'd like to revive this PR, please reopen it!

@github-actions github-actions bot added the Stale label Jan 3, 2020
@github-actions github-actions bot closed this Jan 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

Stale STRUCTURED STREAMING

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@zsxwing @SparkQA @gaborgsomogyi @dongjoon-hyun