Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-13149][SQL]Add FileStreamSource #11034

Closed
wants to merge 16 commits into from
Closed

[SPARK-13149][SQL]Add FileStreamSource #11034

wants to merge 16 commits into from

Commits on Feb 2, 2016

  1. Add FileStreamSource and FileStreamSink

    @zsxwing
    zsxwing committed Feb 2, 2016
    Configuration menu
    Copy the full SHA
    a2784ff View commit details
    Browse the repository at this point in the history

Commits on Feb 3, 2016

  1. Remove FileStreamSink; Rerewrite tests using testStream; fix a bug ab… 

    …out startId; address other comments
    @zsxwing
    zsxwing committed Feb 3, 2016
    Configuration menu
    Copy the full SHA
    6a90c55 View commit details
    Browse the repository at this point in the history

Commits on Feb 4, 2016

  1. Merge branch 'master' into stream-df-file-source

    @zsxwing
    zsxwing committed Feb 4, 2016
    Configuration menu
    Copy the full SHA
    9f5967f View commit details
    Browse the repository at this point in the history

  2. Use the existing files to infer schema and update tests

    @zsxwing
    zsxwing committed Feb 4, 2016
    Configuration menu
    Copy the full SHA
    937b86f View commit details
    Browse the repository at this point in the history

  3. Add stress test

    @zsxwing
    zsxwing committed Feb 4, 2016
    Configuration menu
    Copy the full SHA
    2af6fc8 View commit details
    Browse the repository at this point in the history

  4. maxBatchFile -> maxBatchId

    @zsxwing
    zsxwing committed Feb 4, 2016
    Configuration menu
    Copy the full SHA
    f38153b View commit details
    Browse the repository at this point in the history

Commits on Feb 5, 2016

  1. Handle corrupted metadata files; cache the metadata in the memory

    @zsxwing
    zsxwing committed Feb 5, 2016
    Configuration menu
    Copy the full SHA
    1ccea0f View commit details
    Browse the repository at this point in the history

  2. Add private

    @zsxwing
    zsxwing committed Feb 5, 2016
    Configuration menu
    Copy the full SHA
    ca1e6b8 View commit details
    Browse the repository at this point in the history

  3. Handle a race condition in tests

    @zsxwing
    zsxwing committed Feb 5, 2016
    Configuration menu
    Copy the full SHA
    d911963 View commit details
    Browse the repository at this point in the history

  4. Only text format doesn't need a schema

    @zsxwing
    zsxwing committed Feb 5, 2016
    Configuration menu
    Copy the full SHA
    ce0556d View commit details
    Browse the repository at this point in the history

  5. Check the empty schema and 'text' format case in FileStreamSource

    @zsxwing
    zsxwing committed Feb 5, 2016
    Configuration menu
    Copy the full SHA
    9fd21c2 View commit details
    Browse the repository at this point in the history

  6. Add FileStreamSource.VERSION instead of using spark version

    @zsxwing
    zsxwing committed Feb 5, 2016
    Configuration menu
    Copy the full SHA
    9a1042c View commit details
    Browse the repository at this point in the history

  7. Validate the version line

    @zsxwing
    zsxwing committed Feb 5, 2016
    Configuration menu
    Copy the full SHA
    91a2b74 View commit details
    Browse the repository at this point in the history

  8. Merge remote-tracking branch 'origin/master' into stream-df-file-source

    @zsxwing
    zsxwing committed Feb 5, 2016
    Configuration menu
    Copy the full SHA
    1ffee5f View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2016

  1. Throw a better error if path is not specified

    @zsxwing
    zsxwing committed Feb 8, 2016
    Configuration menu
    Copy the full SHA
    fb0e3f9 View commit details
    Browse the repository at this point in the history

Commits on Feb 9, 2016

  1. Address TD's comments

    @zsxwing
    zsxwing committed Feb 9, 2016
    Configuration menu
    Copy the full SHA
    07e2ddd View commit details
    Browse the repository at this point in the history