Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hive partitioned(v4) rebased #1864

Closed
wants to merge 21 commits into from
Closed

Hive partitioned(v4) rebased #1864

wants to merge 21 commits into from

Commits on Jan 4, 2018

  1. add hive style partitioning for contigName 

    Add partitioning by 1 megabase bin under each chromosome

moved pom back to scala 2_10 and spark 1

added hive partitioning for all types

add loading hive paritions support to all types

cleanup hive paritions code

individual ParttionedParquet save and load function, removed reference to Hive, fixed partitioned flag marker file to work with HDFS

updated tests, fixed boolean error in partition flag test

fixed typo

Address review comments

Addressed review comments, whitespace and misc.

fixed pom

allow paritioned alignment load by ReferenceRegion

move to spark2 and scala 2.11 as otherwise will not compile
    Paschall authored and Paschall committed Jan 4, 2018
    Configuration menu
    Copy the full SHA
    5789181 View commit details
    Browse the repository at this point in the history

  2. move back to scala2.11 spark2

    @jpdna
    jpdna authored and Paschall committed Jan 4, 2018
    Configuration menu
    Copy the full SHA
    55e602a View commit details
    Browse the repository at this point in the history

  3. fixed paritioned read region filter logic

    @jpdna
    jpdna authored and Paschall committed Jan 4, 2018
    Configuration menu
    Copy the full SHA
    71e59f2 View commit details
    Browse the repository at this point in the history

  4. factored out datast query string function

    Paschall authored and Paschall committed Jan 4, 2018
    Configuration menu
    Copy the full SHA
    549fc31 View commit details
    Browse the repository at this point in the history

  5. added scaladoc - wip

    @jpdna
    jpdna authored and Paschall committed Jan 4, 2018
    Configuration menu
    Copy the full SHA
    a32990f View commit details
    Browse the repository at this point in the history

  6. fixed the require check partiions, passes tests

    Paschall authored and Paschall committed Jan 4, 2018
    Configuration menu
    Copy the full SHA
    0c0a98e View commit details
    Browse the repository at this point in the history

  7. fixed the require check partiions, passes tests

    Paschall authored and Paschall committed Jan 4, 2018
    Configuration menu
    Copy the full SHA
    92bc5ba View commit details
    Browse the repository at this point in the history

  8. added support for 'chr' prefix

    Paschall authored and Paschall committed Jan 4, 2018
    Configuration menu
    Copy the full SHA
    42d2da5 View commit details
    Browse the repository at this point in the history

  9. made use_chr_prefix false by default, moved to spark2

    Paschall authored and Paschall committed Jan 4, 2018
    Configuration menu
    Copy the full SHA
    1579f3f View commit details
    Browse the repository at this point in the history

  10. 12/11/07 working

    @jpdna
    jpdna authored and Paschall committed Jan 4, 2018
    Configuration menu
    Copy the full SHA
    8d2163b View commit details
    Browse the repository at this point in the history

  11. move_to_spark 1

    Paschall authored and Paschall committed Jan 4, 2018
    Configuration menu
    Copy the full SHA
    1beff0a View commit details
    Browse the repository at this point in the history

  12. rebased against master and move spark2

    Paschall authored and Paschall committed Jan 4, 2018
    Configuration menu
    Copy the full SHA
    47c16a3 View commit details
    Browse the repository at this point in the history

  13. Added docs

    Paschall authored and Paschall committed Jan 4, 2018
    Configuration menu
    Copy the full SHA
    aa68b29 View commit details
    Browse the repository at this point in the history

  14. moving to spark 1

    Paschall authored and Paschall committed Jan 4, 2018
    Configuration menu
    Copy the full SHA
    062af3d View commit details
    Browse the repository at this point in the history

  15. moved to spark 2

    Paschall authored and Paschall committed Jan 4, 2018
    Configuration menu
    Copy the full SHA
    a45178c View commit details
    Browse the repository at this point in the history

Commits on Jan 17, 2018

  1. Merge branch 'master' into hive_partitioned_v4

    @jpdna
    jpdna committed Jan 17, 2018
    Configuration menu
    Copy the full SHA
    6e59b8d View commit details
    Browse the repository at this point in the history

  2. fixed arbitrary import formatting differnces to match master

    Paschall authored and Paschall committed Jan 17, 2018
    Configuration menu
    Copy the full SHA
    22a7347 View commit details
    Browse the repository at this point in the history

  3. fixed floor function be adding spark.sql.functions_ import back in Ge… 

    …nomicsRDD.scala
    Paschall authored and Paschall committed Jan 17, 2018
    Configuration menu
    Copy the full SHA
    4442736 View commit details
    Browse the repository at this point in the history

  4. improve docs

    Paschall authored and Paschall committed Jan 17, 2018
    Configuration menu
    Copy the full SHA
    1a75ed5 View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2018

  1. create symlink for artificial.fa to match master

    Paschall authored and Paschall committed Jan 19, 2018
    Configuration menu
    Copy the full SHA
    323e13a View commit details
    Browse the repository at this point in the history

  2. Added docs to checkParttionedParquetFlag

    Paschall authored and Paschall committed Jan 19, 2018
    Configuration menu
    Copy the full SHA
    1aefa72 View commit details
    Browse the repository at this point in the history