Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/postgres database #523

Merged
merged 107 commits into from
Mar 22, 2019
Merged

Feature/postgres database #523

merged 107 commits into from
Mar 22, 2019

Commits on Mar 8, 2019

  1. Basics of data importer for database change. Can import dbSNP.

    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    041a377 View commit details
    Browse the repository at this point in the history

  2. Some changes to how arguments are passed.

    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    6975d9f View commit details
    Browse the repository at this point in the history

  3. Adds reference set importer to importer, as well as bug fixes to dbsn… 

    …p importer.
    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    bb36ec7 View commit details
    Browse the repository at this point in the history

  4. Adds importer for old database data tables.

    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    9866099 View commit details
    Browse the repository at this point in the history

  5. removes accidentally committed debugging code.

    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    a585601 View commit details
    Browse the repository at this point in the history

  6. Adds support to move user fields.

    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    150a636 View commit details
    Browse the repository at this point in the history

  7. Adds basic raw data importer.

    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    e327cbb View commit details
    Browse the repository at this point in the history

  8. Quick and dirty postgres

    @viklund @talavis
    viklund authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    e2faa76 View commit details
    Browse the repository at this point in the history

  9. Adds postgres schemas

    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    eff70cd View commit details
    Browse the repository at this point in the history

  10. fixes bug where gziped lines where not read in the same way as unzipp… 

    …ed lines.
    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    4d5bdcb View commit details
    Browse the repository at this point in the history

  11. adds dry-run support to dbsnp importer.

    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    2bed837 View commit details
    Browse the repository at this point in the history

  12. adds mysql-data and downloaded files to gitignore.

    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    1bc9d38 View commit details
    Browse the repository at this point in the history

  13. adds importer requirements.

    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    55b138c View commit details
    Browse the repository at this point in the history

  14. Adds dry run support to reference set import.

    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    c228a88 View commit details
    Browse the repository at this point in the history

  15. updates to old db importer to support auto-picking refset and dry-runs.

    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    7016c5f View commit details
    Browse the repository at this point in the history

  16. Corrects a help message.

    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    ecc2e8e View commit details
    Browse the repository at this point in the history

  17. adds --dry-run, --dataset, and --version flags to raw data importer.

    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    6903360 View commit details
    Browse the repository at this point in the history

  18. Adds number of variants to dataset_versions

    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    40f3029 View commit details
    Browse the repository at this point in the history

  19. Updates importer to match updated postgres schema.

    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    81795b6 View commit details
    Browse the repository at this point in the history

  20. Adds functions from the old db peewee wrapper to the new one.

    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    fce1a30 View commit details
    Browse the repository at this point in the history

  21. Docker-compose for postgres database.

    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    f97f247 View commit details
    Browse the repository at this point in the history

  22. Main site working from postgres database (not browser).

    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    ca939bc View commit details
    Browse the repository at this point in the history

  23. Fix the name of the end_pos column

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    f860f06 View commit details
    Browse the repository at this point in the history

  24. changed database connection type; playhouse.postgres_ext requires Pos… 

    …tgresqlExtDatabase
    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    e4102b9 View commit details
    Browse the repository at this point in the history

  25. Should hopefully give us actual json in the db

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    4b3ceda View commit details
    Browse the repository at this point in the history

  26. gencode files are gtf, ie 1-indexed

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    03b7396 View commit details
    Browse the repository at this point in the history

  27. Should fix gene stop positions

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    ea0123c View commit details
    Browse the repository at this point in the history

  28. Should fix the division of multiple names

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    a296220 View commit details
    Browse the repository at this point in the history

  29. should fix dbsnp problem

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    73de6e2 View commit details
    Browse the repository at this point in the history

  30. quality metrics should also remain as dict

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    40c6420 View commit details
    Browse the repository at this point in the history

  31. other_names for a gene is now in a separate table 

    Import changes done according to planned DB schema changes.
    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    6b0b692 View commit details
    Browse the repository at this point in the history

  32. Updates db handler to match array-to-table changes in schema.

    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    f025a72 View commit details
    Browse the repository at this point in the history

  33. fixed a confusingly named field (gene -> variant)

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    280c7c3 View commit details
    Browse the repository at this point in the history

  34. conversion str->int and a forgotten .execute() for the last batch

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    f33eb97 View commit details
    Browse the repository at this point in the history

  35. hom_count parsed

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    14e3d34 View commit details
    Browse the repository at this point in the history

  36. attempt to add beacon-only import. untested

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    6a62d32 View commit details
    Browse the repository at this point in the history

  37. Fix typos etc

    @MalinAhlberg @talavis
    MalinAhlberg authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    b83e1e9 View commit details
    Browse the repository at this point in the history

  38. make some more stuff optional

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    8301059 View commit details
    Browse the repository at this point in the history

  39. more generic vcf parsing for the beacon

    @MalinAhlberg @talavis
    MalinAhlberg authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    fc49cd6 View commit details
    Browse the repository at this point in the history

  40. Skip lines with non-standard rsid 

    This should probably be handled more carefully
    @MalinAhlberg @talavis
    MalinAhlberg authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    9a37e0a View commit details
    Browse the repository at this point in the history

  41. fix: Restore mysql-settings.

    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    05ac2ea View commit details
    Browse the repository at this point in the history

  42. variant parser updated for new db schema

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    5139922 View commit details
    Browse the repository at this point in the history

  43. and a forgotten get()

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    5a35aa7 View commit details
    Browse the repository at this point in the history

  44. a five-character string probably never matches a four-character one. … 

    …oops.
    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    30b56ae View commit details
    Browse the repository at this point in the history

  45. Wrong db

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    ce06d49 View commit details
    Browse the repository at this point in the history

  46. a couple of pylint fixes

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    b376e3d View commit details
    Browse the repository at this point in the history

  47. slow id lookup

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    7376aa9 View commit details
    Browse the repository at this point in the history

  48. refdata changed to local objects instead of class members

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    25b26ec View commit details
    Browse the repository at this point in the history

  49. forgot a self

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    f3381c5 View commit details
    Browse the repository at this point in the history

  50. update parameters for function call

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    556e26c View commit details
    Browse the repository at this point in the history

  51. feat: Change mysql database to postgres database 

    This removes the MySQL schema and replaces it with postgres schemas, as
well as updating the test dummy data.
    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    19775be View commit details
    Browse the repository at this point in the history

  52. feat: Rewrite travis tests to use postgres 

    Replaces mysql tests with postgres tests. No new tests are added, the
old tests are just rewriten for postgres.
    @talavis
    Martin Norling authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    035406b View commit details
    Browse the repository at this point in the history

  53. Add sampleCount in import script

    @MalinAhlberg @talavis
    MalinAhlberg authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    d0d6961 View commit details
    Browse the repository at this point in the history

  54. added sample_count column to model

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    9332827 View commit details
    Browse the repository at this point in the history

  55. Remove sample_count model again 

    Undoes 6ee3c09
    @MalinAhlberg @talavis
    MalinAhlberg authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    7c04ddb View commit details
    Browse the repository at this point in the history

  56. Only count samples in header, not on each data row

    @MalinAhlberg @talavis
    MalinAhlberg authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    e0ef58c View commit details
    Browse the repository at this point in the history

  57. Save sampleCount in sample_sets, add parameter to import script 

    Information from `NS` or header in vcf gets saved to data.sample_sets.sample_size
(No changes to db schemas needed anymore!)
    @MalinAhlberg @talavis
    MalinAhlberg authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    64ede98 View commit details
    Browse the repository at this point in the history

  58. Fix typo

    @MalinAhlberg @talavis
    MalinAhlberg authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    ecb8eee View commit details
    Browse the repository at this point in the history

  59. Add parameters for assembly_id and beacon_description 

    assembly_id sets data.dataset_versions.var_call_ref
  (remember to update beacon schema)
beacon_description sets data.datasets.beacon_description
sampleset_size sets a cusom sampleset size
set_vcf_sampleset_size sets the sampleset size according to the vcf
    @MalinAhlberg @talavis
    MalinAhlberg authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    70afe46 View commit details
    Browse the repository at this point in the history

  60. Fix small mistakes

    @MalinAhlberg @talavis
    MalinAhlberg authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    9c4dd27 View commit details
    Browse the repository at this point in the history

  61. Improve help messages

    @MalinAhlberg @talavis
    MalinAhlberg authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    1044722 View commit details
    Browse the repository at this point in the history

  62. Add parameter for datasetsize

    @MalinAhlberg @talavis
    MalinAhlberg authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    ef81b39 View commit details
    Browse the repository at this point in the history

  63. Fix parsing of multiple rsids

    @MalinAhlberg @talavis
    MalinAhlberg authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    9742f12 View commit details
    Browse the repository at this point in the history

  64. use beacon fix for A[NC] for all datasets without A[NC]_adj. Also che… 

    …ck that gene identifiers contain ENSG
    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    02b8536 View commit details
    Browse the repository at this point in the history

  65. moved ENS[GT] checks to parser function, also added set to remove dup… 

    …licates
    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    34f51d9 View commit details
    Browse the repository at this point in the history

  66. add tracking of return value from the actual tests

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    eb04242 View commit details
    Browse the repository at this point in the history

  67. apparently {} are not needed on arithmetic variables

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    38dd02a View commit details
    Browse the repository at this point in the history

  68. forgot to remove the other {}

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    d082b72 View commit details
    Browse the repository at this point in the history

  69. added space to make the code easier to read

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    a351e94 View commit details
    Browse the repository at this point in the history

  70. Log info about dryrun

    @MalinAhlberg @talavis
    MalinAhlberg authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    28fd9f3 View commit details
    Browse the repository at this point in the history

  71. new indexes added; sorted alphabetically

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    5808409 View commit details
    Browse the repository at this point in the history

  72. forgotten ;

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    4cf890c View commit details
    Browse the repository at this point in the history

  73. g is before t

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    668cb90 View commit details
    Browse the repository at this point in the history

  74. indexes for genes and transcripts associated with variants

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    12d21eb View commit details
    Browse the repository at this point in the history

  75. and indexes for variant->genes/transcripts

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    f7ef8e6 View commit details
    Browse the repository at this point in the history

  76. Add indices for beacon

    @MalinAhlberg @talavis
    MalinAhlberg authored and talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    0671e15 View commit details
    Browse the repository at this point in the history

  77. make sure the correct reference set is used for genes/transcripts

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    9a22dd1 View commit details
    Browse the repository at this point in the history

  78. refgenes/transcripts -> ref_*

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    7d1dbc7 View commit details
    Browse the repository at this point in the history

  79. added missing function

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    e85c2dd View commit details
    Browse the repository at this point in the history

  80. dbid renamed to refset, function name updated

    @talavis
    talavis committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    f91f49e View commit details
    Browse the repository at this point in the history

  81. feat: Add option to specify settings file

    Martin Norling committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    2fa3408 View commit details
    Browse the repository at this point in the history

Commits on Mar 14, 2019

  1. Adding beacon database schema

    @viklund
    viklund committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    d1393e6 View commit details
    Browse the repository at this point in the history

  2. Add missing quotes and make existing quotes consistent

    @kusalananda
    kusalananda committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    05a60e2 View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2019

  1. Remove $ on variable in arithmetic context

    @kusalananda
    kusalananda committed Mar 15, 2019
    Configuration menu
    Copy the full SHA
    64439a6 View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2019

  1. feat: Remove dbSNP and OMIM data 

    Removes dbSNP and OMIM data, as we felt that they did not contribute
enough to the system.
    Martin Norling committed Mar 18, 2019
    Configuration menu
    Copy the full SHA
    f8e7be8 View commit details
    Browse the repository at this point in the history

  2. feat: Adapt beacon schemat to schema changes 

    Adds indexes to beacon schema, changes views to work with changes in
the data schema, and adds beacon schema to docker-compose.
    Martin Norling committed Mar 18, 2019
    Configuration menu
    Copy the full SHA
    e705ead View commit details
    Browse the repository at this point in the history

  3. better hom_count?

    @talavis
    talavis committed Mar 18, 2019
    Configuration menu
    Copy the full SHA
    8346951 View commit details
    Browse the repository at this point in the history

  4. use minimal representation for variants 

    Also avoid recalculation of identical data structure
    @talavis
    talavis committed Mar 18, 2019
    Configuration menu
    Copy the full SHA
    6efde85 View commit details
    Browse the repository at this point in the history

  5. forgot to take care of unused return value

    @talavis
    talavis committed Mar 18, 2019
    Configuration menu
    Copy the full SHA
    bca9f8d View commit details
    Browse the repository at this point in the history

  6. the return value _is_ needed

    @talavis
    talavis committed Mar 18, 2019
    Configuration menu
    Copy the full SHA
    b5a2800 View commit details
    Browse the repository at this point in the history

  7. Attempt to get code to work: don't reference data before it is created

    @MalinAhlberg
    MalinAhlberg committed Mar 18, 2019
    Configuration menu
    Copy the full SHA
    ebdf393 View commit details
    Browse the repository at this point in the history

  8. fix hotfix

    @talavis
    talavis committed Mar 18, 2019
    Configuration menu
    Copy the full SHA
    fc8aee2 View commit details
    Browse the repository at this point in the history

  9. Merge pull request #527 from NBISweden/hot_fix/bug_fix 

    Attempt to get code to work: don't reference data before it is created
    Linus Östberg authored Mar 18, 2019
    Configuration menu
    Copy the full SHA
    34ca445 View commit details
    Browse the repository at this point in the history

  10. fix for int->tuple

    @talavis
    talavis committed Mar 18, 2019
    Configuration menu
    Copy the full SHA
    f56be4a View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2019

  1. test: Fix travis tests for postgres update

    Martin Norling committed Mar 19, 2019
    Configuration menu
    Copy the full SHA
    46eb946 View commit details
    Browse the repository at this point in the history

  2. data->base; remove unintended debug line

    @talavis
    talavis committed Mar 19, 2019
    Configuration menu
    Copy the full SHA
    406d59e View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2019

  1. perform coverage reformatting immediately

    @talavis
    talavis committed Mar 20, 2019
    Configuration menu
    Copy the full SHA
    e35c557 View commit details
    Browse the repository at this point in the history

  2. a few fixes for batch management 

    - check batch size using >=
- check if batch: instead of if len(batch):
- no need to reset batch at end
- a few formatting changes
    @talavis
    talavis committed Mar 20, 2019
    Configuration menu
    Copy the full SHA
    e618972 View commit details
    Browse the repository at this point in the history

  3. clarify that the values are in data.*

    @talavis
    talavis committed Mar 20, 2019
    Configuration menu
    Copy the full SHA
    7488838 View commit details
    Browse the repository at this point in the history

  4. first implementation of replacement function for progress

    @talavis
    talavis committed Mar 20, 2019
    Configuration menu
    Copy the full SHA
    63c019f View commit details
    Browse the repository at this point in the history

  5. negative last_progress in raw import as well

    @talavis
    talavis committed Mar 20, 2019
    Configuration menu
    Copy the full SHA
    3e53e49 View commit details
    Browse the repository at this point in the history

  6. added missing space

    @talavis
    talavis committed Mar 20, 2019
    Configuration menu
    Copy the full SHA
    ed66320 View commit details
    Browse the repository at this point in the history

  7. another fix for coverage

    @talavis
    talavis committed Mar 20, 2019
    Configuration menu
    Copy the full SHA
    f803efa View commit details
    Browse the repository at this point in the history

  8. possible to only import variants or coverage

    @talavis
    talavis committed Mar 20, 2019
    Configuration menu
    Copy the full SHA
    c673791 View commit details
    Browse the repository at this point in the history

  9. removed need of the function

    @talavis
    talavis committed Mar 20, 2019
    Configuration menu
    Copy the full SHA
    a6e75ed View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2019

  1. last coverage insert did not use execute()

    @talavis
    talavis committed Mar 21, 2019
    Configuration menu
    Copy the full SHA
    3cd211b View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2019

  1. pylint fixes

    @talavis
    talavis committed Mar 22, 2019
    Configuration menu
    Copy the full SHA
    dc7f872 View commit details
    Browse the repository at this point in the history