Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC79: Incremental Upload of Data Entries #48

Open
wants to merge 128 commits into
base: main
Choose a base branch
from
Open

RFC79: Incremental Upload of Data Entries #48

wants to merge 128 commits into from

Commits on Apr 14, 2024

  1. Add clinical_attribute_meta records to the seed mini

    To make the dataset look like real data in the database
    forus committed Apr 14, 2024
    Configuration menu
    Copy the full SHA
    5dfe298 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    531b10a View commit details
    Browse the repository at this point in the history
  3. Add --overwrite-existing for the rest of test cases

    Apperently, the flag does not change anything.
    But we add it anyway as the tests for "incremental" data upload.
    forus committed Apr 14, 2024
    Configuration menu
    Copy the full SHA
    248a08c View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    2bc7271 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    31e3194 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    bd023a9 View commit details
    Browse the repository at this point in the history
  7. Add loader command to update case list with sample ids

    adding to the all case list and case list specified with command arguments is supported
    forus committed Apr 14, 2024
    Configuration menu
    Copy the full SHA
    c49bbf3 View commit details
    Browse the repository at this point in the history
  8. Add option to remove sample ids from the remaining case lists

    From case lists that is not _all case list and not specified with --add-to-case-lists option
    forus committed Apr 14, 2024
    Configuration menu
    Copy the full SHA
    1f5695d View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    77cd6a8 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    bd8c4b2 View commit details
    Browse the repository at this point in the history
  11. Fix test clinical data headers

    forus committed Apr 14, 2024
    Configuration menu
    Copy the full SHA
    5fc633b View commit details
    Browse the repository at this point in the history
  12. Test incremental patient upload

    forus committed Apr 14, 2024
    Configuration menu
    Copy the full SHA
    f7132c9 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    f45e1e8 View commit details
    Browse the repository at this point in the history
  14. Add TODO comment to remove MIXED_ATTRIBUTES data type

    with a reference to the ticket
    forus committed Apr 14, 2024
    Configuration menu
    Copy the full SHA
    8cc95a0 View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    fa32b7f View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    f044c3b View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    48fca03 View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    1302a8e View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    659f352 View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    b5952e3 View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    753119b View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    5725d42 View commit details
    Browse the repository at this point in the history
  23. Configuration menu
    Copy the full SHA
    299466a View commit details
    Browse the repository at this point in the history
  24. Change study_es_0_inc to contain more diverse data

    We changed them to work for the demo.
    Mutation numbers did not change on demo.
    forus committed Apr 14, 2024
    Configuration menu
    Copy the full SHA
    c0c28e2 View commit details
    Browse the repository at this point in the history
  25. Configuration menu
    Copy the full SHA
    c6eddbb View commit details
    Browse the repository at this point in the history
  26. Disambiguate clinical data constants names

    Not it was easy to be confused where sample and clinical_sample (attributes),
    patient and clinical_patient (attributes) related code
    forus committed Apr 14, 2024
    Configuration menu
    Copy the full SHA
    595d24f View commit details
    Browse the repository at this point in the history
  27. Configuration menu
    Copy the full SHA
    c8b4c73 View commit details
    Browse the repository at this point in the history
  28. Configuration menu
    Copy the full SHA
    efd34d8 View commit details
    Browse the repository at this point in the history
  29. Configuration menu
    Copy the full SHA
    3b39e0d View commit details
    Browse the repository at this point in the history
  30. Make --overwrite-existing flag description more generic

    This flag for command to upload molecular profile data
    forus committed Apr 14, 2024
    Configuration menu
    Copy the full SHA
    fc785f6 View commit details
    Browse the repository at this point in the history
  31. Configuration menu
    Copy the full SHA
    e782951 View commit details
    Browse the repository at this point in the history
  32. Update case lists for multiple clinical sample files

    Potentially for different studies
    forus committed Apr 14, 2024
    Configuration menu
    Copy the full SHA
    b53c8c4 View commit details
    Browse the repository at this point in the history
  33. Configuration menu
    Copy the full SHA
    99550b5 View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2024

  1. Fix TestIntegrationTest

    - change location of the files
    - make sure assertions could work on the seed mini db
    - get rid from absent cbioportal dependencies
    forus committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    1829842 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e785a53 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    e09e1e2 View commit details
    Browse the repository at this point in the history

Commits on May 1, 2024

  1. Configuration menu
    Copy the full SHA
    7b527b6 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    f5e8217 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8d3aaed View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    1b6ba41 View commit details
    Browse the repository at this point in the history

Commits on May 3, 2024

  1. Configuration menu
    Copy the full SHA
    d252001 View commit details
    Browse the repository at this point in the history
  2. Extract smaller functions from the big one in py script

    Make process_data_directory(...) smaller
    forus committed May 3, 2024
    Configuration menu
    Copy the full SHA
    c27b8f1 View commit details
    Browse the repository at this point in the history

Commits on May 15, 2024

  1. Merge pull request #32 from se4bio/inc-data-upload-poc

    Incremental data upload for mutations, case lists, patient and sample attributes
    forus committed May 15, 2024
    Configuration menu
    Copy the full SHA
    2e80b73 View commit details
    Browse the repository at this point in the history
  2. Refactor tab delim. data importer

    - Calculate number of lines in the file in the loader
    - Remove unused imports and fields
    - Reuse constructors
    - Reuse common parsing logic in tab delimiter importer
    - Show full stacktrace which helps in dinding where tests errored out
    forus committed May 15, 2024
    Configuration menu
    Copy the full SHA
    b2c1c21 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    a7aab3a View commit details
    Browse the repository at this point in the history
  4. Add RPPA test

    forus committed May 15, 2024
    Configuration menu
    Copy the full SHA
    bd2d8c1 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    8b68331 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    b18aab1 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    ea688c3 View commit details
    Browse the repository at this point in the history
  8. Test for invalid entrez id

    forus committed May 15, 2024
    Configuration menu
    Copy the full SHA
    cdae501 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    cf458a4 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    9ea1ada View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    03f9660 View commit details
    Browse the repository at this point in the history
  12. Remove unused code

    forus committed May 15, 2024
    Configuration menu
    Copy the full SHA
    93cc6ff View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    842bcd3 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    22b688a View commit details
    Browse the repository at this point in the history
  15. Fix integration tests

    forus committed May 15, 2024
    Configuration menu
    Copy the full SHA
    d11a353 View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    7dfb1bd View commit details
    Browse the repository at this point in the history
  17. Check for illegal state in tab delim. data update

    It's dangerous as we would further mess up the data in the row
    forus committed May 15, 2024
    Configuration menu
    Copy the full SHA
    71cdf70 View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    2d31dac View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    4997542 View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    911ae28 View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    c7343f9 View commit details
    Browse the repository at this point in the history
  22. Remove obsolete TODO comment

    forus committed May 15, 2024
    Configuration menu
    Copy the full SHA
    2ed0bd8 View commit details
    Browse the repository at this point in the history

Commits on May 16, 2024

  1. Reuse genetic_profile record if it exists in db already

    Do it for all data types, not only MAF
    forus committed May 16, 2024
    Configuration menu
    Copy the full SHA
    76b52a9 View commit details
    Browse the repository at this point in the history
  2. Test incremental upload of tab delim. data types from umbrella script

    - Split big tab. delim test to multiple tests based on data type.
    
    - Use ImportProfileData instead of ImportTabDelimData for testing.
      - We cover more logic with such tests.
      - This is more stable interface. ImportTabDelimData can be refactored.
    forus committed May 16, 2024
    Configuration menu
    Copy the full SHA
    fa16076 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    e5ccc3e View commit details
    Browse the repository at this point in the history

Commits on May 17, 2024

  1. Configuration menu
    Copy the full SHA
    472f47e View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    c54e303 View commit details
    Browse the repository at this point in the history

Commits on May 28, 2024

  1. Fix cbioportalImport script execution

    args variable was not declared
    forus committed May 28, 2024
    Configuration menu
    Copy the full SHA
    18dbdd3 View commit details
    Browse the repository at this point in the history
  2. Remove not needed spring context initialisation

    that caused different errors to occur
    forus committed May 28, 2024
    Configuration menu
    Copy the full SHA
    c702a8b View commit details
    Browse the repository at this point in the history
  3. Make error message more informative when gene panel is not found

    Do not throw NPE, but NSEE with error message that mentions panel id
    forus committed May 28, 2024
    Configuration menu
    Copy the full SHA
    0ff7031 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    54cc04e View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    a022aab View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    90cc928 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    fb75d7c View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    3331223 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    d7e1918 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    ee183e6 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    697631f View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    65c8b11 View commit details
    Browse the repository at this point in the history
  13. Remove unused code

    forus committed May 28, 2024
    Configuration menu
    Copy the full SHA
    0bf6bf2 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    cc80e56 View commit details
    Browse the repository at this point in the history
  15. Implement incremental upload for gene panel matrix

    The uploader was working in incremental manner already.
    I had to add tests for those only.
    I had to implement incremental upload for gene panel matrix
    from differend data (CNA, Mutations) uploaders though.
    forus committed May 28, 2024
    Configuration menu
    Copy the full SHA
    4070e68 View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    e8bbb34 View commit details
    Browse the repository at this point in the history
  17. Implement incremental upload of structural variants data

    I removed DaoGeneticProfileSamples.addGeneticProfileSamples(geneticProfileId, orderedSampleList);
    as it does not seem to be needed.
    it does not make any sense to store samples in genetic_profile_samples, if you don't use genetic_alteration table at all.
    forus committed May 28, 2024
    Configuration menu
    Copy the full SHA
    feed06c View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    bea4987 View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2024

  1. Configuration menu
    Copy the full SHA
    0cdda9d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d7e8ff3 View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2024

  1. Configuration menu
    Copy the full SHA
    ec849e2 View commit details
    Browse the repository at this point in the history
  2. Rename validation method to better describe it's purpose

    To really validate entrez id, we need to look it up
    forus committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    deb65cb View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8692ead View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    be9082c View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    4e8a7c2 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    b93e741 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    9089e77 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    16f6295 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    79c4041 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    111f58e View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    c4d5ecc View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    13eb147 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    95c32f8 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    e3ec5d6 View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    fc84a41 View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2024

  1. add comma to README

    pieterlukasse committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    4eac259 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d0428f8 View commit details
    Browse the repository at this point in the history
  3. Add join by GENETIC_PROFILE_ID column for sample_cna_event and altera…

    …tion_driver_annotaiton tables
    forus committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    bf2d539 View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2024

  1. Configuration menu
    Copy the full SHA
    37dcc20 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6562716 View commit details
    Browse the repository at this point in the history
  3. Remove TODOs that were done

    forus committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    b0a448e View commit details
    Browse the repository at this point in the history
  4. Rename isInfoLine util. method to isDataLine

    I got feedback that "info line" sounds like the header metadata lines starting with #
    forus committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    f3d76c7 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    f544847 View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2024

  1. Optimize removing genetic alterations

    by removing them for the whole genetic profile at once.
    
    one sql statment instead of N
    forus committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    ab51a4b View commit details
    Browse the repository at this point in the history
  2. Access inherited variables with this. intead of super.

    the confusion that triggered the change: The use of super. indicates that the subclass also declares one with the same name, but you are trying to not set that somehow?
    forus committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    96acec5 View commit details
    Browse the repository at this point in the history
  3. Merge pull request #45 from cBioPortal/inc-tab-delimited-uploader

    (1/7) RFC 79: Implement incremental upload of tab delimited data.
    forus committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    52714d6 View commit details
    Browse the repository at this point in the history
  4. Merge pull request #44 from cBioPortal/make_study_es_0_to_load

    (2/7) RFC79: Make study_es_0_inc data pass validation
    forus committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    074372f View commit details
    Browse the repository at this point in the history
  5. Merge pull request #43 from cBioPortal/inc-timeline-uploader

    (3/7) RFC79: Implement incremental upload for timeline data
    forus committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    a5ac232 View commit details
    Browse the repository at this point in the history
  6. Merge pull request #42 from cBioPortal/inc-cna-discrete-long

    (4/7) RFC79: Implement incremental upload of CNA DISCRETE long data
    forus committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    a47eb49 View commit details
    Browse the repository at this point in the history
  7. Merge pull request #41 from cBioPortal/inc-gene-panel-matrix

    (5/7) RFC79: Implement incremental upload for gene panel matrix
    forus committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    d28d04d View commit details
    Browse the repository at this point in the history
  8. Merge pull request #40 from cBioPortal/inc-sv

    (6/7) RFC79: Implement incremental upload of structural variants data
    forus committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    8c74dbb View commit details
    Browse the repository at this point in the history
  9. Merge pull request #39 from cBioPortal/inc-seg

    (7/7) RFC79: Implement incremental upload of CNA segmented data
    forus committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    d15c579 View commit details
    Browse the repository at this point in the history
  10. Merge pull request #47 from cBioPortal/rfc79-feedback

    RFC79: Implement feedback from previous 7 PRs
    forus committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    d081f8f View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2024

  1. Configuration menu
    Copy the full SHA
    e795449 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    df8f7af View commit details
    Browse the repository at this point in the history
  3. Rename upsertSampleProfiles to upsertSampleToProfileMapping

    method in DaoSampleProfile
    forus committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    f120f5d View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    602cc24 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    28dfa05 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2024

  1. Configuration menu
    Copy the full SHA
    2dd1e62 View commit details
    Browse the repository at this point in the history