(1/7) RFC 79: Implement incremental upload of tab delimited data. #45
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Calculate number of lines in the file in the loader - Remove unused imports and fields - Reuse constructors - Reuse common parsing logic in tab delimiter importer - Show full stacktrace which helps in dinding where tests errored out
It's dangerous as we would further mess up the data in the row
Do it for all data types, not only MAF
- Split big tab. delim test to multiple tests based on data type. - Use ImportProfileData instead of ImportTabDelimData for testing. - We cover more logic with such tests. - This is more stable interface. ImportTabDelimData can be refactored.
…ation rows or not
args variable was not declared
There was a problem hiding this comment.
thanks @forus , amazing work 👍
I do have some questions on sections that I would like to double-check with you, and some naming improvements. Please take a look.
pieterlukasse
approved these changes
Jun 19, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This loader covers the following data types:
Note: We do not support incremental upload of GSVA data although tab-delimited uploader covers this type with non-interactive upload. The incremental upload for GSVA data will throw an error.
We decided to exclude GSVA as there is unclarity from our part on whether GSVA scores of existing samples need to be recalculated when a new sample is added to the data.
This PR makes the tab. delimited data loader run in a single transaction to escape corruption of tsv rows (e.g. sample has been added to the header, but not to the info rows).
The first commit (b2c1c21) contains the code improvements. It could be handy to review it first.
=======
Please review #44 next.
Please note, to avoid the costly process of rebasing all pull requests (PRs), all feedback will be incorporated into the final PR.