Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove files from variants storage #192

Closed
11 tasks done
j-coll opened this issue Aug 17, 2015 · 2 comments
Closed
11 tasks done

Remove files from variants storage #192

j-coll opened this issue Aug 17, 2015 · 2 comments
Assignees
@j-coll
Labels
feature storage
Milestone
v1.2.0

Comments

@j-coll
Copy link
Member

j-coll commented Aug 17, 2015 •
edited

Currently it's not possible to delete specific samples data. Issue extracted from issue #160

Deleting samples data can be implemented by removing specific samples, or removing indexed files.

  • Delete files
    A file contains a set of samples. The storage metadata contains a list of all the indexed samples. Also, delete BatchFileOperations are available.
  • Delete samples
    This feature will require more accurate metadata information, to be able to know exactly which samples are indexed. Now the only way to determine if a sample is indexed or not, is if its file is indexed.
    Deleting samples is out of the scope

Task list:

  • Remove delete methods from VariantDBAdaptor interface
  • Rename delete and drop methods with remove
  • Delete file for MongoDB
    • Collect all existing genotypes
    • Synchronize delete with the stage collection
    • Detect and remove unused secondary alternates
    • Add BatchFileOperation blocking the study to avoid concurrent operations.
    • Migration script to collect loaded genotypes
  • Delete file for Hadoop
  • Invalidate cohorts stats with removed samples
  • Remove deleted samples from cohort ALL.
    Be aware that samples can be in multiple files.
  • Add Remove operations at VariantStorageManager
@j-coll j-coll added the feature label Aug 17, 2015
@j-coll j-coll added this to the Desired milestone Aug 17, 2015
@imedina imedina modified the milestones: v1.0.0-rc2, Desired Sep 15, 2016
@j-coll j-coll modified the milestones: v1.0.0, v1.0.0-rc3 Jan 9, 2017
@j-coll j-coll mentioned this issue Jan 9, 2017
Closed
@j-coll
Copy link
Member Author

j-coll commented Jan 9, 2017

see first approach drop_file.js

@j-coll j-coll modified the milestones: v1.2.0, v1.0.0 Jun 2, 2017
@j-coll j-coll added the storage label Jun 14, 2017
@j-coll j-coll mentioned this issue Jun 14, 2017
Closed
2 tasks
@j-coll j-coll changed the title Delete variant sample data from OpenCGA-Storage Remove files from variants storage Jun 14, 2017
@j-coll j-coll mentioned this issue Jun 15, 2017
Closed
4 tasks
j-coll added a commit that referenced this issue Jun 16, 2017
@j-coll
storage: Remove generic delete methods from VariantDBAdaptor #192
b8fbfc1
j-coll added a commit that referenced this issue Jun 16, 2017
@j-coll
storage: Register all loaded genotypes in variants collection. #192
810672c
j-coll added a commit that referenced this issue Jun 28, 2017
@j-coll
storage: Basic implementation of removeFile #192
1b6af4a
j-coll added a commit that referenced this issue Jun 30, 2017
@j-coll
storage: Clean ?/? array from all documents when deleting. #192
d35a21e
j-coll added a commit that referenced this issue Jul 7, 2017
@j-coll
storage: Rename all delete and drop operations with "remove". #192
f9043f2
j-coll added a commit that referenced this issue Jul 7, 2017
@j-coll
storage: Add BatchFileOperation for remove file operations #192
98d7a87
j-coll added a commit that referenced this issue Jul 12, 2017
@j-coll
storage: Implement VariantRemoveStorageOperation for VariantStorageMa…
67dee21 
…nager. #192
j-coll added a commit that referenced this issue Jul 27, 2017
@j-coll
storage: Remove deleted samples from cohort ALL. #192
b31763f 
Be aware that samples can be in multiple files.
j-coll added a commit that referenced this issue Jul 27, 2017
@j-coll
storage: Implement method removeStudy #192
88d299e
j-coll added a commit that referenced this issue Jul 28, 2017
@j-coll
catalog: Enable clear all samples from cohort in catalog. #192
93bf6a1
j-coll added a commit that referenced this issue Jul 28, 2017
@j-coll
storage: Delete sample columns from HBase. #192
7976bb1
j-coll added a commit that referenced this issue Jul 28, 2017
@j-coll
storage: Add shutdown hook when removing variants from MongoDB #192
1d6fd9f
@j-coll j-coll mentioned this issue Jul 28, 2017
Open
j-coll added a commit that referenced this issue Jul 31, 2017
@j-coll
storage: Remove VariantSource after remove files from variant storage. …
9c96284 
…#192
j-coll added a commit that referenced this issue Jul 31, 2017
@j-coll
storage: Ignore READY stage batchOperations if file is removed. #192
0ab2d06
j-coll added a commit that referenced this issue Jul 31, 2017
@j-coll
storage: Remove all <sid>.<fid> entries from stage collection. #192
f6a1da3
@j-coll
Copy link
Member Author

j-coll commented Aug 3, 2017

Migrate script for MongoDB at https://gist.github.com/j-coll/3dec01abc70644943d33de78105c633e

@j-coll j-coll closed this as completed Aug 3, 2017
@j-coll j-coll mentioned this issue Nov 29, 2017
Closed
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@j-coll j-coll

Labels
feature storage
Projects
None yet
Milestone
v1.2.0
Development

No branches or pull requests
2 participants
@imedina @j-coll