Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Import sda #130

Closed
wants to merge 166 commits into from
Closed

Import sda #130

wants to merge 166 commits into from

Conversation

ryanlong89
Copy link
Contributor

Description

Created an import script both for CPA legacy data and existing SCA archived data.

Changes Made

  • Script will recursively download files from the SDA
  • Script will attempt to stay within size limit on landing area
  • Script will extract tar file into landing space following the naming convention used on SDA

deepakduggirala and others added 30 commits June 23, 2023 08:30
@deepakduggirala
Change UI, API for CPA app
780a0b8
@deepakduggirala
Change workers for CPA app
f9f0742
@deepakduggirala
Merge remote-tracking branch 'bioloop/dev' into dev
7e3662c
@deepakduggirala
Merge remote-tracking branch 'bioloop/dev' into dev
0feb31d
@deepakduggirala
Merge remote-tracking branch 'bioloop/dev' into dev
e368c82
@deepakduggirala
Merge remote-tracking branch 'bioloop/dev' into dev
d50a0e6
@deepakduggirala
Merge remote-tracking branch 'bioloop/dev' into dev
f276e5a
@deepakduggirala
Merge remote-tracking branch 'bioloop/main'
2b73857
@deepakduggirala
Merge remote-tracking branch 'bioloop/download' into download
9103a01
@deepakduggirala
Merge remote-tracking branch 'bioloop/download' into download
93432d0
@deepakduggirala
Merge remote-tracking branch 'bioloop/download' into download
6c9ee1b
@deepakduggirala
update config for stage and download workers
608feb9
@deepakduggirala
Merge remote-tracking branch 'bioloop/download' into download
91f69ca
@deepakduggirala
Merge remote-tracking branch 'bioloop/download' into download
a0fe122
@deepakduggirala
Merge remote-tracking branch 'bioloop/download' into download
38c2101
@deepakduggirala
Merge remote-tracking branch 'bioloop/download' into download
25a1965
@deepakduggirala
Merge remote-tracking branch 'bioloop/download' into download
c378df9
@deepakduggirala
Merge remote-tracking branch 'bioloop/download' into download
0a6a535
@deepakduggirala
Merge remote-tracking branch 'bioloop/download' into download
a196bac
@deepakduggirala
Merge remote-tracking branch 'bioloop/download' into download
09cf5cb
@deepakduggirala
Merge remote-tracking branch 'bioloop/download' into download
38a58d0
@deepakduggirala
Merge remote-tracking branch 'bioloop/main'
792f0a3
@deepakduggirala
consider .raw files as data files and show its count in the dataset page
c7a30ac
turn off nginx response buffering to prevent large file download fail…
a2e7afb 
…ures
@deepakduggirala
Merge remote-tracking branch 'bioloop/main'
b2a582c
@deepakduggirala
Merge remote-tracking branch 'bioloop/main'
eb24d25
@deepakduggirala
Merge remote-tracking branch 'bioloop/main'
740eae9
@deepakduggirala
Merge remote-tracking branch 'bioloop/main'
dedc3f0
@deepakduggirala
Merge remote-tracking branch 'bioloop/main'
4f61b28
@deepakduggirala
Merge remote-tracking branch 'bioloop/main'
c3c769d
deepakduggirala and others added 29 commits January 2, 2024 15:54
@deepakduggirala
add legacy import code
1675352
@ryanlong89
Add import_from_sda worker script to legacy.ecosystem.config.js
d3635da
@ryanlong89
Update dir_path in watch.py script
1f01600
@ryanlong89
testing importing from our SDA structure
50ae98b
@ryanlong89
Refactor file extraction and remove unused code
1b9f7a4
@ryanlong89
Refactor move_folders_with_files function to remove unnecessary code
b11b9c1
@deepakduggirala
Merge remote-tracking branch 'bioloop/main' into import_sda
c7b2bcf
@ryanlong89
Remove unused import and delete tar file
da5080c
@ryanlong89
Remove watch script from legacy.ecosystem.config.js
d205de0
@ryanlong89
Add tarfile extraction function
c15f557
@ryanlong89
Refactor file path variable in import_from_sda.py
9b94080
@ryanlong89
Refactor extract_tarfile function to accept Path objects
657897c
@ryanlong89
Update tar file extraction logic
d75c403
@ryanlong89
Refactor extract_tarfile function call in import_from_sda.py
080a325
@ryanlong89
Refactor extraction_dir assignment in import_from_sda.py
ad84833
@ryanlong89
Refactor extraction_dir variable in import_from_sda.py
bacef8f
@ryanlong89
Add print statement to extract_tarfile function
0556936
@ryanlong89
Fix extraction directory in import_from_sda.py
961ca92
@ryanlong89
Refactor extract_tarfile function to extract the tar file to the targ…
a1f762b 
…et directory
@ryanlong89
Refactor extract_tarfile function to simplify code
5b95797
@ryanlong89
Refactor extract_tarfile function to simplify directory extraction
212436f
@ryanlong89
Fix extraction directory deletion
3e31383
@ryanlong89
Refactor extract_tarfile function to allow overriding extracted direc…
93c2d55 
…tory name
@ryanlong89
Refactor extract_tarfile function to remove unnecessary code and dele…
f5e2b49 
…te tar file
@ryanlong89
Refactor extract_tarfile function to remove unnecessary code
53f6a81
@ryanlong89
Refactor file extraction logic in import_from_sda.py
92b8b58
@ryanlong89
Fix file deletion in import_from_sda.py
ce8bb1e
@ryanlong89
documented import script
0799e7e
@ryanlong89
Commented out legacy observer in watch.py
c634a29
@ryanlong89 ryanlong89 closed this Jan 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ryanlong89 @deepakduggirala