Support for SubmitCGAP in cgap-portal (C4-302) #192

netsettler · 2020-08-28T09:56:17Z

I'm putting this up as a draft PR for "early" review. It will still need more changes.
(There is corresponding support in SubmitCGAP repo on the kmp_file_uploads branch.)

Notes:

There are a number of missing unit tests. I have been doing a number of interactive tests to cover some of that, but obviously that's gotta be automated.
Please ignore the file src/encoded/commands/submit_metadata_bundle.py, which is out of date. I'll remove that on my next pass through. The right thing is to use the scripts in the SubmitCGAP repo.
@willronchetti, this version is using buckets named metadata-bundles-<s3_bucket_env>. Is it worth my renaming the buckets to contain the name elasticbeanstalk- like the other bucket envs do? I'm thinking probably yes, but if you don't think it's important I might leave it. Anyway, at this point it does get configured in the .ini files like we had talked about.

Updating branch from master

Code broken down into smaller functions Functions not all finished currently Significant refactoring

Updating branch from master

….toml

Updating branch from master

…ias instead of atid

Updating branch from master

…some documentation to that script.

…me other logic related to errors in s3.

…with_unit_tests

…bundle

…ans_upload_sbr2

…ng semantic problems, mostly syntactic.

willronchetti · 2020-09-18T18:39:24Z

src/encoded/ingestion_listener.py

@@ -374,15 +527,18 @@ def delete_messages(self, messages):
        """
        failed = self.queue_manager.delete_messages(messages)
        while True:
+            debuglog("Trying to delete messages")


Very minor point - I don't love this because we already have @debug_log decorator for routes

This is for local debugging. It only happens under make deploy.
Largely these can go away for production use once the tool is stable, without affecting anything.
This one can probably go away anyway.
debuglog itself is I think available from dcicutils.py so it should go away and just be imported.
I'll make a pass through to clean those things up separately.

netsettler · 2020-09-18T19:52:06Z

OK, some of @sbreiff's work will follow in another PR after this. There was a last-minute permissions error discovered (see C4-311) that will be followed up under separate cover along with others in the SubmitCGAP (C4-190) epic.

sbreiff added 30 commits May 29, 2020 13:57

added some submit functions - WIP

fb329f5

Merge branch 'master' into submit_data

5e8e336

Updating branch from master

added some comments and edits to submit.py

eb9a022

Major changes to submit.py

a6fe747

Code broken down into smaller functions Functions not all finished currently Significant refactoring

unit tests written for some submit.py functions

ee5fd51

test file added for testing submit functions

8c90f7d

Merge branch 'master' into submit_data

00d12b0

Updating branch from master

edits to validate_and_post in submit.py

26c70c1

Added alias exceptions to validation error handling

1f8cddf

code added for creation of case and report items in submit.py

e51f5e5

cgap_submit_test.xlsx modified for more testing

f72f33f

reorganized code for validation and posting in submit.py

ee73402

a couple submission unit tests added

812340a

script to test submission code so far added

5daf28c

src/encoded/commands/submission_test.py added to scripts in pyproject…

41515ce

….toml

VirtualAppError handling modified to adapt to changes in dcicutils

432f087

Merge branch 'master' into submit_data

14be035

Updating branch from master

bug fixes so that compare_with_db in submit.py works properly with al…

273af15

…ias instead of atid

patch logic for submit.py extracted to new compare_fields function

1f9667f

print statements restructured for submission-test command

5edf747

Merge branch 'master' into submit_data

dbe629c

Updating branch from master

only generate reports when required in submit.py

f6145ea

Merge branch 'master' into submit_data

4992a3f

Updating branch from master

edit to code for handling report items in submit.py

6dcf246

Merged master and fixed conflict in datafixtures

2bbb948

family alias based on proband id rather than family id in submit.py

847e01f

added more metadata fields to digest for submit.py

c7523c7

added a few more fields to sample from accessioning worksheet

7d49c47

more accessioning fields digested by submit.py

be81d70

digestion of second individual id in submit.py

9dced9b

sbreiff and others added 4 commits September 1, 2020 17:25

changes to submission_test.py script to acct for recent submit changes

3d39f28

fixed merge conflicts after pull from remote

fda4a82

Merge branch 'master' into kmp_submit_data_sans_upload_sbr2

ca62ac4

Bump beta version.

bcbe81d

netsettler marked this pull request as ready for review September 8, 2020 19:29

netsettler changed the title ~~Support for SubmitCGAP~~ Support for SubmitCGAP in cgap-portal (C4-302) Sep 8, 2020

netsettler and others added 19 commits September 8, 2020 23:26

Make the submission-test script actually test some expectations. Add …

211121a

…some documentation to that script.

Use latest dcicutils.

9eacb65

Add metadata_bundles_bucket to test settings.

aeefc31

Require the 'add' permission to submit_for_ingestion endpoint. Fix so…

d67d2d1

…me other logic related to errors in s3.

Add some unit tests.

e5b2567

Merge branch 'master' into kmp_submit_data_sans_upload_sbr2

d2f6bb2

Merge branch 'kmp_submit_data_sans_upload_sbr2' into kmp_submit_data_…

ca3d308

…with_unit_tests

Add and test full_class_name.

7c76243

Make SubmissionFailure an HTTPServerError.

ded4df4

Add some unit tests for cases of failing s3 interactions.

704537f

PEP8 considerations.

5b7825f

Merge branch 'master' into kmp_submit_data_with_unit_tests

8580400

Merge branch 'master' into kmp_submit_data_with_unit_tests

e0f0497

Merge branch 'master' into kmp_submit_data_sans_upload_sbr2

dfac7df

param 'final_json' changed to 'result' to fix bug in submit_metadata_…

090f5c6

…bundle

Merge branch 'master' into kmp_submit_data_with_unit_tests

4146fba

Merge branch 'kmp_submit_data_with_unit_tests' into kmp_submit_data_s…

846529c

…ans_upload_sbr2

Bump minor version to 3.1.0, end the beta.

c03d7b5

Fix some problems that PyCharm found in submit.py, some of them lurki…

2770f4c

…ng semantic problems, mostly syntactic.

netsettler requested a review from willronchetti September 18, 2020 18:31

willronchetti approved these changes Sep 18, 2020

View reviewed changes

netsettler merged commit 7fd9d33 into master Sep 18, 2020

netsettler deleted the kmp_submit_data_sans_upload_sbr2 branch September 25, 2020 13:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for SubmitCGAP in cgap-portal (C4-302) #192

Support for SubmitCGAP in cgap-portal (C4-302) #192

netsettler commented Aug 28, 2020 •

edited

Loading

willronchetti Sep 18, 2020

netsettler Sep 18, 2020

netsettler commented Sep 18, 2020 •

edited

Loading

Support for SubmitCGAP in cgap-portal (C4-302) #192

Support for SubmitCGAP in cgap-portal (C4-302) #192

Conversation

netsettler commented Aug 28, 2020 • edited Loading

willronchetti Sep 18, 2020

Choose a reason for hiding this comment

netsettler Sep 18, 2020

Choose a reason for hiding this comment

netsettler commented Sep 18, 2020 • edited Loading

netsettler commented Aug 28, 2020 •

edited

Loading

netsettler commented Sep 18, 2020 •

edited

Loading