Add backend to support uploading large files. #2906

KhalidRmb · 2020-07-13T14:05:02Z

Adds APIs and util methods to enable generation of a presigned url and enable the pipeline to upload large annotation files and submission files through CLI.

Related: Cloud-CV/evalai-cli#277

apps/base/utils.py

apps/challenges/views.py

apps/base/utils.py

apps/challenges/views.py

apps/jobs/views.py

KhalidRmb · 2020-07-17T19:28:26Z

@RishabhJain2018 @deshraj @Ram81 PTAL.

apps/base/utils.py

apps/challenges/views.py

apps/jobs/views.py

apps/challenges/views.py

apps/base/utils.py

apps/jobs/views.py

apps/jobs/urls.py

apps/jobs/views.py

Ram81

LGTM 👍 . @RishabhJain2018 can we delpoy this to staging. I can test it once on staging

RishabhJain2018 · 2020-07-23T04:57:58Z

apps/base/utils.py

@@ -138,6 +138,17 @@ def send_email(
    return


+def get_aws_secret_keys():


We do have this function, can you please use that instead of creating a new one.
cc: @Ram81

Ca you point me to this? Didn't find one in base/utils.py

I think it is in challenge/utils.py

Okay...I've changed the function to get the storage bucket name as well when we are not using host's credentials.
But, do we need to add the bucket name field to the challenge model as well, to get the host's bucket name to the aws_keys when using host credentials? (I can make a separate PR for this.) @RishabhJain2018 @Ram81

Original function for reference:

EvalAI/apps/challenges/utils.py

Line 93 in 3e7286e

def get_aws_credentials_for_challenge(challenge_pk):

apps/base/utils.py

apps/challenges/urls.py

apps/base/utils.py

RishabhJain2018 · 2020-07-23T05:00:16Z

apps/base/utils.py

@@ -168,6 +179,31 @@ def get_boto3_client(resource, aws_keys):
        logger.exception(e)


+def get_presigned_url_for_file_upload(file_key):
+    """
+    Function to get the presigned url to upload a file to s3 with name file_name and key as file_key.


file_key -> Can we please change it something more meaningful.

It's clear that it's the key for the file on S3 right. What else can we call it... file_key_on_s3? cc: @Ram81

apps/jobs/views.py

apps/jobs/urls.py

apps/jobs/views.py

apps/jobs/urls.py

apps/base/utils.py

…PreSigned

RishabhJain2018 · 2020-08-02T16:49:33Z

@KhalidRmb Is this complete?
cc: @Ram81

KhalidRmb · 2020-08-03T14:03:23Z

@KhalidRmb Is this complete?
cc: @Ram81

@RishabhJain2018 Yes

Ram81 · 2020-08-03T14:11:32Z

@RishabhJain2018 I haven't tested the PR yet. Are we planning to test this one staging?

RishabhJain2018 · 2020-08-04T02:19:06Z

@RishabhJain2018 I haven't tested the PR yet. Are we planning to test this one staging?

Yes!

Ram81 · 2020-08-04T08:43:43Z

@KhalidRmb did you also test these changes (for test annotations) with auto launch workers feature when challenge is approved? Because we will be auto approving challenges and launching workers before challenge approval would this break submission worker launch flow?
cc: @RishabhJain2018

KhalidRmb · 2020-08-04T11:14:12Z

@KhalidRmb did you also test these changes (for test annotations) with auto launch workers feature when challenge is approved? Because we will be auto approving challenges and launching workers before challenge approval would this break submission worker launch flow?

I thought the admin won't approve a challenge without an annotation file. But if we're auto approving a challenge, the worker will fail after starting since there's no file yet to fetch. We'll have to catch this case in the auto approval pipeline, where we should check for annotation file being present before auto approving a challenge?

On top this I think we should show an appropriate error message is case the host tries to start the worker through manage tab without an annotation file.

…PreSigned

Ram81 · 2020-08-14T06:16:38Z

@KhalidRmb can you resolve conflicts so that we can test this on staging

codecov-commenter · 2020-08-16T01:39:47Z

Codecov Report

Merging #2906 into master will increase coverage by 1.09%.
The diff coverage is 25.99%.

@@            Coverage Diff             @@
##           master    #2906      +/-   ##
==========================================
+ Coverage   72.93%   74.03%   +1.09%     
==========================================
  Files          83       20      -63     
  Lines        5368     3019    -2349     
==========================================
- Hits         3915     2235    -1680     
+ Misses       1453      784     -669

Impacted Files	Coverage Δ
...ontend/src/js/controllers/featuredChallengeCtrl.js	`68.99% <ø> (ø)`
frontend/src/js/controllers/authCtrl.js	`58.58% <3.70%> (-8.28%)`	⬇️
frontend/src/js/controllers/permissionCtrl.js	`36.36% <22.22%> (-63.64%)`	⬇️
frontend/src/js/controllers/challengeCtrl.js	`66.18% <22.48%> (-7.52%)`	⬇️
frontend/src/js/controllers/challengeListCtrl.js	`95.74% <50.00%> (+1.06%)`	⬆️
...ntend/src/js/controllers/challengeHostTeamsCtrl.js	`70.50% <66.66%> (-1.18%)`	⬇️
frontend/src/js/controllers/teamsCtrl.js	`71.17% <66.66%> (ø)`
frontend/src/js/controllers/changePwdCtrl.js	`100.00% <100.00%> (ø)`
frontend/src/js/controllers/dashCtrl.js	`99.00% <100.00%> (+0.08%)`	⬆️
settings/common.py
... and 36 more

Impacted Files	Coverage Δ
...ontend/src/js/controllers/featuredChallengeCtrl.js	`68.99% <ø> (ø)`
frontend/src/js/controllers/authCtrl.js	`58.58% <3.70%> (-8.28%)`	⬇️
frontend/src/js/controllers/permissionCtrl.js	`36.36% <22.22%> (-63.64%)`	⬇️
frontend/src/js/controllers/challengeCtrl.js	`66.18% <22.48%> (-7.52%)`	⬇️
frontend/src/js/controllers/challengeListCtrl.js	`95.74% <50.00%> (+1.06%)`	⬆️
...ntend/src/js/controllers/challengeHostTeamsCtrl.js	`70.50% <66.66%> (-1.18%)`	⬇️
frontend/src/js/controllers/teamsCtrl.js	`71.17% <66.66%> (ø)`
frontend/src/js/controllers/changePwdCtrl.js	`100.00% <100.00%> (ø)`
frontend/src/js/controllers/dashCtrl.js	`99.00% <100.00%> (+0.08%)`	⬆️
settings/common.py
... and 36 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 334de84...707be4d. Read the comment docs.

Add backend to support uploading large files.

48e4ff3

RishabhJain2018 requested a review from Ram81 July 13, 2020 22:28

Ram81 requested changes Jul 15, 2020

View reviewed changes

apps/base/utils.py Outdated Show resolved Hide resolved

apps/challenges/views.py Outdated Show resolved Hide resolved

apps/base/utils.py Outdated Show resolved Hide resolved

apps/challenges/views.py Outdated Show resolved Hide resolved

apps/jobs/views.py Outdated Show resolved Hide resolved

KhalidRmb added 3 commits July 15, 2020 19:05

Changes

8cc205d

Changes

c8218f5

Changes

101a140

KhalidRmb changed the title ~~(WIP) Add backend to support uploading large files.~~ Add backend to support uploading large files. Jul 16, 2020

KhalidRmb added 6 commits July 16, 2020 22:22

Black reformat.

3ef6814

Move challenge approval validation.

1946aee

Change client method for presigned url

b59663e

Fixes

569dd08

Fixes

f1b212a

Black reformat

229bd14

KhalidRmb added 2 commits July 18, 2020 01:01

Fix flake

619deec

Fix flake

755e544

Ram81 requested changes Jul 18, 2020

View reviewed changes

KhalidRmb added 5 commits July 20, 2020 12:04

Req changes

87e665e

Merge remote-tracking branch 'upstream/master' into PreSigned

1ad5750

Presigned url changes

e9a3b98

Flake8

452763b

Minor change

289a2e3

Ram81 requested changes Jul 20, 2020

View reviewed changes

apps/jobs/views.py Outdated Show resolved Hide resolved

apps/challenges/views.py Show resolved Hide resolved

apps/base/utils.py Outdated Show resolved Hide resolved

apps/jobs/views.py Show resolved Hide resolved

KhalidRmb added 2 commits July 20, 2020 22:39

Make requested changes.

19a2d2d

Make req changes

32f8192

Ram81 requested changes Jul 21, 2020

View reviewed changes

apps/jobs/urls.py Outdated Show resolved Hide resolved

apps/jobs/views.py Outdated Show resolved Hide resolved

apps/jobs/views.py Outdated Show resolved Hide resolved

apps/jobs/views.py Outdated Show resolved Hide resolved

apps/jobs/views.py Outdated Show resolved Hide resolved

Make requested changes.

6d8db4f

Ram81 approved these changes Jul 21, 2020

View reviewed changes

RishabhJain2018 requested changes Jul 23, 2020

View reviewed changes

Merge remote-tracking branch 'upstream/master' into PreSigned

f824c1d

KhalidRmb added 2 commits July 24, 2020 18:36

Requested changes.

20606e6

Change get_aws_credentials_for_challenge function

b186c44

Ram81 reviewed Jul 27, 2020

View reviewed changes

apps/base/utils.py Outdated Show resolved Hide resolved

KhalidRmb and others added 6 commits July 28, 2020 22:10

Replace file

4b52502

Changes

13cca1c

Merge branch 'master' into PreSigned

6163b3e

Merge remote-tracking branch 'upstream/master' into PreSigned

d9e6522

Merge branch 'PreSigned' of https://github.com/KhalidRmb/EvalAI into …

581281c

…PreSigned

Fix settings

0f6739a

KhalidRmb and others added 2 commits August 3, 2020 19:30

Add dev fallback

5577264

Merge branch 'master' into PreSigned

d9ce24b

Merge branch 'master' into PreSigned

313d15d

KhalidRmb added 2 commits August 4, 2020 18:15

Add to docstrings

da15147

Merge branch 'PreSigned' of https://github.com/KhalidRmb/EvalAI into …

6327ebf

…PreSigned

Merge branch 'master' into PreSigned

0fdf6aa

RishabhJain2018 approved these changes Aug 16, 2020

View reviewed changes

Merge branch 'master' into PreSigned

707be4d

RishabhJain2018 added the can-be-merged label Aug 16, 2020

RishabhJain2018 merged commit ac416ec into Cloud-CV:master Aug 16, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add backend to support uploading large files. #2906

Add backend to support uploading large files. #2906

KhalidRmb commented Jul 13, 2020

KhalidRmb commented Jul 17, 2020

Ram81 left a comment

RishabhJain2018 Jul 23, 2020

KhalidRmb Jul 24, 2020

RishabhJain2018 Jul 24, 2020

KhalidRmb Jul 26, 2020 •

edited

Loading

RishabhJain2018 Jul 23, 2020

KhalidRmb Jul 24, 2020

RishabhJain2018 commented Aug 2, 2020

KhalidRmb commented Aug 3, 2020

Ram81 commented Aug 3, 2020

RishabhJain2018 commented Aug 4, 2020

Ram81 commented Aug 4, 2020 •

edited

Loading

KhalidRmb commented Aug 4, 2020 •

edited

Loading

Ram81 commented Aug 14, 2020

codecov-commenter commented Aug 16, 2020

		@@ -138,6 +138,17 @@ def send_email(
		return


		def get_aws_secret_keys():

Add backend to support uploading large files. #2906

Add backend to support uploading large files. #2906

Conversation

KhalidRmb commented Jul 13, 2020

KhalidRmb commented Jul 17, 2020

Ram81 left a comment

Choose a reason for hiding this comment

RishabhJain2018 Jul 23, 2020

Choose a reason for hiding this comment

KhalidRmb Jul 24, 2020

Choose a reason for hiding this comment

RishabhJain2018 Jul 24, 2020

Choose a reason for hiding this comment

KhalidRmb Jul 26, 2020 • edited Loading

Choose a reason for hiding this comment

RishabhJain2018 Jul 23, 2020

Choose a reason for hiding this comment

KhalidRmb Jul 24, 2020

Choose a reason for hiding this comment

RishabhJain2018 commented Aug 2, 2020

KhalidRmb commented Aug 3, 2020

Ram81 commented Aug 3, 2020

RishabhJain2018 commented Aug 4, 2020

Ram81 commented Aug 4, 2020 • edited Loading

KhalidRmb commented Aug 4, 2020 • edited Loading

Ram81 commented Aug 14, 2020

codecov-commenter commented Aug 16, 2020

Codecov Report

KhalidRmb Jul 26, 2020 •

edited

Loading

Ram81 commented Aug 4, 2020 •

edited

Loading

KhalidRmb commented Aug 4, 2020 •

edited

Loading