Add backend for GitHub challenges #2965

KhalidRmb · 2020-08-20T16:07:49Z

This PR includes the backend changes to enable GitHub based challenges.

Changes:

Refactor Abstract out and make relevant changes to Challenge config validation API
Add new API for GitHub challenges MVP
The above API handled both updates and creation for a Github based challenge, and is called from the build step in the host's private repository through Github Actions
Add a field in Challenge model for storing the github_repository identifier which uniquely identifies a private github repository (of the format: <github_account>/<github_repo_name>)
Changes in tests and serialisers to incorporate the changes in the Challenge model

codecov-commenter · 2020-08-20T16:19:23Z

Codecov Report

Merging #2965 into master will increase coverage by 1.09%.
The diff coverage is 25.99%.

@@            Coverage Diff             @@
##           master    #2965      +/-   ##
==========================================
+ Coverage   72.93%   74.03%   +1.09%     
==========================================
  Files          83       20      -63     
  Lines        5368     3019    -2349     
==========================================
- Hits         3915     2235    -1680     
+ Misses       1453      784     -669

Impacted Files	Coverage Δ
...ontend/src/js/controllers/featuredChallengeCtrl.js	`68.99% <ø> (ø)`
frontend/src/js/controllers/authCtrl.js	`58.58% <3.70%> (-8.28%)`	⬇️
frontend/src/js/controllers/permissionCtrl.js	`36.36% <22.22%> (-63.64%)`	⬇️
frontend/src/js/controllers/challengeCtrl.js	`66.18% <22.48%> (-7.52%)`	⬇️
frontend/src/js/controllers/challengeListCtrl.js	`95.74% <50.00%> (+1.06%)`	⬆️
...ntend/src/js/controllers/challengeHostTeamsCtrl.js	`70.50% <66.66%> (-1.18%)`	⬇️
frontend/src/js/controllers/teamsCtrl.js	`71.17% <66.66%> (ø)`
frontend/src/js/controllers/changePwdCtrl.js	`100.00% <100.00%> (ø)`
frontend/src/js/controllers/dashCtrl.js	`99.00% <100.00%> (+0.08%)`	⬆️
apps/web/urls.py
... and 36 more

Impacted Files	Coverage Δ
...ontend/src/js/controllers/featuredChallengeCtrl.js	`68.99% <ø> (ø)`
frontend/src/js/controllers/authCtrl.js	`58.58% <3.70%> (-8.28%)`	⬇️
frontend/src/js/controllers/permissionCtrl.js	`36.36% <22.22%> (-63.64%)`	⬇️
frontend/src/js/controllers/challengeCtrl.js	`66.18% <22.48%> (-7.52%)`	⬇️
frontend/src/js/controllers/challengeListCtrl.js	`95.74% <50.00%> (+1.06%)`	⬆️
...ntend/src/js/controllers/challengeHostTeamsCtrl.js	`70.50% <66.66%> (-1.18%)`	⬇️
frontend/src/js/controllers/teamsCtrl.js	`71.17% <66.66%> (ø)`
frontend/src/js/controllers/changePwdCtrl.js	`100.00% <100.00%> (ø)`
frontend/src/js/controllers/dashCtrl.js	`99.00% <100.00%> (+0.08%)`	⬆️
apps/web/urls.py
... and 36 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fd7147a...9355a29. Read the comment docs.

apps/challenges/challenge_config_utils.py

apps/challenges/views.py

apps/challenges/models.py

apps/challenges/views.py

Ram81 · 2020-09-04T15:11:12Z

apps/challenges/challenge_config_utils.py

-        else:
-            message = (" ERROR: There is no html file present for key"
-                       " {} in phase {}.".format(key, yaml_file_data["name"]))
+        is_valid = True


@KhalidRmb why is this check removed?

Because the validation inside the challenge create API is such that it allows an invalid value for the file. It just replaces it with null before moving on. Validation fails if the entire key is missing. @Ram81 (I'd discussed this with you when working on this, and we decided to keep it this way)

I didn't get the reason for the removal of this check. Can we please discuss about this on slack channel.

Ram81 · 2020-09-04T15:12:37Z

apps/challenges/challenge_config_utils.py

+
+
+def get_value_from_field(data, base_location, field_name):
+    file_path = join(base_location, data.get(field_name))


what about exception handling for missing values for a field name?

If the value is missing, this function will return None. In accordance with the validation. (This is also what happens in the create challenge API)

RishabhJain2018 · 2020-09-08T04:02:02Z

apps/challenges/challenge_config_utils.py

-        else:
-            message = (" ERROR: There is no html file present for key"
-                       " {} in phase {}.".format(key, yaml_file_data["name"]))
+        is_valid = True


I didn't get the reason for the removal of this check. Can we please discuss about this on slack channel.

RishabhJain2018 · 2020-09-08T04:03:43Z

apps/challenges/models.py

@@ -196,6 +200,7 @@ def create_eks_cluster_for_challenge(sender, instance, created, **kwargs):
 class DatasetSplit(TimeStampedModel):
    name = models.CharField(max_length=100)
    codename = models.CharField(max_length=100)
+    config_id = models.IntegerField(default=None, blank=True, null=True)


Can we please add comments for these fields answering why are they needed.

RishabhJain2018 · 2020-09-08T04:03:49Z

apps/challenges/models.py

@@ -263,6 +268,7 @@ def __init__(self, *args, **kwargs):
    is_partial_submission_evaluation_enabled = models.BooleanField(
        default=False
    )
+    config_id = models.IntegerField(default=None, blank=True, null=True)


Same as above

RishabhJain2018 · 2020-09-08T04:03:54Z

apps/challenges/models.py

@@ -317,6 +323,7 @@ def save(self, *args, **kwargs):
 class Leaderboard(TimeStampedModel):

    schema = JSONField()
+    config_id = models.IntegerField(default=None, blank=True, null=True)


Same as above.

RishabhJain2018 · 2020-09-08T04:04:55Z

apps/challenges/views.py

+@permission_classes((permissions.IsAuthenticated, HasVerifiedEmail))
+@authentication_classes((ExpiringTokenAuthentication,))
+def create_or_update_github_challenge(request, challenge_host_team_pk):
+    challenge_queryset = Challenge.objects.filter(


Can we please add check to verify who can use this API?

It should be accessed by hosts only.

Ram81 · 2020-09-08T09:30:44Z

apps/challenges/views.py

+                        challenge_phases_data,
+                        files["challenge_test_annotation_files"],
+                    ):
+                        data["slug"] = "{}-{}-{}".format(


isn't there a util method to get slug?

That's different. That just replaces empty spaces with '-', and is used only for challenge. I can use it here only to modify the codename.

KhalidRmb · 2020-09-09T17:15:52Z

@RishabhJain2018
Done

RishabhJain2018 · 2020-09-10T17:27:40Z

@KhalidRmb Can you please resolve the conflicts here?

KhalidRmb · 2020-09-10T17:48:00Z

@RishabhJain2018 Done

KhalidRmb and others added 3 commits August 20, 2020 21:25

Add backend for GitHub challenges

682f36f

Minor change

10620d1

Merge branch 'master' into GithubChall

9355a29

RishabhJain2018 requested a review from Ram81 August 21, 2020 18:34

Ram81 requested changes Aug 23, 2020

View reviewed changes

KhalidRmb and others added 5 commits August 25, 2020 21:48

Make requested changes and add config_id field to models

5f05713

Merge branch 'master' into GithubChall

cd4e2fc

Fixes

adf7a63

Fix tests

e52b0dd

Merge remote-tracking branch 'upstream/master' into GithubChall

79dd5b6

KhalidRmb mentioned this pull request Aug 31, 2020

Add submission meta attribute validation to config validation API #2917

Closed

Fix test

6cc72df

KhalidRmb mentioned this pull request Sep 2, 2020

Add meta attribute validation to challenge_config_validation API. #2881

Closed

1 task

Merge branch 'master' into GithubChall

a73372a

KhalidRmb mentioned this pull request Sep 3, 2020

GitHub based challenges MVP Cloud-CV/EvalAI-Starters#26

Merged

Ram81 requested changes Sep 4, 2020

View reviewed changes

Merge branch 'master' into GithubChall

7e94278

RishabhJain2018 requested changes Sep 8, 2020

View reviewed changes

Ram81 requested changes Sep 8, 2020

View reviewed changes

KhalidRmb and others added 2 commits September 9, 2020 19:53

Requested changes

939525a

Merge branch 'master' into GithubChall

0f26ec1

Merge branch 'master' into GithubChall

cd6f52f

RishabhJain2018 approved these changes Sep 10, 2020

View reviewed changes

Merge branch 'master' into GithubChall

93ba07c

Merge branch 'master' into GithubChall

0d2aa32

RishabhJain2018 approved these changes Sep 10, 2020

View reviewed changes

RishabhJain2018 merged commit 47af1db into Cloud-CV:master Sep 10, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add backend for GitHub challenges #2965

Add backend for GitHub challenges #2965

KhalidRmb commented Aug 20, 2020 •

edited

Loading

codecov-commenter commented Aug 20, 2020

Ram81 Sep 4, 2020

KhalidRmb Sep 6, 2020 •

edited

Loading

RishabhJain2018 Sep 8, 2020

Ram81 Sep 4, 2020

KhalidRmb Sep 6, 2020

RishabhJain2018 Sep 8, 2020

RishabhJain2018 Sep 8, 2020

RishabhJain2018 Sep 8, 2020

RishabhJain2018 Sep 8, 2020

RishabhJain2018 Sep 8, 2020

RishabhJain2018 Sep 8, 2020

Ram81 Sep 8, 2020

KhalidRmb Sep 9, 2020

KhalidRmb commented Sep 9, 2020

RishabhJain2018 commented Sep 10, 2020

KhalidRmb commented Sep 10, 2020



		def get_value_from_field(data, base_location, field_name):
		file_path = join(base_location, data.get(field_name))

Add backend for GitHub challenges #2965

Add backend for GitHub challenges #2965

Conversation

KhalidRmb commented Aug 20, 2020 • edited Loading

codecov-commenter commented Aug 20, 2020

Codecov Report

Choose a reason for hiding this comment

KhalidRmb Sep 6, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KhalidRmb commented Sep 9, 2020

RishabhJain2018 commented Sep 10, 2020

KhalidRmb commented Sep 10, 2020

KhalidRmb commented Aug 20, 2020 •

edited

Loading

KhalidRmb Sep 6, 2020 •

edited

Loading