-
Notifications
You must be signed in to change notification settings - Fork 767
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update awscli, botocore, boto3 and moto in Travis #2627
Closed
Closed
Changes from 29 commits
Commits
Show all changes
40 commits
Select commit
Hold shift + click to select a range
5c4e91d
Add tests
nikochiko c2c1a30
Add unit tests for restart_workers
nikochiko 09f462a
Merge branch 'master' into aws-utils-test-RestartWorker
nikochiko 6f5c98a
Correct mock.patch syntax
nikochiko 9036276
Merge branch 'aws-utils-test-RestartWorker' of https://github.com/col…
nikochiko 0d0354a
Fix syntax
nikochiko d20446b
Reorder imports and use random client token
nikochiko 47856f5
Use aws_utils.client_token_generator for token
nikochiko 3edeb61
Fix linting
nikochiko 513d706
Reorder imports, add class variables
nikochiko d8d5fbb
Reorder imports
nikochiko 8adb62b
Try fix for deploymentController
nikochiko 718ec95
Merge branch 'aws-utils-test-RestartWorker' of https://github.com/col…
nikochiko 92e4708
Revert trial changes
nikochiko 3451a1f
Add test changes
nikochiko b24e85d
Update .travis.yml
nikochiko bc4a515
Add --update flag
nikochiko 94508c8
Update .travis.yml
nikochiko 6e01ae8
Update test_aws_utils.py
nikochiko df24164
install latest aws deps in travis
nikochiko 3f36692
Merge branch 'master' into UpdateAWSdeps
nikochiko 158d171
Only install awscli without toching other deps
nikochiko 3982354
Add sample tests for aws_utils
nikochiko ccd5933
Fix linting
nikochiko 979e568
Fix typo
nikochiko f87dad8
Try uninstalling and reinstallilng boto3 botocore
nikochiko 034531d
Add -y flag to pip uninstall
nikochiko f4b60c5
Remove versioning from requirements/dev.txt
nikochiko 1752d5c
Try reverting changes to confirm results
nikochiko 562186c
Revert "Try reverting changes to confirm results"
nikochiko b43ca45
Use moto==1.3.8
nikochiko e5cdf4a
Revert last change
nikochiko 2bc8d94
Remove test_aws_utils
nikochiko ed10453
Remove unnecessary change
nikochiko 15cc0ad
Make random change
nikochiko 809458a
Revert random change
nikochiko 74f5bcb
Merge branch 'UpdateAWSdeps' of https://github.com/coldk91ght/EvalAI …
nikochiko b2c4f86
Try some changes
nikochiko 601e3e9
Remove test_aws_utils
nikochiko e43ae8c
Use mock_sqs on BaseTestClass for submission_worker
nikochiko File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -21,4 +21,3 @@ sphinx-markdown-tables==0.0.9 | |
sphinx-rtd-theme==0.3.0 | ||
pytest-django==3.1.2 | ||
Faker==0.7.7 | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,137 @@ | ||
import boto3 | ||
import challenges.aws_utils as aws_utils | ||
import os | ||
|
||
from allauth.account.models import EmailAddress | ||
from challenges.models import Challenge | ||
from datetime import timedelta | ||
from django.contrib.auth.models import User | ||
from django.utils import timezone | ||
from hosts.models import ChallengeHost, ChallengeHostTeam | ||
from http import HTTPStatus | ||
from moto import mock_ecs | ||
from rest_framework.test import APITestCase, APIClient | ||
|
||
|
||
class BaseTestClass(APITestCase): | ||
def setUp(self): | ||
aws_utils.COMMON_SETTINGS_DICT["EXECUTION_ROLE_ARN"] = "arn:aws:iam::us-east-1:012345678910:role/ecsTaskExecutionRole" | ||
|
||
self.client = APIClient(enforce_csrf_checks=True) | ||
|
||
self.user = User.objects.create( | ||
username="someuser", | ||
email="user@test.com", | ||
password="secret_password", | ||
) | ||
|
||
EmailAddress.objects.create( | ||
user=self.user, email="user@test.com", primary=True, verified=True | ||
) | ||
|
||
self.participant_user = User.objects.create( | ||
username="someparticipantuser", | ||
email="participantuser@test.com", | ||
password="secret_password", | ||
) | ||
|
||
EmailAddress.objects.create( | ||
user=self.participant_user, | ||
email="participantuser@test.com", | ||
primary=True, | ||
verified=True, | ||
) | ||
|
||
self.challenge_host_team = ChallengeHostTeam.objects.create( | ||
team_name="Test Challenge Host Team", created_by=self.user | ||
) | ||
|
||
self.challenge = Challenge.objects.create( | ||
title="Test Challenge", | ||
short_description="Short description for test challenge", | ||
description="Description for test challenge", | ||
terms_and_conditions="Terms and conditions for test challenge", | ||
submission_guidelines="Submission guidelines for test challenge", | ||
creator=self.challenge_host_team, | ||
published=False, | ||
is_registration_open=True, | ||
enable_forum=True, | ||
queue="test_queue", | ||
anonymous_leaderboard=False, | ||
start_date=timezone.now() - timedelta(days=2), | ||
end_date=timezone.now() + timedelta(days=1), | ||
approved_by_admin=False, | ||
) | ||
self.challenge.slug = "{}-{}".format( | ||
self.challenge.title.replace(" ", "-").lower(), self.challenge.pk | ||
)[:199] | ||
self.challenge.save() | ||
|
||
self.challenge_host = ChallengeHost.objects.create( | ||
user=self.user, | ||
team_name=self.challenge_host_team, | ||
status=ChallengeHost.ACCEPTED, | ||
permissions=ChallengeHost.ADMIN, | ||
) | ||
self.challenge2 = Challenge.objects.create( | ||
title="Test Challenge 2", | ||
creator=self.challenge_host_team, | ||
queue="test_queue_2", | ||
start_date=timezone.now() - timedelta(days=2), | ||
end_date=timezone.now() + timedelta(days=1), | ||
) | ||
|
||
self.client.force_authenticate(user=self.user) | ||
self.ecs_client = boto3.client( | ||
"ecs", | ||
region_name=os.environ.get("AWS_DEFAULT_REGION", "us-east-1"), | ||
aws_secret_access_key=os.environ.get("AWS_SECRET_ACCESS_KEY"), | ||
aws_access_key_id=os.environ.get("AWS_ACCESS_KEY_ID"), | ||
) | ||
|
||
def set_challenge_workers(self, challenge, num_workers): | ||
challenge.workers = num_workers | ||
challenge.save() | ||
|
||
|
||
@mock_ecs | ||
class BaseAdminCallsClass(BaseTestClass): | ||
def setUp(self): | ||
super(BaseAdminCallsClass, self).setUp() | ||
|
||
self.challenge3 = Challenge.objects.create( | ||
title="Test Challenge 3", | ||
creator=self.challenge_host_team, | ||
queue="test_queue_3", | ||
start_date=timezone.now() - timedelta(days=2), | ||
end_date=timezone.now() + timedelta(days=1), | ||
) | ||
self.ecs_client.create_cluster(clusterName="cluster") | ||
self.client_token = aws_utils.client_token_generator() | ||
|
||
@classmethod | ||
def queryset(cls, pklist): | ||
queryset = Challenge.objects.filter(pk__in=pklist) | ||
queryset = sorted(queryset, key=lambda i: pklist.index(i.pk)) | ||
return queryset | ||
|
||
|
||
class TestCreateServiceByChallengePK(BaseAdminCallsClass): | ||
|
||
def test_create_service_by_challenge_pk(self): | ||
self.ecs_client.create_cluster(clusterName=aws_utils.COMMON_SETTINGS_DICT["CLUSTER"]) | ||
response = aws_utils.create_service_by_challenge_pk(self.ecs_client, self.challenge, self.client_token) | ||
self.assertEqual(response["ResponseMetadata"]["HTTPStatusCode"], HTTPStatus.OK) | ||
self.assertEqual(self.challenge.workers, 1) | ||
|
||
|
||
class TestRegisterTaskDefByChallengePK(BaseAdminCallsClass): | ||
|
||
def test_register_task_def_by_challenge_pk(self): | ||
self.ecs_client.create_cluster(clusterName=aws_utils.COMMON_SETTINGS_DICT["CLUSTER"]) | ||
response = aws_utils.register_task_def_by_challenge_pk( | ||
self.ecs_client, self.challenge.queue, self.challenge) | ||
self.assertEqual( | ||
response["ResponseMetadata"]["HTTPStatusCode"], HTTPStatus.OK) | ||
self.assertEqual( | ||
self.challenge.task_def_arn, response["taskDefinition"]["taskDefinitionArn"]) |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
update boto3, botocore, moto in requirements/dev.txt as well
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it okay if the changes are made in requirements/common.txt?
https://github.com/Cloud-CV/EvalAI/blob/master/requirements/common.txt#L2-L3
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes.