[MXNET-691] Add Email Bot #11861

YuelinZhang0822 · 2018-07-23T21:24:22Z

Description

An email bot which can automatically send daily GitHub issue reports using Amazon Simple Email Service and AWS Lambda.

@lanking520 @Roshrini
@nswamy @marcoabreu @eric-haibin-lin

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

lanking520 · 2018-07-23T21:33:25Z

mxnet-bot/EmailBot/EmailBot.py

+    # Send emails using Amazon Simple Email Service(SES)
+    # Sender's email address must be verified.
+    # Replace it with sender's email address
+    sender = "sender@email.com"


Can we also set this as an Environment Variable instead of dig into the code and fix it?
Does the user required to use Gmail to send Email, can MS Email also using the same tool?

Yes, every email address verified in SES can use this tool

lanking520 · 2018-07-23T21:35:41Z

mxnet-bot/EmailBot/EmailBot.py

+    sender = "sender@email.com"
+    # Recipients' email address must be verified
+    # Replace it with recipients' email addresses
+    recipients = ["recipient1@email.com", "recipient2@email.com"]


Same applies to here, can user define them outside? How about storing them into a json file and then use the code to load these information?

Store information in a json file is approachable

lanking520 · 2018-07-23T21:36:03Z

mxnet-bot/EmailBot/EmailBot.py

+    # Replace it with recipients' email addresses
+    recipients = ["recipient1@email.com", "recipient2@email.com"]
+    # If necessary, replace us-west-2 with the AWS Region you're using for Amazon SES.
+    aws_region = "us-west-2"


Same concerns mentioned above

lanking520 · 2018-07-23T21:37:24Z

mxnet-bot/EmailBot/EmailBot.py

+            return 1
+        # response.headers['link'] will looks like:
+        # <https://api.github.com/repositories/34864402/issues?state=all&page=387>; rel="last"
+        # In this case we need to extrac '387' as the count of pages


haojin2 · 2018-07-24T18:05:47Z

@CathyZhang0822 test test

lanking520 · 2018-07-31T18:45:46Z

@haojin2 Test added, review?

Roshrini

Nice work Cathy!
Added a few comments:

Remove extra empty lines from all files.
Add docstring comment at the top of file mentioning usage of that particular file.

Roshrini · 2018-08-01T20:41:06Z

mxnet-bot/EmailBot/EmailBot.py

+logging.getLogger('botocore').setLevel(logging.CRITICAL)
+
+LOGGER = logging.getLogger(__name__)
+


Remove empty lines

Roshrini · 2018-08-01T20:41:30Z

mxnet-bot/EmailBot/EmailBot.py

+
+
+class EmailBot:
+


Same as above

Roshrini · 2018-08-01T20:43:02Z

mxnet-bot/EmailBot/EmailBot.py

+    # Replace it with recipients' email addresses
+    recipients = ["recipient1@email.com", "recipient2@email.com"]
+    # If necessary, replace us-west-2 with the AWS Region you're using for Amazon SES.
+    aws_region = "us-west-2"


Roshrini · 2018-08-01T20:43:43Z

mxnet-bot/EmailBot/EmailBot.py

+        self.end = datetime.datetime.today()+datetime.timedelta(days=2)
+
+    def __clean_string(self, raw_string, sub_string):
+        # covert all non-alphanumeric characters from raw_string to sub_string


Roshrini · 2018-08-01T20:46:54Z

mxnet-bot/EmailBot/EmailBot.py

+        # In this case we need to extrac '387' as the count of pages
+        return int(self.__clean_string(response.headers['link'], " ").split()[-3])
+
+    def read_repo(self, periodically=True):


add method description in comments

Done. All methods descriptions are added.

Roshrini · 2018-08-01T20:53:38Z

mxnet-bot/EmailBot/EmailBot.py

+                non_responded += [{k: v for k, v in item.items()
+                                   if k in ['number', 'html_url', 'title']}]
+                non_responded_urls = non_responded_urls + url
+                if (datetime.datetime.strptime("2018-05-15", "%Y-%m-%d") < created


Keep sla_start_date variable in class then, use that variable everywhere instead

Roshrini · 2018-08-01T20:55:37Z

mxnet-bot/EmailBot/README.md

+* [Verify Email Addresses](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/verify-email-addresses-procedure.html) Go to AWS console -> SES -> Email Addresses to verify email address. Then fill in `sender` and `recipients` in `EmailBot.py`.
+* Test the Lambda Function. On the lambda function's console, click Test.
+
+


Remove these extra lines

Roshrini · 2018-08-01T20:58:03Z

mxnet-bot/README.md

@@ -0,0 +1,25 @@
+# MXNet Bot
+This is a directory contains bots served to improve operational efficiency.


*This directory contains bots served to improve operational efficiency.

Roshrini · 2018-08-01T20:58:31Z

mxnet-bot/README.md

+## Label Bot
+This bot will help automate/simplify issue labeling process, which mainly contains 3 parts:
+* Machine Learning part:
+  A web server built based on AWS Elastic Beanstalk which can response to GET/POST requests and realize self-maintenance. It mainly has 2 features:


Add link to "AWS Elastic Beanstalk"

1. Set constructor arguments for EmailBot class 2. Automate deployment process 3. Improve coding style 4. Edit README

Add Email bot

0f69d4c

YuelinZhang0822 changed the title ~~[MXNET-691][W Add Email Bot~~ [MXNET-691][WIP] Add Email Bot Jul 23, 2018

Add License

e56cb5a

lanking520 suggested changes Jul 23, 2018

View reviewed changes

Add unit tests

dc9d0ef

YuelinZhang0822 changed the title ~~[MXNET-691][WIP] Add Email Bot~~ [MXNET-691] Add Email Bot Jul 30, 2018

Roshrini suggested changes Aug 1, 2018

View reviewed changes

YuelinZhang0822 mentioned this pull request Aug 5, 2018

Email Bot [Send daily github issue reports] MXNetEdge/mxnet-infrastructure#3

Merged

3 tasks

revise code

f5948f6

1. Set constructor arguments for EmailBot class 2. Automate deployment process 3. Improve coding style 4. Edit README

nswamy added Feature Build pr-awaiting-review PR is waiting for code review labels Aug 9, 2018

YuelinZhang0822 closed this Aug 9, 2018

szha added Feature request and removed Feature labels Nov 14, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MXNET-691] Add Email Bot #11861

[MXNET-691] Add Email Bot #11861

YuelinZhang0822 commented Jul 23, 2018 •

edited

lanking520 Jul 23, 2018

YuelinZhang0822 Jul 23, 2018

lanking520 Jul 23, 2018

YuelinZhang0822 Jul 23, 2018

lanking520 Jul 23, 2018

Roshrini Aug 1, 2018

lanking520 Jul 23, 2018

haojin2 commented Jul 24, 2018

lanking520 commented Jul 31, 2018

Roshrini left a comment

Roshrini Aug 1, 2018

Roshrini Aug 1, 2018

Roshrini Aug 1, 2018

Roshrini Aug 1, 2018

Roshrini Aug 1, 2018

YuelinZhang0822 Aug 7, 2018

Roshrini Aug 1, 2018

Roshrini Aug 1, 2018

Roshrini Aug 1, 2018

Roshrini Aug 1, 2018

YuelinZhang0822 Aug 7, 2018

		logging.getLogger('botocore').setLevel(logging.CRITICAL)

		LOGGER = logging.getLogger(__name__)

		* [Verify Email Addresses](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/verify-email-addresses-procedure.html) Go to AWS console -> SES -> Email Addresses to verify email address. Then fill in `sender` and `recipients` in `EmailBot.py`.
		* Test the Lambda Function. On the lambda function's console, click Test.

		@@ -0,0 +1,25 @@
		# MXNet Bot
		This is a directory contains bots served to improve operational efficiency.

[MXNET-691] Add Email Bot #11861

[MXNET-691] Add Email Bot #11861

Conversation

YuelinZhang0822 commented Jul 23, 2018 • edited

Description

Checklist

Essentials

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haojin2 commented Jul 24, 2018

lanking520 commented Jul 31, 2018

Roshrini left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YuelinZhang0822 commented Jul 23, 2018 •

edited