Merge pull request #1 from freedomofkeima/feature-rss

[Feature] RSS Feed Notifier

README.md

@@ -1,4 +1,4 @@
-# Maid-chan feat Facebook Messenger (Work In Progress)
+# Maid-chan feat Facebook Messenger
 
 Maid-chan name is inspired from [Sakurasou's Artificial Intelligence](http://sakurasounopetnakanojo.wikia.com/wiki/Maid).
 
@@ -17,6 +17,8 @@ If you have any other ideas, I am accepting contributions :) For developers, you
 <img src="https://freedomofkeima.com/images/maid-chan/daily_night_offerings.png" width="350">  
 - Daily Japanese Kanji & Vocabulary (**Requires** `maidchan_scheduler` running in the background)  
 <img src="https://freedomofkeima.com/images/maid-chan/daily_japanese.png" width="350">
+- [Experimental] RSS Feed Notifier for Anime, Manga, etc which is similar to [my previous RSS Twilio bot](https://github.com/freedomofkeima/rss-twilio-bot) (**Requires** `maidchan_scheduler` running in the background)  
+<img src="https://freedomofkeima.com/images/maid-chan/rss_notification.png" width="350">
 
 All time-related features are currently handled in **UTC+9 (Japan Time)**.
 
@@ -36,6 +38,9 @@ All time-related features are currently handled in **UTC+9 (Japan Time)**.
 <img src="https://freedomofkeima.com/images/maid-chan/japanese_subscribe.png" width="350">  
 - `update name`: By default, Maid-chan will call you with `onii-chan`  
 <img src="https://freedomofkeima.com/images/maid-chan/name.png" width="350">  
+- [Experimental] `subscribe rss`: You could add an RSS feed with its pattern and let Maid-chan notify you when there is an update
+- [Experimental] `unsubscribe rss`: You could remove one of your RSS feed subscription each time you call this command  
+<img src="https://freedomofkeima.com/images/maid-chan/rss_1_manga.png" width="350">  
 - `show profile`: Do you want to know what Maid-chan knows about you? Then, you could use this command!
 
 
@@ -52,7 +57,6 @@ All time-related features are currently handled in **UTC+9 (Japan Time)**.
 
 ## Priority Ideas
 
-- RSS monitoring (similar to my previous RSS Twilio bot: https://github.com/freedomofkeima/rss-twilio-bot)
 - Translate text (feat: https://github.com/soimort/translate-shell)
 - (Admin only) Send link to download at home, e.g.: Youtube (feat https://github.com/soimort/you-get), image files, etc
 
@@ -85,4 +89,4 @@ All time-related features are currently handled in **UTC+9 (Japan Time)**.
 
 This project itself is licensed under MIT License. All images are owned by their respective creators.
 
-Last Updated: January 13, 2017
+Last Updated: January 30, 2017

docs/features.rst

@@ -45,6 +45,13 @@ Currently, Maid-chan has the following features:
     :align: center
     :width: 300pt
 
+- :ref:`RSS Feed Notifier`: The idea is similar to `rss-twilio-bot`_, where all subscribed RSS feed is aggregated to Facebook Messenger via Maid-chan scheduler
+
+.. image:: https://freedomofkeima.com/images/maid-chan/rss_notification.png
+    :alt: maidchan-rss-preset-manga
+    :align: center
+    :width: 300pt
+
 Currently, Maid-chan only supports `Asia/Tokyo` timezone (**UTC +9**).
 
 
@@ -85,10 +92,19 @@ All commands receive 2 parameters: `redis_client` as `RedisDriver` object and `r
 
 **update name** is used to change user's nickname. By default, Maid-chan will use `onii-chan` to call users.
 
+.. autofunction:: maidchan.command.process_subscribe_rss
+
+**subscribe rss** is used to subscribe a new RSS Feed with its pattern and let Maid-chan sends a notification if there is an update.
+
+.. autofunction:: maidchan.command.process_unsubscribe_rss
+
+**unsubscribe rss** is used to remove one of the registered RSS Feed.
+
 .. autofunction:: maidchan.command.process_show_profile
 
 **show profile** is used to show user's nickname, subscription status, and preference.
 
 .. _Primitive: https://github.com/fogleman/primitive
 .. _ChatterBot: https://github.com/gunthercox/ChatterBot
 .. _langdetect: https://github.com/Mimino666/langdetect
+.. _rss-twilio-bot: https://github.com/freedomofkeima/rss-twilio-bot

docs/idea.rst

@@ -5,7 +5,6 @@ Ideas
 Priority Ideas
 --------------
 
-- RSS monitoring (similar to my previous RSS Twilio bot: https://github.com/freedomofkeima/rss-twilio-bot)
 - Translate text (feat: https://github.com/soimort/translate-shell)
 - (Admin only) Send link to download at home, e.g.: Youtube (feat https://github.com/soimort/you-get), image files, etc
 

docs/index.rst

@@ -20,9 +20,9 @@ with billion of people around the world. It also eliminates the necessity of dev
 
 Maid-chan itself is not a general-purpose bot. If you like Japanese culture or have an interest in learning Japanese,
 then Maid-chan will probably useful for you. Currently, it supports several basic features, such as simple chatbot,
-daily scheduler for Kanji & Vocabulary of the day and daily greetings. In addition, Maid-chan can be used as an
-intermediary between mobile and server-side, where you could leverage server capability (CPU) for hard tasks such as
-Machine Learning (image processing).
+daily scheduler for Kanji & Vocabulary of the day, daily greetings, and RSS Feed notifier.
+In addition, Maid-chan can be used as an intermediary between mobile and server-side, where you could leverage server
+capability (CPU) for hard tasks such as Machine Learning (image processing).
 
 Maid-chan is mainly written in Python.
 
@@ -38,6 +38,7 @@ Contents
    chatbot
    offerings
    japanese
+   rss
    idea
 
 New Ideas & Issues

docs/rss.rst

@@ -0,0 +1,56 @@
+=================
+RSS Feed Notifier
+=================
+
+How It Works
+------------
+
+Maid-chan accepts RSS Feed subscription and sends out notification for each RSS update. By default, user can use 2 default preset:
+
+- Manga via `mangaupdates.com`_
+
+.. image:: https://freedomofkeima.com/images/maid-chan/rss_1_manga.png
+    :alt: maidchan-rss-preset-manga
+    :align: center
+    :width: 300pt
+
+- Nyaa (`nyaa.se`_)
+
+.. image:: https://freedomofkeima.com/images/maid-chan/rss_1_nyaa.png
+    :alt: maidchan-rss-preset-manga
+    :align: center
+    :width: 300pt
+
+In addition, user can also use its own custom RSS Feed source.
+
+.. image:: https://freedomofkeima.com/images/maid-chan/rss_2_custom.png
+    :alt: maidchan-rss-preset-manga
+    :align: center
+    :width: 300pt
+
+When the user subscribes using custom RSS, Maid-chan validates whether given URL has a valid RSS format via `feedparser`_ library.
+
+.. autofunction:: maidchan.rss.is_valid_feed_url
+
+Maid-chan uses regex comparison from user's input to the title of feed entries. Maid-chan stores list of all matched titles from the RSS Feed.
+
+.. autofunction:: maidchan.rss.validate_and_create_entry
+
+If there is a new title which is not stored in the database, Maid-chan will send out a notification to the user that a new update is available.
+
+Initially, timestamp is used instead of list of titles. However, some sources (mangaupdates) don't provide timestamp information.
+
+How to Run
+----------
+
+1. You can start running scheduler by executing:
+
+.. code-block:: bash
+
+   $ maidchan_scheduler
+
+2. You can add & remove RSS to Maid-chan's RSS Feed notifier via `subscribe rss` and `unsubscribe rss` command in the Messenger.
+
+.. _mangaupdates.com: https://www.mangaupdates.com/
+.. _nyaa.se: https://www.nyaa.se/
+.. _feedparser: https://github.com/kurtmckee/feedparser

docs/setup.rst

@@ -75,7 +75,7 @@ Maid-chan has 3 executables (recommended to use `screen`):
 
 - **maidchan_primitive**: CPU intensive task which processes image with Machine Learning and returns an abstract GIF (:ref:`Image Processing with Primitive`).
 
-- **maidchan_scheduler**: Scheduler which is used to handle daily and repetitive tasks, such as :ref:`Daily Offerings` and :ref:`Daily Japanese Lesson`.
+- **maidchan_scheduler**: Scheduler which is used to handle daily and repetitive tasks, such as :ref:`Daily Offerings`, :ref:`Daily Japanese Lesson`, and :ref:`RSS Feed Notifier`.
 
 .. _Facebook Messenger Bot: https://developers.facebook.com/docs/messenger-platform/guides/quick-start
 .. _davidchua/pymessenger: https://github.com/davidchua/pymessenger/blob/master/examples/echo_bot.py

maidchan/base.py

@@ -64,7 +64,15 @@ def set_user(self, recipient_id, data):
                   "kanji_level": "N3",
                   "schedules": {
                       "morning_offerings_mt": ... # UTC epoch
-                  }
+                  },
+                  "rss": {
+                      1: {
+                          "url: "...",
+                          "pattern": "...",
+                          "title_list": []
+                      }
+                  },
+                  "rss_id": 1
               }
         """
         redis_key = "{}_{}".format(USER, recipient_id)
@@ -78,6 +86,9 @@ def get_user(self, recipient_id):
             # Update existing DB
             if "schedules" not in data:
                 data["schedules"] = {}
+            if "rss" not in data:
+                data["rss"] = {}
+                data["rss_id"] = 0
             return data
         else:
             return {
@@ -86,7 +97,9 @@ def get_user(self, recipient_id):
                 "morning_time": Constants.DEFAULT_MORNING_TIME,
                 "night_time": Constants.DEFAULT_NIGHT_TIME,
                 "kanji_level": "N3",
-                "schedules": {}
+                "schedules": {},
+                "rss": {},
+                "rss_id": 0
             }
 
     def push_primitive_queue(self, data):

maidchan/cmds/scheduler_worker.py

@@ -1,6 +1,7 @@
 # -*- coding: utf-8 -*-
 import concurrent.futures
 import logging
+import re
 import sys
 import time
 from random import randint
@@ -14,6 +15,7 @@
 from maidchan.offerings import get_morning_offerings_text,\
     get_night_offerings_text, get_offerings_image, remove_offerings_image,\
     SPECIAL
+from maidchan.rss import get_feed
 from pymessenger.bot import Bot
 
 from maidchan.helper import time_to_next_utc_mt
@@ -44,7 +46,7 @@ def send_offerings(recipient_id, text_message, image_path):
         )
 
 
-def process_user(redis_client, recipient_id, metadata, current_mt):
+def process_user_schedules(redis_client, recipient_id, metadata, current_mt):
     user = redis_client.get_user(recipient_id)
     schedules = user["schedules"]
     for schedule_type, mt in schedules.items():
@@ -93,10 +95,52 @@ def process_user(redis_client, recipient_id, metadata, current_mt):
             redis_client.set_user(recipient_id, user)
             logging.info("Japanese scheduler for {} - {} is executed!".format(
                 recipient_id,
-                user["nickname"]
+                user.get("nickname", DEFAULT_NICKNAME)
             ))
 
 
+def process_user_rss(redis_client, recipient_id):
+    user = redis_client.get_user(recipient_id)
+    for key, entry in user["rss"].iteritems():
+        feed = get_feed(entry["url"]).get("entries", {})
+        for record in feed:
+            title = record.get("title", "")
+            try:
+                m = re.search(
+                    entry["pattern"].encode("utf-8").lower(),
+                   title.lower()
+                )
+            except:
+                m = None
+            if m and title not in user["rss"][key]["title_list"]:
+                url = entry["url"]
+                if "link" in record:
+                    url = record["link"]
+                message = "\"{}\" in {} is now available, {}!".format(
+                    title,
+                    url,
+                    user.get("nickname", DEFAULT_NICKNAME)
+                )
+                bot.send_text_message(recipient_id, message)
+                user["rss"][key]["title_list"].append(title)
+                redis_client.set_user(recipient_id, user)
+
+
+def process_user(redis_client, recipient_id, metadata, current_mt):
+    # Process schedule-based operation
+    process_user_schedules(
+        redis_client,
+        recipient_id,
+        metadata,
+        current_mt
+    )
+    # Process RSS-based operation
+    process_user_rss(
+        redis_client,
+        recipient_id
+    )
+
+
 def adjust_offerings_mt(redis_client, users, metadata,
                         morning_offset, night_offset):
     m_adj = morning_offset - metadata.get("morning_offering_mt_offset", 0)