-
Notifications
You must be signed in to change notification settings - Fork 55
Creating An Extractor
This guide explains how to create an extractor to extract content from a web domain that is not yet supported by the Downloader For Reddit.
- Please be familiar with the python language.
- You should also be at least somewhat familiar with the website that you are making the extractor for.
- Please do your best to learn about any usage restrictions that they may have and follow any rules and or guidelines that they may have about downloading their content.
- It is preferable to use a websites api if available as opposed to extracting directly from the html.
- For any websites that require you to register an account in order to get an application key, this key will need to be changed upon being merged. Please include this information in your pull request.
- Do not create extractors for websites that have been banned by reddit, these will not be merged.
- Please read the contribution guide.
Step 1: Set up your development environment
It is a good idea before you create an extractor of your own to study some of the existing extractors and see what methods they use to extract content.
The VidbleExtractor is a good example of a simple extractor that parses through html to find and extract content links.
The ImgurExtractor is a more complex example which not only uses Imgur's api, but an external package developed by Imgur to aid in interacting with the website. This extractor also shows various error handling techniques that should be used when at all possible.
This file should be created in the Extractors package and should follow the naming scheme <site>Extractor.py.
Every extractor class is a subclass of the BaseExtractor class. This provides an easy interface to make sure that content and errors are named and handled in a consistent manner and also provides some methods for website interaction.
Your class must take three arguments: post, reddit_object, and content_display_only. Post is the post taken from reddit
that is to be extracted and reddit_object is the user or subreddit it is to be extracted for.
The third argument content_display_only should default to False. This is a variable only used by the UserFinder class for
display purposes.
These methods must be supplied to the super call of the base extractor.
from Extractors.BaseExtractor import BaseExtractor
class WebsiteExtractor(BaseExtractor):
def __init__(self, post, reddit_object, content_display_only=False):
super().__init__(post, reddit_object, content_display_only)This is a list or tuple of strings that contain some unique identifying text that is contained in every url from the website. This is how the extraction runner knows which extractor to use based on the supplied url.
But first, here are some methods that should not be overridden. These are methods to handle common functions for you and to aid in interacting with websites.
This method is responsible for making the Content object from the extracted links and data that you extract. It should be called from each of your overridden extractor methods once the link urls and data have been extracted.
Below is the method declaration and documentation that explains the variables that must be supplied and what they do.
Note: the count variable defaults to None and should only be supplied for an album extraction or a situation in which multiple links are extracted from the same url.
def make_content(self, url, file_name, extension, count=None):
"""
Takes content elements that are extracted and creates a Content object with the extracted parts and the global
extractor items, then sends the new Content object to the extracted content list.
:param url: The url of the content item.
:param file_name: The file name of the content item, either the post name or the album id depending on user
settings.
:param count: The number in an album sequence that the supplied url belongs. Used to number the file.
:param extension: The extension of the supplied url and the url used for the downloaded file.
:type url: str
:type file_name: str
:type extension: str
:type count: int
"""This method handles logging, reporting, and cleanup of links that fail to extract. This should be called when things go wrong.
Below is the declaration for this method.
def handle_failed_extract(self, message=None, save=False, **kwargs):This method has a standard log and output message that indicates a failed extract. It also logs several extractor parameters
(see get_log_data below) to help diagnose problems from the log. There are no required arguments that you must supply
to use this method. Calling this method without supplying any parameters will suffice for most situations.
The message parameter will only be seen by the user in the output window. Only provide a message parameter if you feel it is of some use to the user.
Be careful in telling this method to save a post. Posts should only be saved in rare circumstances, such as connection errors, in which the error is not likely to be repeated if extraction is attempted for the post again. Saving posts that will always fail to extract will lead to many saved posts in the reddit_object's saved post list. This will use memory unnecessarily and the posts can only be removed by the user, which is not ideal.
Keyword arguments are used to log parameters that will be useful in diagnosing extractor problems from the log. These
arguments are only used in the log. Do not supply any of the variables that are supplied in the get_log_data method (below)
as these are called automatically and appear in each log message generated from a BaseExtractor subclass.
Example method use:
def rate_limit_exceeded_error(self):
message = 'Imgur rate limit exceeded'
self.handle_failed_extract(message=message, save=True, imgur_error_message='rate limit exceeded')In this case we save the post because it failed due to exceeding the rate limit, this is unlikely to happen next time.
The imgur_error_message will appear as text in the log file indicating the message that imgur provided.
This method is called automatically to log useful variables and should not be used elsewhere. It is shown below only as a
reference of what variables are logged automatically and need not be supplied as keyword arguments to the handle_failed_extracts
method.
def get_log_data(self):
"""
Returns a loggable dictionary of the extractors current variables to be put into the log.
"""
return {'url': self.url,
'user': self.user,
'subreddit': self.subreddit,
'post_title': self.post_title,
'creation_date': self.creation_date,
'save_path': self.save_path,
'content_display_only': self.content_display_only,
'subreddit_save_method': self.subreddit_save_method,
'name_downloads_by': self.name_downloads_by,
'extracted_content_count': len(self.extracted_content),
'failed_extract_message_count': len(self.failed_extract_messages),
'failed_extracts_to_save_count': len(self.failed_extracts_to_save)}Used to get the file name of the extracted url depending on the settings from the supplied user or subreddit. The
image/album id should be extracted along with the downloadable url from the extractor you build. This id should then be
supplied to this method and the return value used as the file_name variable supplied to the make_content method.
Used when calling a website api to get the json data from the response. Returns the json data if available and handles without errors if it fails
Gets the raw html text from a url response and handles without errors if unsuccessful. Mostly used for the html parser extraction process.
This method must be overridden.
This is responsible for determining if an extraction is a single, album, direct, or other type of extraction. It should determine which of the following methods to call depending on how your website is set up.
The content of this method should be wrapped in a try/except statement to handle any errors that occur if a proper extraction method cannot be determined.
def extract_content(self):
try:
# your assignment code here
except:
message = 'Failed to locate content'
self.handle_failed_extract(message=message, extractor_error_message=message)Each of the methods below should make use of the make_content method.
This method is used to extract a single link from a container website from a url that does not end in an extension (e.g. https://imgur.com/GJYYNle). An extracted link will be a url that ends with an extension (.jpg, .png, .webm)
This method extracts several links from an album hosted on the site. This method will need to provide some means of keeping a count of the content extracted. Please ensure that extracted content is numbered the same way that it appears on the website.
make_content will be called for each link extracted with this method. Provide the count variable as a named argument
to the make_content method. (self.make_content(extracted_url, extracted_file_name, count=count_var))
This method is fully functional in the BaseExtractor class, there should be no need to override it. This method should be assigned
in your extract_content method when the url is a downloadable url which ends in an extension.
There may be other methods necessary for types of extraction specific to the website you are extracting from. Please try to make your extractor as complete as possible and cover every possible situation that may be encountered.
Add your extractor import statement to the end of the imports in this file. This is necessary so the extraction runner can find your extractor when testing a posts url.
Please be sure and test each extraction method of your new extractor to ensure it is functioning correctly. There is a subreddit created specifically for testing extractors where you can post content from your website to test extraction.