Home

Welcome to the RSS_Scraper wiki!

RSS and Atom Feed Scraper

Introduction

This Python program is designed to scrape articles from RSS and Atom feeds, check for alert words in the content, and store relevant articles in a SQLite database. Additionally, it generates an HTML file containing hyperlinks to the scraped articles for easy access.

Table of Contents

1. Installation
2. Usage
3. Description
4. Contributing
5. License

Installation

To use this program, follow these steps:

1. Clone this repository to your local machine:

   git clone https://github.com/BobPick/RSS_Scraper.git

2. Navigate to the directory containing the rss.py file.

3. Install the required dependencies:

   pip install feedparser
   pip install beautifulsoup4

Usage

Once installed, you can run the program with the following command:

python rss.py

Description

• The program imports necessary libraries and defines utility functions for HTML cleaning, duplicate purging, and alert word checking.
• It specifies the paths to the second Python program and the file for storing article links.
• The program scrapes articles from specified RSS and Atom feed URLs, processes each article, and stores relevant information in a SQLite database.
• Duplicate entries based on the article link are purged from the database.
• Hyperlinks to relevant articles are appended to an HTML file for easy access.
• Finally, the second Python program is executed.

Contributing

Contributions are welcome! If you find any issues or have suggestions for improvements, please open an issue or submit a pull request.

License

This project is licensed under the Mozilla License. See the LICENSE file for details.