Skip to content

v0.0.1

Latest
Latest

Choose a tag to compare

View all tags
@Fionn88 Fionn88 released this 16 Jun 16:30
v0.0.1
733db6f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Description:

This is the initial version of our DevOps News Web Scraping Service. The service is capable of scraping DevOps news articles and storing the data in a database. The functionality is divided into two modes based on the environment variable settings:

1. First Crawl Mode:

  • When FIRST_CRAWL is set to True, the service initializes the database and performs a comprehensive crawl, capturing articles from the past six months.

2. Daily Crawl Mode:

  • When FIRST_CRAWL is not set, the service performs a daily crawl, capturing articles from the past two days.

Features:

  • Scrapes DevOps news articles.
  • Stores scraped data in a database.
  • Two modes of operation (initial and daily crawl) controlled by environment variables.

How to Test:

  1. Set up the environment variable FIRST_CRAWL=True for the initial comprehensive crawl.
  2. Run the service and verify that the database is populated with data from the past six months.
  3. For daily operations, remove or set FIRST_CRAWL=False and verify that only articles from the past two days are added to the database.

Notes:

Ensure to check the documentation for detailed instructions on setting up and running the service.

Assets 2
Loading