BE-5: Website Availability & Broken Link Detection System Research

[tecnodeveloper]

Description:
Research how to detect whether a website is live or broken before scraping. Understand HTTP status codes, uptime checks, broken link detection methods, and automated monitoring systems. This is required for Price WatchDog to avoid scraping invalid product pages.

---

User Story

Given I have a product URL
When the system checks the link
Then it should confirm if the website is live, broken, or inaccessible before scraping

---

Tasks

---

Website Availability Basics

1. Understand Website Status Checking

   • What is website uptime
   • What is downtime
   • What causes site failure

2. HTTP Status Codes

   • 200 → OK (working site)
   • 301/302 → Redirect
   • 404 → Not Found
   • 500 → Server Error
   • 403 → Forbidden

---

Broken Link Detection

3. What is a Broken Link

   • Dead URL
   • Removed product page
   • Redirect loop

4. Detection Strategy

   • Send HTTP request
   • Check status code
   • Validate page content

---

Request-Based Checking

5. Basic URL Validation

   • Send GET request
   • Check response status
   • Timeout handling

6. Header Inspection

   • Content-Type check
   • Server response validation

---

Advanced Detection Methods

7. Content Validation

   • Detect “Product Not Found” text
   • Detect empty pages
   • Detect error templates

8. Redirect Handling

   • Follow redirects
   • Detect infinite redirect loops

---

Tools for Link Checking

9. HTTP Libraries

   • requests (Python)
   • aiohttp (async requests)

10. Automation Tools

• Playwright
• Selenium for rendering pages

---

Uptime Monitoring Concepts

11. Website Monitoring System

• Periodic URL checks
• Store status history
• Alert on failure

12. Polling Strategy

• Check every 15 min
• Increase interval for stable sites

---

Error Handling

13. Common Failures

• Timeout errors
• DNS failures
• Server overload

14. Retry Logic

• Retry failed requests
• Exponential backoff

---

Performance Optimization

15. Efficient Checking System

• Batch URL checks
• Async requests
• Cache results

---

Scalability Design

16. Large Scale Monitoring

• 1000+ URLs handling
• Queue-based system
• Distributed checking

---

Data Storage Design

17. Store URL Status

• URL
• Status code
• Last checked time
• Availability flag

---

Real-World Use Cases

18. E-Commerce Scenarios

• Product removed
• Out of stock page
• Region-based blocking

---

False Positives Handling

19. Avoid Wrong Detection

• JS-rendered pages
• Lazy loading content
• Bot-block pages

---

Monitoring Strategy

20. Continuous Monitoring System

• Scheduled checks
• Alert system
• Logging failures

---

Acceptance Criteria

• Website availability concepts understood
• Broken link detection strategy defined
• HTTP status handling mapped
• Tools identified
• Monitoring system design completed

---

Testing Steps

1. Test live URL
2. Test broken URL (404)
3. Test redirect URL
4. Test blocked URL (403)
5. Simulate timeout

---

Definition of Done

• Link detection system fully designed
• Monitoring strategy defined
• Tools selected