🌐 Universal Web Scraper 🚀

📌 Infosys Springboard Internship Project (Certified Completion)
✅ Python + Gemini Flash + Multi-format Data Export

---

📖 Project Overview

This project was built during my Infosys Springboard Internship, where I focused on developing a robust and modular web scraping tool capable of scraping data from complex directory-style websites and providing the results in various formats including CSV, JSON, and XML.

🎓 Internship Focus Areas:

• Web scraping dynamic & structured web pages
• Clean data storage and export
• UI/UX enhancements via Gemini Flash integration
• Modular, user-driven scraping pipeline

---

🧩 Milestone Breakdown

🥇 Milestone 1: Directory Website Scraper

• ✅ Scraped a complex directory-style website
• ✅ Extracted structured data (name, contact, address, etc.)
• ✅ Exported results into CSV

🥈 Milestone 2: Advanced Data Handling

• ✅ Handled pagination and nested elements
• ✅ Implemented error handling & logging
• ✅ Ensured scraping works even if structure changes slightly

🥉 Milestone 3: Universal Web Scraper (Gemini Flash UI)

• ✅ Built a universal scraper that:
  • Accepts URL
  • Accepts fields to extract
  • Accepts model name (default: Gemini Flash)
• ✅ Scrapes user-defined targets dynamically
• ✅ Exports data in CSV, JSON, XML
• ✅ User gets a Download button after scraping completes

---

💡 Features

• 🌍 Universal scraping engine (input-driven scraping)
• ⚡ Powered by Gemini Flash for responsive, fast performance
• 📄 Supports CSV, XML, and JSON output
• 🧠 Intelligent field detection and schema mapping
• 🛡️ Error-tolerant scraping logic with graceful fallbacks
• 📥 File download support after data generation

---

🔧 Tech Stack

• Language: Python
• Libraries: BeautifulSoup, Requests, Pandas, lxml, Flask (for interface), Gemini Flash
• Tools: VS Code, GitHub, Postman (for testing)
• Export Formats: CSV, JSON, XML

---

---

🖥️ How to Run the Project

1. Clone the Repository

git clone https://github.com/kmanishprogrammar/Universal_Web_Scrapper.git
cd Universal_Web_Scrapper

2. Install Dependencies

pip install -r requirements.txt

3. Run the Universal Scraper UI (Milestone 3)

cd milestone-3
cd task8
cd task8_part4
streamlit run app.py

4. Open in Browser

Visit http://localhost:5000/ or Streamlit's local URL, then:

• Enter the target URL
• Provide the fields to extract
• Click on “Scrape”
• Download data as CSV, JSON, or XML

---

📸 Sample UI

🔗 UI Demo (Google Drive)

---

📜 Certificate

🎓 Successfully completed Infosys Springboard Internship
📄 View Certificate (Google Drive)

---

🎯 Key Learnings

• Scraping directory-style websites with nested structures
• Modular and reusable scraping architecture
• Dynamic input handling for scraping
• Gemini Flash integration with Python backend
• Clean data transformation and export
• User interface design for data scraping tools

---

🙋‍♂️ Author

Name: Manish Kumar
📧 Email: kmanishprogrammar@gmail.com
🔗 GitHub: github.com/kmanishprogrammar
🔗 LinkedIn: linkedin.com/in/manish-kumar-921111256

---

📝 License

This project is for educational and demonstration purposes only.
Feel free to fork, contribute, or customize with attribution.

---

🙏 Acknowledgements

• Infosys Springboard for mentorship and project opportunity
• Open-source libraries and frameworks
• Gemini Flash for seamless UI support