Task Report: Data Scraping and Database Interaction

Objective: The objective of this task was to develop a Python application that scrapes data from a website and an external API, then stores it in a MongoDB database. The task is divided into two parts:

1. Part A: Scraping user data from an external API and storing it in a MongoDB database.
2. Part B: Scraping book data from a website and storing it in a MongoDB database.

Functional Requirements:

1. Login to the website and create an app_id: Obtain an API key (app_id) to access the APIs.
2. Fetch Users Data: Fetch users' data from the API endpoint https://dummyapi.io/data/v1/user and store it in the users table of the database.
3. Fetch Users' Posts Data: For each user fetched, retrieve their corresponding posts data using the API endpoint https://dummyapi.io/data/v1/user/{{user_id}}/post. Store this data in the database.

Approach: The task was approached systematically, with each part divided into distinct modules and scripts. Here's how the approach was implemented:

1. Part A:

   • Database Setup: Establish a connection with the MongoDB server using a connection function.
   • Fetch Users Data: Develop a script to fetch user data from the API using the requests library. Process and store this data in the MongoDB database using PyMongo.

2. Part B:

   • Database Setup: Create a MongoDB connection function to establish a connection with the server.
   • Scrape Books Data: Utilize BeautifulSoup for scraping book data from the website. Process the scraped data and store it in the MongoDB database.

Challenges: During development, several challenges were encountered, including:

• Accurate parsing of scraped data, especially dealing with nested HTML structures.
• Ensuring proper interaction with the MongoDB database, including data insertion and connection handling.

Solution: To overcome these challenges, the following solutions were implemented:

• Robust data processing techniques using BeautifulSoup for accurate extraction of relevant information from the website.
• Modular code structure, with separate modules for database interaction, data scraping, and main execution, facilitating better organization and debugging.

Conclusion: The task was completed successfully, with data scraped from both the external API and the website and stored in the MongoDB database. The modular approach ensured code reusability and maintainability, laying a solid foundation for future enhancements.

Recommendation: To improve the application's robustness, implementing error handling mechanisms and logging functionality is recommended. This would aid in debugging and monitoring the application's behavior, especially during data scraping and database interaction.

Future Work: Future iterations of the application could include features such as data validation, update functionalities for existing records, and scheduling automated data scraping tasks. Additionally, implementing a user interface or API endpoints for interacting with the database would enhance usability and functionality. This task provided valuable experience in web scraping, database interaction, and modular application development, setting the stage for building more complex data-driven applications in the future.

How to Run the Code in Your Local Machine: To run the code on your local machine, follow these steps:

1. Install Python: If you haven't already, install Python on your machine. You can download Python from the official website: https://www.python.org/downloads/

2. Install Required Libraries: Use pip, the package installer for Python, to install the required libraries. Run the following commands in your terminal or command prompt:

   pip install pymongo requests beautifulsoup4
   

3. Clone the Repository: Clone the repository containing the code to your local machine using Git:

   git clone <repository-url>
   

4. Navigate to the Project Directory: Use the cd command to navigate to the directory where you cloned the repository:

   cd <project-directory>
   

5. Run the Python Scripts: Execute the Python scripts for Part A and Part B by running the following commands:

   python part_a/fetch_users.py
   python part_a/fetch_posts.py
   python part_b/scrape_books.py
   

6. Or Use Simple Command:

    python main.py
   

   Make sure to replace <repository-url> and <project-directory> with the actual URL of the repository and the directory where you cloned the repository, respectively.

7. Provide App ID: For Part A, you'll need to provide an app_id to access the external API. Update the app_id variable in the fetch_users.py and fetch_posts.py scripts with your own app_id obtained from the API provider.

8. Verify Data: After running the scripts, verify that the data has been successfully scraped and stored in the MongoDB database. You can use a MongoDB client or shell to connect to the database and inspect the collections.

By following these steps, you should be able to run the code successfully on your local machine and interact with the MongoDB database containing the scraped data.