Web Scraper is a simple Flask-based API that allows users to retrieve meta data from any website by passing its URL as a request parameter.
To use the API, simply make a GET request to the following endpoint:
https://web-scrapper-coral.vercel.app/api/read_web_meta_data?url={website_url}
Here, {website_url} refers to the URL of the website whose metadata you want to retrieve. The API returns a JSON object containing the website's title, description, and image etc.
For example, to retrieve the metadata for https://www.theverge.com, you would make the following request:
https://web-scrapper-coral.vercel.app/api/read_web_meta_data?url=https://www.theverge.com
Project Setup To set up the project, follow these steps:
- Clone the repository to your local machine.
git clone https://github.com/TheAlphamerc/web-scrapper- Change into the project directory.
cd web-scrapper - Create a virtual environment and activate it.
python3 -m venv venv`source venv/bin/activate
- Install the project dependencies.
pip install -r requirements.txt- Setup the development environment by running these commands in your terminal.
export FLASK_APP=index.py
export FLASK_ENV=development- Run the Flask development server.
python app.pyThe API will be accessible at http://127.0.0.1:5000/api/read_web_meta_data?url={website_url}.
Contributing Contributions to the project are welcome. If you'd like to contribute, please create a pull request with your changes.
Sonu Sharma (Twitter) (Youtube) (Insta) (Dev.to)
If you found this project helpful or you learned something from the source code and want to thank me, consider buying me a cup of ☕
You can also nominate me for Github Star developer program https://stars.github.com/nominate
