Home

Welcome to the 15-Web_Metadata_Retrieval wiki!

API Documentation

                                                        Introduction

A self-hosted API for web metadata retrieval, offers the web-scraping functionality when any valid URL is provided. The identified stakeholders are:

• Developers (Who contributes to this open-source project)

• Companies/Developers (Who integrates our service with their product(Web/Mobile App))

• End-users of the system (Everyone who uses our service integrated products.) This will be a very useful tool for those who are interested in extracting the data and use it for any analysis.

                                                         Description
  

1. Registration :

   • Enter Email address
   • Password
   • Confirm password (Password should be min 8 characters. Shouldn't match to email address, and cannot be too common. If so, then it will show an error message in red letters.)

   Otherwise, registration has been performed successfully.

2. Consume API Following are the ways to consume the self-hosted API:

   • API_Key: captured from the registration screen

   • Valid Web URL that is required to scraped

   • Cached= true/false (optional): It is useful for optimising the access time.

3. Accessing with URL

   • Url to access API Endpoint: https://web-metadata.herokuapp.com/api/

   • Input Parameters for GET Request.:

     • api_key ( Enter your valid API key)
     • web_page ( Enter a valid website)
     • cached ( Enter true/false) (default=true) (optional)

   • Metadata API Url: https://web-metadata.herokuapp.com/api/?api_key=34fb8a53-9633-4f84-aa4d-f010c62b853b&format=api&web_page=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DXQgXKtPSzUI

Success Response:

{
  "web_page": "https://www.youtube.com/watch?v=XQgXKtPSzUI",

  "title": "Intro to Web Scraping with Python and Beautiful Soup",

  "description": "Web scraping is a very powerful tool to learn for any data professional. With web scraping the entire internet becomes your database. In this tutorial we sho...",

  "thumbnail": "https://i.ytimg.com/vi/XQgXKtPSzUI/maxresdefault.jpg"

}

Daily limit: Every user is provided 1000 as a daily limit of accessing the API. Example: After calling the API Key 4 times, on checking the daily limit: we can see it has been reduced to 996 out of 1000.

For further details and to get indepth more information refer to : https://docs.google.com/document/d/1chmmfWFlP5Hf5uhSkZ_u31DODUu3Trtp4VTlmLdkvDQ/edit?usp=sharing