serverless

A playground to test out the serverless functions.

crawler-bs4

It scrapes the content of url given in the GET request.

Technology Stack

• Serverless framework
• Beautiful Soup 4 for scraping
• AWS Lambda
• AWS API Gateway

Test locally in serverless framework:

$ cd crawler-bs4
$ sls invoke local -f main --path events/sample.json

Deploy to AWS:

$ sls deploy

Sample Url & Output:

Request to AWS API Gateway: https://test-id.amazonaws.com/test/crawler-bs4?url=https://www.sonymobile.com/gb/products/phones/xperia-1/

{  
    "data":{  
        "url":"https://www.sonymobile.com/gb/products/phones/xperia-1/",
        "date":"2019-07-12 14:54:05.679168",
        "parsedContent":{  
            "product":{  
                "price":"£849.00",
                "description":"The new Xperia 1 was made with the advanced technology from Sony’s professional monitors, cameras and audio devices, to deliver the best experience in a smartphone. It features the world´s first 21:9 CinemaWide™ 4K HDR OLED display and a pro-quality triple lens camera.",
                "title":"Xperia 1"
            }
        }
    },
    "success":true
}