Middle. Java web scraping API

Team members: Andrii Yaroshevych, Pavlo Kryven, Nazar Demchuk, Tymur Krasnianskyi

Task

Develop the solution that will allow you to extract information about a company from its domain. Your application should support the following information:

• Company name
• Twitter URL
• Facebook URL
• Company logo
• Company icon
• Number of employees
• Company address

Solution

We have developed a simple web scraping API based on Spring Boot and Jsoup

Diagrams and used patterns

Diagram of the project structure, use case diagram, and description of the used patterns are available in the tasks folder.

Usage

General usage

The API is available at middle-scraping-api.herokuapp.com

To get information about a company, send a GET request to the scraping.api/scrape endpoint with the domain parameter:

curl https://middle-scraping-api.herokuapp.com/scraping.api/scrape?domain={{domain}}

or, if you prefer to use a UI, go to middle-scraping-api.herokuapp.com

Here, you can enter the domain name and click the submit button to get the information about the company.

Build locally

To build the project, run the following command:

mvn package

Run the tests:

mvn test

Run locally

To run the project, run the following command:

mvn spring-boot:run

Examples

You can find examples of the API usage in the examples directory.

Let's send test request to the API:

Example response:

License

The MIT License (MIT)

Copyright © 2022. Andrii Yaroshevych, Pavlo Kryven, Nazar Demchuk, Tymur Krasnianskyi