Welcome to soal-jawab-scrape 👋

Soal-Jawab-Scrape is a search api inspiration from website https://zulkiflialbakri.com/category/soal-jawab-agama/ .This project developed using APIScrapper and Spring JAVA 17. This project aims to learn how scrape API technology work and also give user to search their answer based on API listed.

Usage

mvn clean compile spring-boot:run -f

Batch JOB (Disable as of now)

• Run every 3am to scrape all the link from website and save into table scrap_website_source
• After the scrape done, the data will be insert into table scrap_website_soal_jawab

API

1. Add configuration

• Please add the link for website that will be scrape

curl --location 'http://localhost:8080/soaljawab/configuration/scrap' \
--data '{
"method": "CREATE",
"url": "https://zulkiflialbakri.com/category/soal-jawab-agama/page/2/"
}'

Below is the method available to be run in the API:
a. LIST - list out all the website to scrape
b. CREATE - create new scrape website
c. CREATE-REPEAT - create new scrape website and repeat the page number
d. UPDATE - update website scrape link
d. DELETE - delete website scrape link

2. Search

curl --location --request POST 'http://localhost:8080/soaljawab/get-websites?keyword=surah'

Response:
{
    "content": [
        {
            "id": "101cb370-522d-4ffe-ba5f-ab7350f625da",
            "active": true,
            "version": 0,
            "restricted": false,
            "createdDate": null,
            "createdBy": null,
            "lastModifiedDate": null,
            "lastModifiedBy": null,
            "author": null,
            "link": "https://zulkiflialbakri.com/mualaf-tidak-mahir-surah-al-fatihah-dalam-solat/",
            "title": "mualaf tidak mahir surah al fatihah dalam solat"
        },
        {
            "id": "b98e48f6-875b-4039-975d-49eeb013fae3",
            "active": true,
            "version": 0,
            "restricted": false,
            "createdDate": null,
            "createdBy": null,
            "lastModifiedDate": null,
            "lastModifiedBy": null,
            "author": null,
            "link": "https://zulkiflialbakri.com/makhraj-huruf-surah-al-fatihah/",
            "title": "makhraj huruf surah al fatihah"
        }
    ],
    "pageable": {
        "pageNumber": 0,
        "pageSize": 10,
        "sort": {
            "empty": true,
            "sorted": false,
            "unsorted": true
        },
        "offset": 0,
        "paged": true,
        "unpaged": false
    },
    "totalPages": 1,
    "totalElements": 2,
    "last": true,
    "size": 10,
    "number": 0,
    "sort": {
        "empty": true,
        "sorted": false,
        "unsorted": true
    },
    "numberOfElements": 2,
    "first": true,
    "empty": false
}

Author

👤 Amir.Bahar

• Twitter: @https://twitter.com/amirbahar_
• Github: @ashraphs
• LinkedIn: @https://www.linkedin.com/in/amir-asyraf-83b940111/

Show your support

Give a ⭐️ if this project helped you!

---

This README was generated with ❤️ by readme-md-generator