mundo-hoje WEB scraping API

To access Swagger API documentation CLICK HERE

---

Technologies used in development

• Spring boot
• Spring Security
• Lombok
• Jsoup

---

Motivation

• This project was created to solve the need for the mobile application mundo-hoje that consumes the IBGE news API, the mundo-hoje project can be accessed by CLICKING HERE
• The open IBGE news API does not return the complete news, but it does return the link to the complete news.
• I was forced to develop an application that converted the news link into a json with the information that was relevant to my needs.
• CLICK HERE to access the official and complete documentation of the IBGE news API.

---

Operation

• This API has only 1 POST type endpoint (baseUrl/api/news/scraping-article)
• This single endpoint receives in its body an item according to the IBGE news API.
• Example of an item that must be passed in the request body:

{
     "id": 35875,
     "tipo": "Notícia",
     "titulo": "Mais de 70% das empresas industriais com 100 ou mais pessoas ocupadas inovaram em 2021",
     "introducao": "Setor químico concentra a maior proporção de indústrias inovadoras - Foto: Freepik Em 2021, a taxa de inovação no Brasil foi de 70,5%, percentual relativo às empresas industriais com 100 ou mais pessoas ocupadas que lançaram um produto ou implementaram...",
     "data_publicacao": "15/12/2022 10:00:00",
     "produto_id": 35867,
     "produtos": "35867|Pesquisa de Inovação Semestral|pesquisa-de-inovacao-semestral|3065",
     "editorias": "economicas",
     "imagens": "{\"image_intro\":\"images\\/agenciadenoticias\\/estatisticas_economicas\\/2022_12\\/pintec_THUMB_freepik.jpg\",\"float_intro\":\"\",\"image_intro_alt\":\"\",\"image_intro_caption\":\"\",\"image_fulltext\":\"images\\/agenciadenoticias\\/estatisticas_economicas\\/2022_12\\/pintec_HOME_freepik.jpg\",\"float_fulltext\":\"\",\"image_fulltext_alt\":\"\",\"image_fulltext_caption\":\"\"}",
     "produtos_relacionados": "35867",
     "destaque": true,
     "link": "http://agenciadenoticias.ibge.gov.br/agencia-noticias/2012-agencia-de-noticias/noticias/35875-mais-de-70-das-empresas-industriais-com-100-ou-mais-pessoas-ocupadas-inovaram-em-2021.html"
   }

• The API will then add a new attribute to that object, an object I called article.
• Structure of an article-type object:

export class Article{
title: string;
subtitle: string;
metadata: string;
text: string;
}

---

Questions or suggestions?

Feel free to open a new issue.

---

Thanks for visiting this repository!:sparkling_heart:

If you liked it, please leave a star.:star2: