Skip to content

RianTavares/node-crawler

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

48 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

AutoEsporteBot.js

What does it do

This bot is a part of a back-end challenge. His main function is catch information from Auto Esporte (https://revistaautoesporte.globo.com/rss/ultimas/feed.xml) that is a Brazilian TV program and a Brazilian magazine dedicated to the automotive sector and published for Globo. It works as a crawler and catch information of the website above and return in Json format exactly like was requested in the challenge

Features

  • [✓] Easy to use
  • [✓] Catches each item and their respective title, link, p, div//ul and div//img
  • [✓] Formats the response to Json
  • [✓] Test coverage (using JEST)
  • [✓] Organizable and readable code (using ESlint)
  • [✓] "Dockerized"
  • [✓] Implemented as a Webservice for test environments

Fast setup (CLI Version)

  1. Download or Clone the project and extract it
  2. Download Node.js and install it.
  3. Run npm install in node-crawler folder
  4. Start the bot via command line node ./src/app.js
  5. The project now is running on 3000 port. In the next topic you can see how to login and get a response from the API

How to login and run the bot

This project has a database simulated in a js file /src/models/dbSimulator.js, because the intention is to do a login validation for a test enviroment and it will just works if there is only single entries into dbSimulator.js

  1. Download Postman and install it
  2. With the API running just do a POST request with the URL http://localhost:3000/api/v1/login
  3. Remember to pass username:root and password:123 into the body of this request with the option x-www-for-urlencoded selected
  4. Copy the request response
  5. Open a GET request with the URL http://localhost:3000/api/v1/info and choose the tab "Authorization" with the type Bearer Token
  6. Then paste the request response previously copied in step 4 into "Token" input and send the request

Demo:

Run the project using container

Dockerfile:

FROM node:8
WORKDIR /docker/src/app/
COPY package.json /docker/src/app/
RUN npm install
COPY . /docker/src/app/
CMD node ./src/app.js
EXPOSE 3002
  1. First Install docker and let it running
  2. Into the project folder node-crawler run the command docker build -t node-crawler . to create the docker image
  3. Then run docker run -p 3002:3000 node-crawler
  4. run docker ps if you want to see the containers started
  5. Now with the API running into the container just do a POST request with the URL http://localhost:3002/api/v1/login
  6. Remember that as configured in Dockerfile the port now is 3002
  7. Don't forget to pass username:root and password:123 into the body of this request with the option x-www-for-urlencoded selected
  8. Copy the request response
  9. Open a GET request with the URL http://localhost:3000/api/v1/info and choose the tab "Authorization" with the type Bearer Token
  10. Then paste the request response previously copied in step 7 into "Token" input and send the request

Demo:

Last but not least

To run the tests

  1. As simple as possible, just run the command npm test into the project folder

To run ESlint

ESLint is a code linting that is frequently used to find problematic patterns or code that doesn't adhere to certain style guidelines.

Demo:

  1. Run the command $ ./node_modules/.bin/eslint yourfile.js
  2. Change yourfile.js for the file wanted
  3. exemple: $ ./node_modules/.bin/eslint ./src/app.js

:)

  • if you enjoyed about this project add a star ⭐
  • Don't forget to follow me on Instagram @riantba ❤️
  • And follow me also here on github :)

License

GNU GENERAL PUBLIC LICENSE

Copyright (c) 2018 Rian Tavares

About

🤖 A NodeJs bot to scrape information from a specific website

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors