dgapScraper

This projects intends to gather financial news about companies together with its impact on their corresponding stock course.

Running and Building

To start the Docker Compose run the buildAllAndRun.ps1 File inside powershell. It automates all building and running of the docker commands.

via the flag -rebuild you can choose which docker container should be rebuild from the sources. If the -rebuild option is missing all containers will be new build.

If the -rebuild option is empty or none only docker-compose down and docker-compose up will be run sequentially.

Example Options

The Last command works with all mircoservices

Command	Action
buildAllAndRun.ps1	Everything will be rebuild
buildAllAndRun.ps1 -rebuild none	nothing will be rebuild
buildAllAndRun.ps1 -rebuild articleprocessor	only the article processor will be rebuild

Manual Steps if liked

To run the scraper, you need to have docker, docker-compose and maven installed. Further you need to build the following .jar artifacts via maven with:

mvn clean install

in the following folders:

1. /ArticleProcessor
2. /StockProcessor

You need to build all Node Modules for the SingleScraper via:

npm install

inside:

1. /singleScraper

To build the docker images from the root directory you need to run each command:

1. /ArticleProcessor: docker build -t articleprocessor .\articleprocessor
2. /StockProcessor: docker build -t articleprocessor .\articleprocessor
3. /database: docker build -t database .\database
4. /singleScraper: docker build -t singlescraper .\singleScraper

After that you are able to start the docker-compose via:

docker-compose up