Jobs Watcher

Queries the Indeed API and puts the data in an elastic instance that is assumed to be running on your localhost. The description below has information on the details of making this happen.

Requirements

1. git
2. Java 8 JDK
3. Maven
4. Docker
5. Mongo

CI Configuration / GitLab Pipelines

Dependencies Setup

See below for additional setup options (Docker)

1. docker-compose --file docker/docker-compose-dependencies.yml up creates Elasticsearch and Kibana linked Docker containers
2. . bin/index-create.sh this will setup an elastic index called jobs.
3. . bin/mongo-config-create.sh will create the initial job scraping configuration

Please note that this will expose the following ports on your host:

1. 9200 - Default port for the Elastic API
2. 9300 - Communication channel between Elastic and Kibana
3. 5601 - Default port for the Kibana service
4. 27017 - Mongo

Seed the data

You must first create an index by running the following script: ``

So there are a couple different ways to seed the data. You can import this project into intellij or Eclipse and run the main line (com.labs1904.jobs.watch.App). Right now the api key is hard coded along with the indeed api, it also assumes the elastic endpoint is configured to hit the ports suggested above. There are plans to provide all of this via configuration or service discovery.

Running via Eclipse

1. Install eclipse.
2. right click in package explorer -> import.
3. Existing maven projects
4. Browse to <jobswatcher>
5. right click on pom.xml -> run as : mvn clean
6. right click on pom.xml -> run as : mvn install
7. right click on App.java -> run as : java Application

Docker

Run the Entire Ecosystem

Still under development, jobswatcher does not wait for healthy elk, or an index to have been created

1. mvn clean package
2. cd docker
3. docker-compose build
4. docker-compose up

Run the Dependencies Only (Elasticsearch and Kibana)

See above Elasticsearch & Kibana setup

Run the Jobswatcher App Only

1. mvn clean package
2. docker-compose --file docker/docker-compose-jobswatcher.yml build
3. docker-compose --file docker/docker-compose-jobswatcher.yml up
4. docker-compose --file docker/docker-compose.yml build
5. docker-compose --file docker/docker-compose.yml up

Viewing the data within Kibana

When you ran the docker command above that should have populated a kibana index called jobs. If you have elastic and kibana running you can travel to the kibana homepage. There you will be asked to provide a default index. Put in jobs and you should be able to browse the data.