This repository provides workers for getting job postings with aditional data.
Workers are divided into two groups based of main purpose of the job.
-
feeders — sidekiq based jobs for getting job postings from the Internet sources. For example job for getting job postings from Ministry of Labour and Social Affairs: MPSV.CZ database.
-
enrichment — sidekiq based job for getting additional data about individual job postings from external sources. For example getting geo coordinates for work location.
After new job posting is created by some feeder using API call, API creates Enrichment::All job for data enrichment of this particular job posting. Right now this job creates Enrichment::Geo job.
If there is some aditional data enrichment needed, you can create new job definition and add it into Enrichment::All same way as Geo is defined.
git clone https://github.com/OPLZZ/pickwick-workers.git
cd pickwick-workers
... install the required rubygems:
bundle install
You can find configuration of sidekiq queues and their limits in sidekiq_queues.yml file.
To specify periodic jobs, you can use sidekiq_scheduler.yml config file.
Several environment variables needs to be set for sidekiq web application
export SIDEKIQ_REDIS_URL='redis://127.0.0.1:6379'
export SIDEKIQ_USERNAME='sidekiq web user'
export SIDEKIQ_PASSWORD='sidekiq web password'
... after that you can run web server providing sidekiq web interface
bundle exec puma
... you can access sidekiq web interface at http://localhost:9292
To run sidekiq worker process you need to specify following environment variables
export SIDEKIQ_REDIS_URL='redis://127.0.0.1:6379'
export PICKWICK_API_URL='http://api.damepraci.cz' # url of [API repository](https://github.com/OPLZZ/pickwick-api)
export PICKWICK_API_TOKEN='API_TOKEN'
... after that you can run worker process by
bundle exec sidekiq --config config/sidekiq_queues.yml
##Funding
