Custom Jobs Aggregator

Provides functionality to fetch and merge job offers from multiple languages concurrently.

Each job from different languages is merged based on its unique identifier identifier_field, producing a master job object with translated fields merge_by_lang_on included for every language langs.

Aggregator setup

The working directory is assumed to be /root/app/. You may need to create the app/ folder.

1. Download the source code from GitHub

Download the source code from GitHub onto the host server

⚠️ This is a private repo. You will need an SSH key.

git clone git@github.com:joaooliveirapro/radagg.git

2. Implement cron jobs

The first cron job will run the aggregator script every hour, capture stdout and redirect it to cron.log. The container is removed after it finishes.

The second cron will run logrotate every day at 2 AM.

> crontab -e
...
# add these lines
# aggregator cron
0 */1 * * * docker-compose run --rm aggregator >> /root/app/radagg/cron.log 2>&1
# logrotate cron
0 2 * * * /usr/sbin/logrotate /etc/logrotate.d/aggregator

3. Copy logrotate cron

Copy the logrotate config available in cron/

> cp /root/app/radagg/cron/aggregator-logrotate.conf /etc/logrotate.d/aggregator

This ensures container logs are rotated daily, old logs are kept for 14 days, and compressed to save space.

4. Aggregator config file

The config.json file needs to be populated with the respective values.

{
    "access_token_url": "https://{company}.profils.org/api/token",
    "credentials": {
        "grant_type": "client_credentials",
        "client_id": "secret",
        "client_secret": "secret"
    },
    "offers_list_config": {
        "url": "https://{company}.profils.org/api/v2/offersummaries",
        "count": 500,
        "langs": [
            "en-US",
            "fr-FR",
            "nl-NL",
            "de-DE"
        ],
        "identifier_field": "reference",
        "merge_by_lang_on": [
            "organisationDescription"
        ]
    },
    "performance": {
        "max_concurrent_requests": 5,
        "max_workers": 5
    }
}

CEGID API documentation developers.cegid.com/api-details

• count maximum number of items per page. ATS suggests not going above 500
• langs lists all languages for which jobs are going to be requesed
• merge_by_lang_on lists all fields that will be merged by language

After each language request, the application will merge the fields like so:

{
    ... // all other job fields
    "field": "original field in lang1",
    "{field}_{lang1}": "value for field in lang1",
    "{field}_{lang2}": "value for field in lang2",
    "{field}_{lang3}": "value for field in lang3",
}

• max_concurrent_requests maximum number of concurrent HTTP requests (to avoid HTTP:429)
• max_workers number of threads making HTTP requests

5. Build Aggregator Docker image

After changing the config file, build the aggregator Docker image like so:

docker build -t aggregator .

There are mounted volumes for:

Host path	Container path	Purpose
radagg/output/	/app/output/	Stores generated XML files from aggregator
radagg/logs/	/app/logs/	Stores aggregator and Flask log files
radagg/config.json	/app/config.json	Aggregator configuration file

Exporter setup

When the aggregator finishes running, it generates an .xml file with all the jobs, located in output/. To export this file there's two options:

6. Flask app .env

The docker-compose.yml file contains a service called flask which can be run to start a server that exposes an endpoint to list the newest .xml file in /output.

The endpoint is: /latest/xml.

The .env which must be placed inside server/ must the following entry:

• AUTH_CODE string this can be any value but will act as the access token to protect the /latest/xml route

After changing the .env file, build the flask Docker image like so:

docker build -t flask_exporter ./server

To start this service run:

docker-compose up -d flask  # workdir /root/app/radagg

Testing flask app

When makig a get request, the AUTH_CODE value defined in .env should be sent in the Authorization header alone.

{ 'Authorization': 'AUTH_CODE' } // No 'Bearer ...'

Quick test using curl

curl -i http://{your_vps_ipv4}:8080/

# HTTP/1.1 200 OK
# Content-Type: text/html; charset=utf-8
# Content-Length: 28
# ...
# <h1>Flask app is running</h1>

6. SFTP

Alternatively you can upload the exported file to an SFTP/FTP

!NOT IMPLEMENTED!