Skip to content

Deploy a test DB, API and crawler

Édith Darin edited this page Apr 1, 2024 · 1 revision

Context

The social media audience database is currently hosted on an aws server with the corresponding API that checks data entries. This is the API that crawlers ping to write into the database.

In practice the database, API, API limiter and web server are Docker containers gathered under the name datahub. The API checks and refactors the input csv to be integrated to the database. The web server does the linkage between API request and API.

Each datahub component is defined in the docker-compose.yml file stored in mySocialWatcher/docker/datahub.

Deploying the datahub

To set up the datahub locally, you need to create an environment file in .../mySocialWatcher/docker/datahub/.env with the following content

PGDATA=/datahub/data
POSTGRES_HOST=datahub_db
POSTGRES_PORT=5432
POSTGRES_DB=social_media_audience
POSTGRES_USER=postgres
POSTGRES_PASSWORD= pwd
POSTGRES_RPASS= pwd
POSTGRES_WPASS= pwd
LIMITER_HOST=datahub_limiter

Then, execute the docker-compose to build containers for the database, api, api limiter, and web server. .../mySocialWatcher/docker/datahub/ docker-compose up -d --build

You can then access the API at http://127.0.0.1/api/v1, and you can connect to the local database using hostname=127.0.0.1 with the login credentials from the .env file (above).

Deploying the crawler

The crawler environment file contains the URL of the datahub and an API access token that authorizes write-access to it. An example of this file is in ./config/_example_private/crawler_env. You need to create a copy of this file in ./config/private/crawler.env and fill it in with the following informarion

DATA_DIR=./data
API_URL="http://127.0.0.1/api/v1/social_media_audience/write"
API_TOKEN=apitoken

An API_TOKEN is automatically created when deploying the test datahub. To find it, connect with DBeaver to the datahub with the following credentials (which are the one defined above in the .env file of the datahub):

Host: 127.0.0.1
Database: social_media_audience
Username: postgres
Password: pwd
Port: 5432

Inside the social_media_audience db there is one table called token.

Clone this wiki locally