This project is a python service for data ingestion inside a Postgres
database and inside an Elastic
.
- Postgres : It takes all
csv files
fromdata/postgres
and insert data inside database, - Elastic : It takes all documents from
data/elastic
and insert them inside elastic.
-
Setup environment :
mv .env.example .env
-
Fill
.env
with your values -
Launch databases and Python service :
make start
-
Put your csv file(s) inside
data/postgres
folder and then run :python utils/ingestPg.py
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
---------- Start Data Ingestion ----------
We found 3 csv files to process
-------------------------
2 lines from .../data-ingestor-service/data/test.csv inserted into test
.../data-ingestor-service/fastapi_service/postgresIngestor.py:57: UserWarning: We can't add data from .../data-ingestor-service/data/test2.csv, columns doesn't match tables from public schema...
warnings.warn(f"We can't add data from {path}, columns doesn't match tables from {SCHEMA} schema...")
2 lines from .../data-ingestor-service/data/test_copy.csv inserted into test
-------------------------
Total : 4 inserted lines
---------- End Data Ingestion ----------
INFO: 127.0.0.1:64419 - "POST /ingest HTTP/1.1" 200 OK
-
Put your documents inside
data/elastic
folder and then run :python utils/ingestElastic.py
data-ingestor-service-backend-1 | ---------- Start Data Ingestion ----------
data-ingestor-service-backend-1 |
data-ingestor-service-backend-1 | We found 2 documents to process
data-ingestor-service-backend-1 |
data-ingestor-service-backend-1 | -------------------------
data-ingestor-service-backend-1 |
data-ingestor-service-backend-1 |
data-ingestor-service-backend-1 |
data-ingestor-service-backend-1 |
data-ingestor-service-backend-1 | -------------------------
data-ingestor-service-backend-1 |
data-ingestor-service-backend-1 |
data-ingestor-service-backend-1 | Total : 2 documents inserted
data-ingestor-service-backend-1 |
data-ingestor-service-backend-1 | ---------- End Data Ingestion ----------
data-ingestor-service-backend-1 | INFO: 192.168.0.1:61372 - "GET /ingestElastic HTTP/1.1" 200 OK