SentifyFlow is Automation of Scraping & Sentiment Analysis. SentifyFlow is an end-to-end data pipeline for performing the tasks below.
- Crawling Twitter (X) data
- Sentiment Analysis using fine-tuned BERT
- Sentiment Weighting & Scoring
- Data persistence to PostgreSQL
- Data lake storage in MinIO
- Alerting to Telegram & Email
- Fully orchestrated using Apache Airflow
- Fully containerized using Docker
This project is designed as a production-ready sentiment pipeline that is modular, scalable, and maintainable.
| Component | Technology |
|---|---|
| Orchestration | Apache Airflow (Dockerized) |
| Containerization | Docker |
| Object Storage | MinIO |
| Database | PostgreSQL |
| Sentiment Model | Fine-Tuned BERT |
| Alerting | Telegram Bot + Email |
| Configuration | YAML |
| Language | Python |
┌──────────────────┐
│ Apache Airflow │
│ (Dockerized) │
└─────────┬────────┘
│
┌─────────────────────┴───────────────────┐
│ │
DAG Crawler DAG Analyzer
│ │
▼ ▼
Docker: Crawler Docker: Analyzer
│ │
▼ ▼
MinIO ─ ─ ─ ─ ─► PostgreSQL ◄─ ─ ─ ─ ─ ─ MinIO
│ │
└────────► Telegram & Email Alert ◄───────┘
SentifyFlow/
│ .env
│ .env.example
│ .gitignore
│ docker-compose.yaml
│ Dockerfile
│ README.md
│ requirements.txt
│
├───airflow
│ ├───config
│ │ airflow.cfg
│ │ catalog.yml
│ │ config.yml
│ │ credential.json.example
│ │
│ ├───dags
│ │ │ dag_analyzer.py
│ │ │ dag_crawler.py
│ │
│ ├───logs
│ │ │ airflow.log
│ │ │ sentifyflow.log
│ │
│ └───script
│ │ analyzer.py
│ │ crawler.py
│ │ helpers.py
│
├───analyzer
│ │ docker-compose.yaml
│ │ Dockerfile
│ │ requirements.txt
│ │
│ ├───config
│ │ catalog.yml
│ │
│ ├───model
│ │ └───sentiment-classifier-fine-tuned
│ │ config.json
│ │ pytorch_model.bin
│ │ special_tokens_map.json
│ │ tokenizer_config.json
│ │ vocab.txt
│ │
│ └───script
│ helpers.py
│ model.py
│ predict.py
│
└───crawler
│ .env
│ docker-compose.yaml
│ Dockerfile
│ requirements.txt
│
├───config
│ catalog.yml
│
└───script
crawling_data.py
find_username.py
get_token.py
helpers.py
SentifyFlow consists of two main DAGs:
| DAG | Purpose |
|---|---|
dag_crawler |
Retrieve Twitter data and save it |
dag_analyzer |
Performing sentiment analysis & alerting |
NB: The DAG Analyzer runs after the DAG Crawler finishes.
DAG Crawler is used to retrieve Twitter data, enhance the data (username) from results of crawling, and store it in a bucket storage & database. Below is the task flow of this DAG.
check_credentials
↓
crawling_tweets (Docker)
↓
find_username (Docker)
↓
insert_tweets
↓
export_raw_tweets
DAG Analyzer is used to perform sentiment analysis using fine-tuned BERT, perform weighting, create a summary of the analysis results, and send alert via Telegram & Email. Below is the task flow of this DAG.
check_credentials
↓
sentiment_tweets (Docker)
↓
listing_file
↓
weighting_sentiment
↓
insert_sentiment
↓
send_alert
↓
housekeeping_file
SentifyFlow uses two storage methods for the results of each process for each pipeline. SentifyFlow uses MinIO as object storage and PostgreSQL as a database.
The following is the structure of the bucket storage that is used.
sentifyflow-bucket/
│
├───auth
│
├───labeled
|
└───raw
| Folder | Purpose |
|---|---|
auth |
Stores JSON files containing credentials |
labeled |
Stores CSV files containing labeled data |
raw |
Stores CSV files containing raw data |
The following is the structure schema of the database that is used.
| Table | Purpose |
|---|---|
tweets_raw |
Stores data containing raw data |
tweets_sentiment |
Stores data containing labeled data |
All components in this SentifyFlow project is running on Docker.
| Service | Container |
|---|---|
| Airflow | Docker |
| Crawler | Docker Image |
| Analyzer | Docker Image |
| MinIO | Docker |
| PostgreSQL | Docker |
DockerOperator is used to run crawling and sentiment analysis processes with the following objectives.
- Isolated environment
- Secure dependencies
- Scalable
Here is an example of the alert output of the analysis results that will be sent via Telegram and email.
📊 SentifyFlow Market Sentiment Alert (2026-02-20)
Period: 2026-02-13 — 2026-02-19
💡 Market Signal:
Index: 100.00/100 → 🟢 Bullish
Weighted Score: +1.0000
Net Score: +1.0000
📌 Sentiment Summary:
Positive: 59 (100.00%)
Neutral: 0 (0%)
Negative: 0 (0%)
⚖️ Pressure:
Bullish Pressure: 100.00%
Bearish Pressure: 0.00%
🎯 Model Confidence:
Avg Probability: 39.56%
🚀 Top Market Movers (highest weight):
1. @BudiBukanIntel | positive | weight: 6.3742
https://x.com/undefined/status/2024105000167723113
2. @TradingDiary2 | positive | weight: 5.6419
https://x.com/undefined/status/2023075414802137324
3. @TradingDiary2 | positive | weight: 2.3514
https://x.com/undefined/status/2024127800916021428
🔗 Link Task:
http://localhost:8080/dags/DEVdaganalyzer/grid?dagrunid=scheduled2026-02-20T07:02:00+00:00&taskid=sendalert
- Fully automated
- End-to-end pipeline
- Fine-tuned BERT model
- Engagement-based sentiment weighting
- Dockerized & modular
- Production-style orchestration
- Alerting system
- Database consistency control
- Storage housekeeping
- Dashboard Visualization (Streamlit / Superset)
- Real-time streaming (Kafka)
- Model retraining DAG
- MLflow integration
- Data quality validation (Great Expectations)
- API layer untuk expose sentiment index
SentifyFlow is an end-to-end data pipeline implementation that combines data ingestion from Twitter (X), sentiment analysis using fine-tuned BERT, and automatic alerting in an architecture orchestrated with Apache Airflow and containerized using Docker. With a clean split between DAG Crawler and DAG Analyzer, the system is designed to be modular, scalable, and production-ready. The integration of MinIO as object storage and PostgreSQL as a persistent database ensures data consistency and reliability at every stage of the pipeline. Overall, SentifyFlow represents a real-world application of Data Engineering and MLOps practices in building a structured, automated sentiment analytics system that is ready for further development to meet data-driven monitoring needs.
Project Impact:
-
End-to-End ML Pipeline: From data crawling, preprocessing, BERT inference, weighting, to alerting, everything runs automatically.
-
Data Engineering Pipeline: Implementation of ingestion, storage, transformation, and persistence in a structured architecture.
-
Production-Oriented Architecture: Orchestration using Airflow with containerized execution (DockerOperator) for environment isolation and scalability.
-
MLOps Implementation: Integration of model inference in scheduled workflows with object storage, database persistence, and monitoring alerts.
-
Alerting System: Automatically deliver sentiment summaries and weighting to Telegram and email as decision-support signals.
-
Real-World Use Case: It can be used for market sentiment monitoring, brand tracking, and financial signal intelligence.
The project is licensed under the MIT license - see the LICENSE file for more details.