$ pip install dehub-knowledge --upgrade
✓ Loaded: 500+ Resources | 50+ Tools | 10+ Roadmaps | 1 Community

The command center for data engineers worldwide. From zero to petabyte-scale — everything you need to become an elite data engineer.

🚀 Get Started • 📚 Resources • 🗺️ Roadmap • 🛠️ Tools • 🌐 Website • 🤝 Contribute

╔══════════════════════════════════════════════════════════════╗
║  dehub@engineer:~$                                           ║
╠══════════════════════════════════════════════════════════════╣
║                                                              ║
║  $ dbt run --select +orders_mart                             ║
║  [INFO] Running with dbt=1.8.0                               ║
║  [INFO] Found 47 models, 312 tests                          ║
║  ✓ Completed successfully                                    ║
║                                                              ║
║  $ spark-submit --master yarn pipeline.py                   ║
║  [INFO] SparkContext initialized                             ║
║  [INFO] Processing 2.4B records...                          ║
║  ✓ Job completed: 847s elapsed                              ║
║                                                              ║
║  $ kafka-topics --create --topic events --partitions 12     ║
║  ✓ Created topic "events"                                   ║
║                                                              ║
║  $ SELECT COUNT(*) FROM iceberg.prod.events;                ║
║  > 1,427,893,004 rows                                       ║
║                                                              ║
╚══════════════════════════════════════════════════════════════╝

New to Data Engineering? Follow this path:

📍 You are here
    │
    ▼
[1] 📖 Read "Fundamentals of Data Engineering" (Joe Reis & Matt Housley)
    │
    ▼
[2] 🐍 Learn Python + SQL to professional level
    │
    ▼
[3] 🌊 Build your first pipeline (Batch → Streaming)
    │
    ▼
[4] ☁️  Get cloud certified (AWS/GCP/Azure)
    │
    ▼
[5] 🚀 Contribute to open-source, get hired

Experienced Engineer? Jump to:

• Advanced Tools & Frameworks
• System Design Resources
• Senior-level Interview Prep
• Open Source Projects

Tool	Stars	Description
Apache Airflow		Industry-standard workflow orchestrator
Dagster		Data-aware orchestration platform
Prefect		Modern workflow automation
Mage		Modern data pipeline tool
Kestra		Declarative orchestration
Hamilton		Function-based DAG framework

Tool	Stars	Description
Apache Iceberg		Open table format for huge datasets
Delta Lake		ACID transactions for big data
Apache Hudi		Incremental data processing
Apache Polaris		Open catalog for Apache Iceberg
DuckLake	-	SQL-native lakehouse

Tool	Stars	Description
Apache Spark		Unified analytics engine
Apache Flink		Stream & batch processing
DuckDB		In-process analytical DB
Trino		Distributed SQL query engine
Polars		Fast DataFrame library

Tool	Stars	Description
dbt		SQL-first transformation
SQLMesh		Next-gen dbt alternative
Coalesce	-	Cloud-native transformation

Tool	Stars	Description
Great Expectations		Data quality framework
Soda		Data quality platform
dbt tests	-	Built-in dbt testing
Metaplane	-	Data observability
DQOps		Automated data quality

Build these projects to demonstrate real-world data engineering skills:

1. End-to-End NYC Taxi Data Pipeline

   • Tools: Python, BigQuery, Looker Studio
   • Skills: ETL, cloud storage, BI visualization

2. Weather Data Pipeline

   • Tools: Airflow, PostgreSQL, dbt
   • Skills: Orchestration, scheduling, transformation

3. Extract YouTube Metadata

   • GitHub Project
   • Tools: AWS Lambda, S3, Free Tier
   • Skills: Serverless, cloud storage, API ingestion

4. Real Estate Data Platform

   • GitHub
   • Tools: S3, Spark, Delta Lake, Dagster, Superset
   • Skills: Lakehouse, orchestration, visualization

5. Azure End-to-End Analytics Platform

   • Tools: ADF, ADLS, Databricks, Synapse, Power BI
   • Skills: Azure ecosystem, medallion architecture

6. LLM Data Pipeline

   • Lecture
   • Tools: OpenAI API, vector databases, Airflow
   • Skills: AI integration, vector search

7. Real-Time Streaming Analytics

   • Tools: Kafka, Flink, Iceberg, Grafana
   • Skills: Event-driven architecture, stream processing

8. SQL Query Engine with LLMs

   • Tutorial
   • Tools: LangChain, LLMs, databases
   • Skills: AI-powered tooling

9. Multi-Cloud Lakehouse

   • Tools: Apache Iceberg, AWS + GCP, dbt, Airflow
   • Skills: Cloud-agnostic architecture

Join these communities to learn, network, and grow as a data engineer.

Technical Interview Topics:
├── SQL
│   ├── Window Functions (ROW_NUMBER, RANK, LAG, LEAD)
│   ├── CTEs and Recursive CTEs
│   ├── Query Optimization & EXPLAIN plans
│   └── Aggregations & Subqueries
├── Python
│   ├── PySpark DataFrames
│   ├── Pandas/Polars operations
│   ├── Generators & Iterators
│   └── OOP for data pipelines
├── System Design
│   ├── Design a data warehouse
│   ├── Design a real-time analytics system
│   ├── Design a CDC pipeline
│   └── Design a data lake
├── Data Modeling
│   ├── Star Schema vs Snowflake
│   ├── Slowly Changing Dimensions (SCD)
│   ├── Data Vault 2.0
│   └── Kimball vs Inmon
└── Infrastructure
    ├── Docker & Kubernetes
    ├── Cloud platforms (AWS/GCP/Azure)
    ├── CI/CD for data pipelines
    └── Monitoring & Alerting

• Data Engineering Interview Questions — Comprehensive list
• ByteByteGo System Design — System design prep
• SQLZoo — SQL practice
• LeetCode Database Problems — SQL challenges
• Glassdoor DE Interview Reviews — Company-specific prep

• Quantify impact: "Reduced pipeline runtime by 67%" beats "improved pipeline"
• Include GitHub links to real projects
• Mention data volumes (TB, PB scale)
• List certifications (AWS, GCP, Databricks, dbt)

Source: levels.fyi, LinkedIn Salary, Glassdoor (2024)

Contributions are what make the open source community amazing! Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingResource)
3. Commit your Changes (git commit -m 'Add some AmazingResource')
4. Push to the Branch (git push origin feature/AmazingResource)
5. Open a Pull Request

See CONTRIBUTING.md for detailed guidelines.

Distributed under the MIT License. See LICENSE for more information.

Built with ❤️ for the Data Engineering Community

🌐 Website • ⭐ Star this repo • 🐛 Report Bug • 💡 Request Feature