Azure Data Engineer with 5+ years of hands-on experience designing and delivering scalable, production-grade data solutions on Microsoft Azure. Currently at Canada Life, building enterprise-grade batch and real-time data pipelines that power critical business reporting and analytics.

Passionate about Medallion Architecture, metadata-driven pipelines, and building cloud-first data platforms that are reliable, cost-efficient, and easy to maintain.

• Large-scale PySpark jobs on Azure Databricks for batch & streaming workloads
• • Apache Spark optimization: partitioning, caching, broadcast joins, query tuning

  • • Distributed computing for enterprise-scale analytics

    • 🏗️ Modern Data Architecture

    • • Medallion Architecture (Bronze → Silver → Gold) on ADLS Gen2
      • • Delta Lake with ACID transactions, time travel, schema evolution, and merge operations

        • • Metadata-driven pipeline frameworks for zero-maintenance scalability

          • 🔄 Real-Time Streaming

          • • Spark Structured Streaming for real-time data pipelines
            • • Event-driven architectures with cloud-native message brokers

              • • Low-latency data ingestion and processing

                • ⚙️ ETL/ELT & Pipeline Orchestration

                • • End-to-end data pipeline design and implementation with Azure Data Factory
                  • Metadata-driven architectures for scalable automation

                  • CI/CD pipelines and automated monitoring/alerting frameworks

                  • 📊 Data Warehousing & Analytics

                  • Azure Synapse Analytics for large-scale analytical workloads

                  • Power BI reporting and dashboard development

                  • 🔐 Data Security & Governance

                  • Azure Key Vault for secrets management — zero hardcoded credentials

                  • Azure RBAC, encryption, and access control implementation

                  • Data governance and quality frameworks

                  ---

                  🛠️ Tech Stack

                  ☁️ Cloud & Data Platforms

                  

                  💻 Big Data & Processing

                  

                  🧑‍💻 Languages & Frameworks

                  

                  🔧 DevOps & Tools

                  

                  ---

                  📁 Featured Projects

                  Project	Description	Tech
                  Azure Data Factory Projects	Metadata-driven pipeline orchestration and data movement solutions	ADF, Azure Storage, ETL
                  Azure Synapse Projects	Real-time data ingestion, transformation, and analytics pipelines	Synapse Analytics, SQL, Spark
                  Databricks DAB	Databricks asset bundles and advanced analytics workflows	Databricks, PySpark, Notebooks
                  Snowflake Projects	Cloud data warehouse optimization and SQL transformations	Snowflake, SQL, Snowpark
                  Microsoft Fabric Projects	Real-time analytics, lakehouses, Power BI, and dataflow pipelines	Fabric, Power BI, Dataflows
                  Cloud Migration Project	On-premises to cloud data migration architecture and implementation	Azure, ADLS, ADF

                  ---

                  🎓 Certifications

                  🏅 Credential	📅 Status
                  Microsoft DP-900: Azure Data Fundamentals	✅ Certified — Jan 2026

                  ---

                  📊 GitHub Stats

                  

                  

                  ---

                  🌐 Connect & Collaborate

                  📧 shivakumaryallanti5@gmail.com | 📞 +1 226-210-1440 | 📍 Toronto, Canada

                  

                  ---

                  "Architecting scalable data solutions that drive business intelligence."