I build AI-powered data pipelines that make LLMs actually useful in production. Over the past 4+ years, I've shipped systems for Pfizer, Covetrus, and NJIT — from event-driven streaming architectures to GPU-clustered LLM inference engines.

• 🎓 M.S. Computer Science — New Jersey Institute of Technology
• 🤖 Current role: AI Data Engineer — I don't just move data, I make it think
• ⚡ Superpower: Turning research papers into running code (see my "From Scratch" projects below)

5 LLM inference pipelines. 2 fine-tuning loops. 2 eval gates. One very busy GPU cluster.

I architected and shipped a production LLM stack for a healthcare data platform that uses Gemma-4 models (4B → 31B parameters) to classify, normalize, and QA-score patient records at scale:

• 🏷️ Taxonomy Classification — FAISS-GPU + EmbeddingGemma vector search → Gemma-4-E4B-it breed/species classification with 3-stage QA scoring
• ⚧ Gender Normalization — LLM-driven standardization of raw gender descriptions into 7 clinical categories with plausibility scoring
• 📝 Transcription QA — Gemma-4-31B-IT-QAT evaluating raw clinical transcripts for quality, plausibility, and completeness
• 📋 SOAP Note QA — Faithfulness scoring of AI-generated summaries against source transcripts (hallucination detection, omission tracking, medication error flags)
• 🔄 Continuous Fine-Tuning — Closed-loop eval gates: train → holdout eval → alias promotion only if accuracy beats production by margin

Stack: vLLM · Ray on Spark · FAISS-GPU · dbt · MLflow · L40S GPUs · Unity Catalog

"Don't just use the tool. Understand the tool. Then build the tool."

I got tired of just using Kafka and Spark, so I built them from the ground up in Python — straight from the original research papers. No libraries. No shortcuts. Just pure computer science.

Built from: "Kafka: a Distributed Messaging System for Log Processing" (Kreps et al., LinkedIn, 2011)

A working TCP broker with:

• ✅ Append-only log storage with .log + sparse .index files
• ✅ Fixed-header binary format ([4B Length | 8B Offset | 8B Timestamp | Payload])
• ✅ Consumer-driven offsets — the broker is completely stateless
• ✅ O(log n) binary search on the sparse index for fast seeks
• ✅ PRODUCE / FETCH wire protocol over TCP
• ✅ ~500 lines of Python, zero dependencies

Because reading the paper is cool. Making the paper actually run is cooler.

Built from: "Resilient Distributed Datasets" (Zaharia et al., UC Berkeley, 2012)

A mini Spark engine featuring:

• ✅ RDD lineage graph — fault tolerance without replication
• ✅ Lazy evaluation — transformations build a DAG, actions trigger execution
• ✅ Narrow dependencies — pipelined with Python generators (no intermediate storage!)
• ✅ Wide dependencies — shuffle stages with HashPartitioner
• ✅ DAG Scheduler — automatically splits jobs at shuffle boundaries
• ✅ Thread-pool executor with retry logic (lineage-based recovery)
• ✅ ~600 lines of Python, zero dependencies

I now understand why groupByKey is a shuffle boundary on a spiritual level.

LLMs     ████████████████████  Production inference, fine-tuning, eval gates
Python   ████████████████████  Data pipelines & "from scratch" engines
Kafka    ████████████████████  Event streaming & log-centric architectures
Spark    ████████████████████  Distributed computing & DAG optimization
SQL      ████████████████████  The OG data language
NoSQL    ████████████████      When relationships get complicated
AWS      ████████████████      Cloud infrastructure
Azure    ████████████████      More cloud infrastructure
Tableau  ████████████████      Making data pretty
CI/CD    ████████████████      Shipping things that don't break
Java     ████████████████      Building robust backends
ML       ████████████          Teaching machines to be slightly less dumb

• AI/LLM Engineering — inference optimization, structured output, hallucination detection
• Data Engineering — streaming, ELT, dbt, modern data stacks
• Distributed Systems — one paper implementation at a time

• 📧 Email: shbhamdbey@gmail.com
• 💼 LinkedIn: dubshubham
• 🐦 Twitter: @dbyshubh
• 🏆 HackerRank: sidshubham

A pizza that has radius "z" and height "a" has volume π × z × z × a.

Yes, I will bring this up in every technical interview. No, I will not apologize. 🍕

Built with caffeine, curiosity, and a stubborn refusal to accept "it just works" as an answer.