Repository navigation
Data Engineering
Data ingestion, transformation, storage, streaming, quality, and repeatable infrastructure
General data engineering foundations with practical applications to cybersecurity and operational telemetry
Part of the ULTIMATE CYBERSECURITY MASTER GUIDE
Index reviewed: 2026-11-11 for directory and navigation presence; individual lab verification dates remain in each guide. The learning path contains 13 guides organized in Phase1/, Phase2/, and Phase3/.
All three phase indexes and their linked guides are present in the current repository. Their local verification records do not establish live-service or production deployment testing. This index does not assert a tested
production deployment or new execution of the existing Phase 1 labs.
Data-store documentation: Curated external technical docs includes PostgreSQL and Redis for reliable query/telemetry references. For containerized lab services, use the Docker operational cheat sheet.
Suggested data-source practice: Security API Catalog & Cookbook provides authorized read-only security feeds and example JSON lookups for ingestion, contract validation, provenance and SIEM enrichment labs.
Security telemetry contracts: Sigma/Elastic/Wazuh detection content validation provides expected event-field and timestamp checks; rtl_433 passive lab provides an illustrative synthetic radio telemetry schema. Ingest each event source separately with provenance.
Provide a central learning and reference section for building systems that collect, move, transform, store, and deliver useful data. The scope includes application records, API responses, database extracts, operational metrics, and security logs.
Organize data engineering knowledge across the full data lifecycle, link to relevant guides already in the repository, and connect a three-phase learning path. Security telemetry provides the first practical pipeline example, while the broader learning path extends to databases, analytical storage, data quality, and workflow orchestration.
Help practitioners build understandable, reliable, secure, and maintainable data pipelinesโand explain where data came from, how it changed, whether it is complete, and how to recover when processing fails.
- Learning data engineering fundamentals alongside cybersecurity and infrastructure work.
- Moving data from files, APIs, applications, or databases into a central destination.
- Cleaning inconsistent records and designing a reusable event or dataset schema.
- Building batch jobs, event streams, or log aggregation pipelines.
- Replacing manual deployment and processing steps with repeatable workflows.
- Investigating missing records, stale data, duplicates, failed jobs, or unexpected storage growth.
- Overview & Scope
- Start Here: Choose Your Entry Point
- Phase 1: Foundations
- Secure Data Pipelines
- Existing Repository Resources
- Core Data Engineering Categories
- Phase 2: Storage & Integration
- Phase 3: Operations & Governance
- Learning Path & Folder Organization
- Learning & Implementation Workflow
- Data Handling & Operational Practices
- Contributing
- Quick Links
- Section Status
Data engineering makes data usable by other systems and people. A pipeline may be a small scheduled file import or a distributed stream processing system; the appropriate design depends on the data, latency requirements, failure tolerance, and operating budget.
This section treats cybersecurity as one application of data engineering. The same foundations also support inventory reporting, application analytics, service monitoring, business reporting, and internal automation.
| Area | Main Question |
|---|---|
| Data engineering | How do we deliver trustworthy data to its destination reliably? |
| Analytics | What does the prepared data tell us about an activity or outcome? |
| Detection engineering | Which patterns in telemetry should trigger a security investigation? |
| Infrastructure engineering | How do we deploy, secure, and operate the systems doing the work? |
These areas overlap. Use this section for the data pipeline foundations and the Incident Response section for security investigations and response procedures.
Choose an entry point based on the work you need to do.
| If you want toโฆ | Start with | Why |
|---|---|---|
| Learn data engineering from the beginning | Phase 1 foundations | Lifecycle, Python, SQL, ETL/ELT, and quality โ each with a runnable lab requiring no server or third-party package |
| Add storage and integrations | Phase 2 | Storage formats, ingestion recovery, orchestration, and change streams |
| Operate and govern an existing pipeline | Phase 3 | Reliability, testing, access, lineage, backup and replay |
| Build a secure security-telemetry pipeline now | Secure Data Pipelines | TLS, SSH, Git, parsing, caching, Kafka, and Ansible in one applied project |
Recommended combined path: Complete Phase 1 foundations, extend them through Phase 2 storage and integration,
then use Phase 3 to plan operation and governance. Apply the relevant concepts to data_pipelines.md as you go.
The applied guide remains a separate security-telemetry project.
Five published guides in Phase1/. Each is self-contained, states its prerequisites, and ends with a Verification Record listing the checks actually performed and what remains unverified.
| # | Guide | Coverage | Lab |
|---|---|---|---|
| 1 | ๐งฑ Data Engineering Fundamentals | Lifecycle, terminology, requirements template, batch/streaming decisions, source-to-destination design | CSV โ SQLite loader with validation, quarantine, idempotent upserts, and a run log |
| 2 | ๐ Python for Data Processing | Bounded-memory iteration, encoding, CSV and JSON Lines, timestamps, logging, packaging, tests | Installable assetpipe package with a unittest suite and a console entry point |
| 3 | ๐๏ธ SQL & Data Modeling | Keys, constraints, NULL semantics, joins, aggregates, window functions, transactions, indexes, query plans | Inventory and telemetry schema, then a star schema with an enforced grain |
| 4 | ๐ ETL & ELT Pipeline Design | ETL versus ELT, staging layers, watermarks, overlap windows, merge writes, checkpoints, backfills | Incremental pipeline demonstrating idempotent reruns, a backfill repair, and reconciliation |
| 5 | โ Data Quality & Schema Contracts | Quality dimensions, record and batch validation, nulls, duplicates, schema evolution, quarantine, producer/consumer obligations | Contract-driven validator and a compatibility gate that fails a build on breaking changes |
flowchart TD
A[1. Fundamentals] --> B[2. Python]
A --> C[3. SQL]
B --> D[4. ETL and ELT]
C --> D
D --> E[5. Data Quality]
Guides 2 and 3 can be read in either order. Guide 4 assumes both. Guide 5 assumes guide 4's load structure.
| Property | Detail |
|---|---|
| Dependencies | Core labs use Python standard library and SQLite; the Python guide also covers a local package install |
| Data | Synthetic fixtures throughout; nothing contacts a network service |
| Verification | Existing Phase 1 guides report their executed checks and limitations in their own Verification Records |
| Failure handling | Each guide covers what breaks, how it presents, and which check detects it |
Note
The labs use SQLite to avoid requiring a database server; confirm that your Python installation includes sqlite3. Each guide flags where the patterns differ on PostgreSQL, MySQL, or SQL Server, and the Verification Record states which engine was actually tested.
The applied security guide in this section connects six practical capabilities in one document:
| Capability | Covered Topics | Jump to Guide Section |
|---|---|---|
| ๐ Protect data in transit | TLS certificates, hostname verification, SSH identities, and administrative access | TLS ยท SSH |
| ๐๏ธ Version code and configuration | Git branches, staged review, commits, and rollback concepts | Git workflow |
| ๐งน Transform messy logs | Python parsing, JSON Lines, timestamps, validation, and quarantine | Structured events |
| ๐ง Enrich records with context | Redis and Memcached lookup caches, expiry, misses, and outage handling | Cached threat intelligence |
| ๐ก Stream and centralize events | Kafka topics, partitions, consumer groups, replay, and delivery considerations | Kafka |
| โ๏ธ Automate deployment | Ansible inventory, templates, validation, and idempotence | Ansible |
Suggested first project: Follow the guide's synthetic SSH-log exercise, trace each accepted event through parsing and enrichment, and then work through the Kafka and deployment milestones.
Foundations behind this guide: its parsing and quarantine approach is explained generally in Data Quality & Schema Contracts, its event-identity and replay handling in Data Engineering Fundamentals, and its Python patterns in Python for Data Processing.
Note
The guide distinguishes runnable local exercises, configuration templates, and further integration work. Read its verification record for the checks performed and the deployment steps that remain unverified.
These documents already exist elsewhere in the repository. Link to them for their established subject areas rather than duplicating their content here.
| Existing Resource | Relevance to Data Engineering |
|---|---|
| Log Aggregation & Visibility | Log-source collection and forwarding into centralized security tooling. |
| Linux Auditd & Syslog | Linux audit sources, logging configuration, and forwarding considerations. |
| SIEM Deployment Index | Entry point to the repository's security information and event management platform guides. |
| ELK Stack Guide | Logstash processing, Elasticsearch indexing, and Kibana visualization in a security logging context. |
| Graylog Guide | Inputs, stream routing, processing pipelines, and log retention topics. |
| Existing Resource | Relevance to Data Engineering |
|---|---|
| Applied Cryptography | Transport protection, key management, and authentication foundations. |
| Container Image & Runtime Security | Image and runtime security considerations for containerized pipeline components. |
| Cybersecurity Homelab | Lab infrastructure and isolation planning before experimenting with services and data flows. |
Scope of this index review: Repository file paths were checked and existing resource descriptions retained. This does not certify every command, dependency version, or deployment claim inside those separate documents.
| Category | Learning Focus | Starting Point |
|---|---|---|
| ๐งฑ Foundations | Lifecycle, requirements, latency and failure boundaries | Fundamentals |
| ๐งน Transformation & modeling | Python, SQL, identity, constraints and reusable transforms | Python ยท SQL |
| โ Quality & contracts | Validity, completeness, compatibility and rejects | Quality & Contracts |
| ๐๏ธ Storage | Storage patterns, formats, partitions, publication and retention | Storage & File Formats |
| ๐ฅ Ingestion | APIs, files, pagination, retries and progress | API & File Ingestion |
| โ๏ธ Orchestration | Schedules, dependencies, concurrency and backfills | Workflow Orchestration |
| ๐ก Streaming | Change capture, event time, ordering and replay | Streaming & CDC |
| ๐ง Enrichment | Lookup freshness, expiry and authoritative data | Redis and Memcached |
| ๐ Reliability | Freshness, lag, error budgets, alerting and reconciliation | Observability & Reliability |
| ๐งช Change control | Tests, dependencies, review gates and staged releases | Testing & CI/CD |
| ๐ Governance | Ownership, lineage, access, secrets and lifecycle | Governance, Lineage & Access |
| โป๏ธ Recovery | Backup, restore, replay and recovery objectives | Backup, Replay & Disaster Recovery |
Useful distinction: Deployment automation configures services. Workflow orchestration coordinates data jobs. The Ansible material and the orchestration guide address different responsibilities.
Four completed guides belong in Phase2/. The supplied phase package contains the files
linked below; they are no longer proposed document outlines.
| # | Guide | Coverage | Lab |
|---|---|---|---|
| 1 | Data Storage & File Formats | Relational, object and analytical storage; CSV, JSON Lines, Parquet; partitioning, compression, retention | Round trips, partitioned files, SQL checks, and corruption detection |
| 2 | API & File Ingestion | Authentication, pagination, rate limits, retries, watermarks, changed files and incomplete downloads | Simulated pages, transactional restart, replay and partial files |
| 3 | Workflow Orchestration | Schedules, dependencies, retries, concurrency, parameterized jobs and backfills | Local dependency runner, quality gate and overlap refusal |
| 4 | Streaming & Change Data Capture | Snapshots, ordering, duplicates, event time, late records, replay and consumer compatibility | One-partition CDC simulation with transactional state and progress |
Suggested order: Storage โ ingestion โ orchestration โ streaming. Core labs are local and synthetic; the Parquet extension requires PyArrow and is marked unexecuted. HTTP and CDC examples simulate transport behavior. The orchestration exercise is a teaching runner, not a deployed scheduling service.
Four completed guides belong in Phase3/, extending the earlier foundations into
day-to-day operation and recovery.
| # | Guide | Coverage | Lab |
|---|---|---|---|
| 1 | Pipeline Observability & Reliability | Freshness, throughput, lag, service levels, error budgets, alerting, reconciliation, runbooks, and recovery drills | Synthetic monitoring and budget calculation; stale/unknown detection; equal-count mismatch |
| 2 | Pipeline Testing & CI/CD | Unit and integration tests, synthetic fixtures, schema checks, dependency pinning, review gates, and staged releases | Eight unit/integration tests and a deliberate mutation; optional GitHub Actions example |
| 3 | Data Governance, Lineage & Access | Ownership, source lineage, sensitive fields, access boundaries, secrets, retention, deletion, and audit trails | Catalog, field-policy model, lineage impact, retention candidate report, and audit-chain checks |
| 4 | Backup, Replay & Disaster Recovery | Backups versus replication, restore exercises, replay boundaries, recovery objectives, deduplication, and rebuilding derived data | SQLite backup, isolated restore, replay, deletion protection, and derived rebuild |
Suggested order: Observability โ testing โ governance โ recovery. Four local labs were executed. Live monitoring, remote CI, real access policies, deletion, offsite recovery, and production recovery targets remain deployment-specific validation work, not claims made by these exercises.
| Phase | Folder | Guides | Outcome |
|---|---|---|---|
| 1 โ Foundations | Phase1/ |
5 | Understand and build correct local processing |
| 2 โ Storage & Integration | Phase2/ |
4 | Store, collect and coordinate data across boundaries |
| 3 โ Operations & Governance | Phase3/ |
4 | Observe, test, control and recover the pipeline |
Total: 13 phase guides, plus the separate Secure Data Pipelines & Security Automation applied guide.
Placement convention: All phase documents live in their corresponding Phase1/, Phase2/, or Phase3/
folder. This general README and data_pipelines.md remain directly under Data-Engineering/.
Inside a phase, use ./ for siblings, ../PhaseN/ for another phase, and ../../ for repository-level resources.
Keep the existing index capitalization: Phase 1 uses README.md; Phases 2 and 3 use readme.md.
Future additions should fill a demonstrated gap, include observable success and failure cases, and state exactly what was verified. The original three-phase document list is now covered.
| Step | Action | Deliverable | Related Guide |
|---|---|---|---|
| 1. Define the purpose | Identify the source, consumer, sensitivity, expected volume, and acceptable delay. | Short requirements and ownership record. | Requirements before code |
| 2. Build a small path | Move a synthetic sample from one source to one destination. | Reproducible working example. | File-to-database lab |
| 3. Establish the contract | Define field types, event identity, timestamps, and rejected-record handling. | Schema and accepted/rejected fixtures. | What a contract contains |
| 4. Protect access | Configure verified transport, least-privilege identities, and secret handling. | Documented access and trust configuration. | TLS and SSH |
| 5. Make writes repeatable | Decide how incremental loads, retries, and duplicate records behave. | Replay and reconciliation checks. | Idempotent writes and merge |
| 6. Automate execution | Version configuration, deploy consistently, and schedule jobs where needed. | Reviewed configuration and run procedure. | Git and Ansible |
| 7. Observe failures | Test unavailable sources, bad records, destination failures, and stale data. | Metrics, alerts, and troubleshooting notes. | Failure drills and failure modes |
| 8. Expand deliberately | Measure bottlenecks before introducing new services or partitions. | Capacity notes and tested recovery plan. | Indexes and query plans |
- Define freshness and reconciliation checks with Observability & Reliability.
- Add review and release evidence with Testing & CI/CD.
- Record owners, access boundaries and lineage with Governance.
- Rehearse an isolated restore and replay with Recovery.
- Inventory reporting: Import synthetic asset CSV files into a relational database, validate serial-number uniqueness, and produce a queryable current inventory. Implemented as the file-to-database lab, extended by the star schema lab.
- Incremental load with recovery: Run a watermark-based pipeline, prove the second run changes nothing, then simulate data loss and repair it with a backfill. Implemented as the incremental pipeline lab.
- Contract enforcement: Declare a contract, validate a defective batch against it, and gate a schema change on compatibility. Implemented as the contract validator lab.
- Security telemetry: Parse synthetic authentication logs, add cached context, and publish structured events using data_pipelines.md.
- API ingestion: Exercise pagination and checkpoints, then demonstrate recovery after an interrupted run. The API & File Ingestion lab exercises the recovery policy with simulated pages; live API behavior requires a separate adapter test.
- Use synthetic or sanitized samples in public documentation and tests.
- Keep credentials, private keys, customer records, and sensitive raw logs out of Git.
- Preserve source identity and processing history so results can be traced and reconciled.
- Separate an empty result from a failed lookup or incomplete ingestion.
- Define retention and access rules for raw, quarantined, transformed, and archived data.
- Test restore and replay procedures before relying on them during an outage.
- Keep documented test results specific to the environment and versions actually checked.
Caution
Quarantined records inherit the classification of their source. A reject file assembled from security logs or customer data carries the same handling obligations as the original, and the fact that a record failed validation does not make it non-sensitive. See quarantine and reject handling.
Contributions can extend the general foundations, add small reproducible labs, or improve links to existing repository material.
Submission Guidelines:
- Check this index and related sections before creating a duplicate guide.
- Follow the repository style guide and contribution guidance.
- State prerequisites, tested versions, expected results, and known limitations.
- Include sanitized examples and failure-handling exercises where practical.
- Add relative links here and in the appropriate phase index when a document is created.
- Include a verification record stating what was executed, the output observed, and what remains unverified.
- Prefer standard-library and single-file-database examples; clearly identify optional dependencies.
- When placing a guide in a phase subfolder, check that its relative links account for the extra directory level.
- Update the master index when adding a new guide or major section.
- ๐งฑ Data Engineering Fundamentals
- ๐ Python for Data Processing
- ๐๏ธ SQL & Data Modeling
- ๐ ETL & ELT Pipeline Design
- โ Data Quality & Schema Contracts
- Data Storage & File Formats
- API & File Ingestion
- Workflow Orchestration
- Streaming & Change Data Capture
- Pipeline Observability & Reliability
- Pipeline Testing & CI/CD
- Data Governance, Lineage & Access
- Backup, Replay & Disaster Recovery
- ๐ก๏ธ Secure Data Pipelines & Security Automation
- ๐จ Incident Response
- ๐ Log Aggregation
- ๐ Applied Cryptography
- ๐ณ Container Security
- ๐ Homelab
- ๐ Glossary
| Item | Status |
|---|---|
| Section directory | Data-Engineering/ |
| Applied security guide | data_pipelines.md |
| Phase 1 foundations | Published โ 5 guides in Phase1/
|
| Phase 2 storage & integration | 4 completed guides in Phase2/, supplied in its phase package |
| Phase 3 operations & governance | 4 completed guides in Phase3/, supplied with this index |
| Phase guide total | 13 guides across three folders |
| Existing supporting resources | Linked from their current repository locations |
| Core phase labs | Python standard library and SQLite where used; synthetic local fixtures |
| Optional integrations | Separate dependencies and live services; consult each Verification Record |
| Verification scope | Phase 2/3 local labs executed; no new Phase 1 or applied-service deployment verification |
| Index review | September 11, 2026 |
๐๏ธ Collect Carefully. Transform Clearly. Deliver Reliably.
Build useful data systems with traceable inputs, explicit failure handling, and repeatable operations.
Repository: ULTIMATE CYBERSECURITY MASTER GUIDE
Pacific Northwest Computers: PNWC on GitHub
๐ Master Index | ๐ฏ Role Navigation | ๐ Contents | ๐ Legal Notice
โญ Star the repository if you find it useful! โญ
- ๐งฑ Data Engineering Fundamentals
- โ Data Quality & Schema Contracts
- ๐ ETL & ELT Pipeline Design
- ๐งฑ Phase 1: Data Engineering Foundations
- ๐ Python for Data Processing
- ๐๏ธ SQL & Data Modeling
- ๐ฅ API & File Ingestion
- ๐๏ธ Data Storage & File Formats
- ๐๏ธ Phase 2: Storage & Integration
- ๐ Streaming & Change Data Capture
- โ๏ธ Workflow Orchestration
- โป๏ธ Backup, Replay & Disaster Recovery
- ๐ Data Governance, Lineage & Access
- ๐ ๏ธ Phase 3: Operations & Governance
- ๐ Pipeline Observability & Reliability
- ๐งช Pipeline Testing & CI/CD
๐ Folder: Data-Engineering | โ๏ธ Suggest an edit | ๐บ๏ธ Browse by Topic
- ๐บ๏ธ Browse by Topic
- ๐ All Repository Files
- ๐งญ Learning Paths
- ๐ Master Guide
- ๐ค Glossary
- โ๏ธ Legal & Disclaimer
๐ Foundations & Learning
๐ก๏ธ Defense & Investigations
- Incident Response
- Open Source Intelligence (OSINT)
- Reverse Engineering
- Operations Security (OPSEC)
- Compliance
โ๏ธ Platforms & Infrastructure
๐ก Hardware, Wireless & Field Projects
- PNWC Cyberdeck
- Hardware Testing
- Hardware Hacking
- Software-Defined Radio
- Wardriving
- uConsole
- Space Security
๐งฐ Assessment & Field Reference
๐ฆ Downloads & Project Resources
๐๏ธ Additional Topics