Repository navigation
Python for Data Processing
Bounded memory, correct encoding, honest timestamps, and testable pipeline code
CSV โข JSON Lines โข Iterators โข Encoding โข datetime โข logging โข Packaging โข unittest
Part of the ULTIMATE CYBERSECURITY MASTER GUIDE
Last reviewed: 2026-09-11. Examples executed locally on the versions listed in the Verification Record.
Prerequisites: Data Engineering Fundamentals for lifecycle vocabulary, record identity, and quarantine concepts.
Teach the Python patterns that make data processing code correct under real conditions: files larger than memory, mixed encodings, ambiguous timestamps, partial failures, and code that must be changed six months later without breaking.
Cover streaming iteration, encoding handling, the sharp edges of the csv module, JSON versus JSON Lines, timezone-correct timestamp conversion, structured logging, project layout and packaging, and unit tests built on small fixtures โ then assemble them into an installable package with a passing test suite.
Enable a practitioner to write a processing script that handles a multi-gigabyte input in constant memory, fails loudly on ambiguous data, logs enough to diagnose a production incident, and is covered by tests that run in under a second.
- Writing or reviewing any Python script that reads data files and produces output.
- Diagnosing a job that works on a sample and fails on the full dataset.
- Investigating mojibake,
UnicodeDecodeError, or timestamps that are off by hours. - Converting a single ad-hoc script into a maintainable, tested module.
- Preparing the transformation layer described in ETL & ELT Pipeline Design.
Audience: Practitioners who can write basic Python and now need it to be reliable under production data conditions.
Prerequisites: Python 3.10 or later (3.11+ recommended for the fromisoformat improvements noted in ยง6). Familiarity with functions, dictionaries, and the terminal. No third-party packages are required; pytest is mentioned as an option but every runnable test here uses the standard-library unittest.
Note
All examples use synthetic data and run locally. The packaging section builds a local editable install only; nothing is published to an index.
- ๐ 1. Stream, Do Not Load
- ๐ค 2. Text, Bytes, and Encoding
- ๐ 3. CSV and Its Sharp Edges
- ๐งพ 4. JSON and JSON Lines
- ๐ 5. Generator Pipelines
- ๐ 6. Timestamps Done Correctly
- ๐ 7. Logging for Pipelines
- ๐ฆ 8. Project Layout and Packaging
- ๐งช 9. Testing with Fixtures
- ๐ 10. Lab: Build the Package
- ๐ 11. Self-Check
- โ Verification Record
- ๐ค Contributing
- ๐ Resources
- ๐ Quick Links & Related Guides
- ๐ Guide Details
The single most common cause of a pipeline that "worked in testing" is reading an entire file into memory.
# Unbounded โ memory grows with file size
rows = open("events.csv").readlines() # entire file in RAM
data = json.load(open("events.json")) # entire structure in RAM
results = [transform(r) for r in rows] # a second full copy
# Bounded โ memory stays roughly constant
with open("events.csv", encoding="utf-8") as fh:
for line in fh: # one line at a time
handle(transform(line))A file object is already an iterator over lines. A generator function lets each stage of processing consume the previous one lazily, so only one record is in flight at any moment regardless of whether the file holds a thousand rows or a hundred million.
| Pattern | Peak memory | Use when |
|---|---|---|
fh.read() / json.load()
|
Size of file | Small config files of known bounded size |
for line in fh |
One line | Line-delimited text of any size |
| Generator chain | One record | Multi-stage transformation of any size |
list(...) anywhere in the chain |
Entire result | Only when the result is deliberately small (lookups, aggregates) |
Warning
Sorting, grouping, and deduplication are inherently unbounded operations โ they must hold state. When the state is the problem, push it into a database (see SQL & Data Modeling) rather than a Python dictionary. A set of every seen identifier will exhaust memory on a large enough input; a UNIQUE index will not.
Every file on disk is bytes. Text only exists after a decoding decision. When that decision is left implicit, Python uses a platform default that differs between machines โ the classic cause of a script that works on one host and raises UnicodeDecodeError on another.
Always state the encoding explicitly.
# Implicit: behavior depends on the platform's locale
with open(path) as fh: ...
# Explicit: behavior is identical everywhere
with open(path, encoding="utf-8", newline="") as fh: ...Real exports from legacy systems contain bytes that are not valid UTF-8. Three strategies, in order of preference:
# 1. Correct: decode with the encoding the source actually uses
open(path, encoding="cp1252") # common for Windows-generated exports
# 2. Lossless round-trip: preserve undecodable bytes so they survive re-encoding
open(path, encoding="utf-8", errors="surrogateescape")
# 3. Last resort: substitute, and COUNT the substitutions
open(path, encoding="utf-8", errors="replace")Option 3 destroys information. If it is used, the number of affected records must be logged and monitored โ a rising count means the upstream encoding changed.
Tip
A byte order mark (BOM) at the start of a UTF-8 file becomes a literal \ufeff in the first field name, which makes row["serial"] raise KeyError while the header looks correct in an editor. Use encoding="utf-8-sig" when reading files produced by Excel or PowerShell's Export-Csv; it strips the BOM if present and behaves like utf-8 if not.
Two visually identical strings can hold different code points. Normalize before using text as a key:
import unicodedata
def normkey(value: str) -> str:
"""Normalize text used for identity comparison."""
return unicodedata.normalize("NFKC", value).strip().casefold()Use casefold() rather than lower() โ it handles cases that lower() does not, such as the German ร.
Note
NFKC folds fullwidth, circled, and mathematical styled letters to ASCII, but not small caps or Cyrillic/Greek homoglyphs (Cyrillic ะฐ stays distinct from Latin a). See filter evasion and normalization for a tested comparison table.
CSV looks simple and is not. Never parse it by splitting on commas; quoted fields legitimately contain commas and newlines.
import csv
with open(path, newline="", encoding="utf-8-sig") as fh:
for row in csv.DictReader(fh):
...| Rule | Reason |
|---|---|
Pass newline="" to open()
|
The csv module handles line endings itself; omitting this corrupts quoted fields containing newlines |
Never use str.split(",")
|
Breaks on any quoted comma โ silently, producing shifted columns |
| Treat every value as a string | The reader does no type conversion; "007", "1e5", and "" all arrive as str
|
| Distinguish empty from missing |
"" is a present empty value; None from DictReader means the row had fewer fields than the header |
DictReader does not raise when a row's field count differs from the header. It silently produces None for missing fields and collects extras under a restkey. Make that explicit:
import csv
def read_csv_strict(path, expected: tuple[str, ...]):
"""Yield (line_number, row) and raise on structural problems."""
with open(path, newline="", encoding="utf-8-sig") as fh:
reader = csv.DictReader(fh, restkey="__extra__", restval=None)
if tuple(reader.fieldnames or ()) != expected:
raise ValueError(
f"header mismatch: expected {expected}, got {tuple(reader.fieldnames or ())}"
)
for row in reader:
if row.get("__extra__") is not None:
raise ValueError(f"line {reader.line_num}: too many fields")
if any(row[f] is None for f in expected):
raise ValueError(f"line {reader.line_num}: too few fields")
yield reader.line_num, rowNote reader.line_num rather than a manual counter: it reports the physical line in the file, which stays accurate when a quoted field spans multiple lines โ the number a human needs to find the bad record.
with open(out_path, "w", newline="", encoding="utf-8") as fh:
writer = csv.DictWriter(fh, fieldnames=["serial", "hostname", "site"])
writer.writeheader()
writer.writerows(rows)Caution
A CSV field beginning with =, +, -, or @ is interpreted as a formula when the file is opened in a spreadsheet application. If the data originates from user or untrusted input and the output will be opened in Excel, this is a live code-execution path (CSV injection). Prefix such values with a single quote, or deliver the data in a format that is not spreadsheet-executable.
A single JSON document must be parsed in full before any element is available โ unbounded memory and unusable for large data. JSON Lines (one complete JSON object per line) fixes this and is the default interchange format for pipeline records.
import json
# Reading: bounded memory, and one bad line does not lose the file
def read_jsonl(path):
with open(path, encoding="utf-8") as fh:
for line_no, line in enumerate(fh, start=1):
line = line.strip()
if not line:
continue
try:
yield line_no, json.loads(line)
except json.JSONDecodeError as exc:
yield line_no, {"__error__": str(exc), "__raw__": line[:500]}
# Writing: append-friendly, streamable
def write_jsonl(path, records):
with open(path, "a", encoding="utf-8") as fh:
for record in records:
fh.write(json.dumps(record, ensure_ascii=False, sort_keys=True) + "\n")| Choice | Why |
|---|---|
ensure_ascii=False |
Keeps non-ASCII text readable instead of \uXXXX escapes; the file is already UTF-8 |
sort_keys=True |
Deterministic output, so diffs and content hashes are stable |
Truncate __raw__
|
A malformed line can be enormous; bounded capture prevents a log-flooding failure |
| One object per line, no trailing commas | Any line can be processed independently and appended to safely |
Note
JSON numbers do not preserve integer precision beyond 2^53 in many consumers, and json will happily emit NaN and Infinity, which are not valid JSON and are rejected by strict parsers. Pass allow_nan=False when the output crosses a system boundary, and serialize large identifiers as strings.
Compose processing as a chain of generators. Each stage does one thing, is testable in isolation, and holds one record at a time.
import itertools
def parse(lines):
for line_no, line in lines:
yield line_no, line.rstrip("\n")
def validate(records, required):
for line_no, record in records:
missing = [f for f in required if not record.get(f)]
if missing:
yield line_no, None, f"missing: {','.join(missing)}"
else:
yield line_no, record, None
def enrich(records, sites):
for line_no, record, error in records:
if error is None:
record["region"] = sites.get(record["site"], "unknown")
yield line_no, record, errorEach stage is called with the previous one's output; nothing executes until the final loop pulls records through.
from itertools import islice, groupby
def batched(iterable, size):
"""Yield lists of at most `size` items. Use for chunked database writes."""
iterator = iter(iterable)
while chunk := list(islice(iterator, size)):
yield chunkBatching database writes is the usual fix for a loader that is correct but slow: one transaction per 1,000 records instead of one per record typically improves throughput by an order of magnitude, with the trade-off that a failure loses the current batch rather than a single row.
Tip
Python 3.12 provides itertools.batched directly. The implementation above is included because much operational tooling still runs on 3.10 and 3.11; prefer the standard-library version when the minimum target allows it.
Warning
itertools.groupby only groups consecutive equal keys. It does not sort. Applying it to unsorted input produces multiple groups for the same key and is a frequent source of undercounted aggregates. Either sort first โ which is unbounded โ or aggregate in SQL.
Timestamps cause more silent data corruption than any other field type.
- Store UTC. Convert at the edge, store one representation, format for display only at presentation time.
- Reject naive timestamps from external sources. A timestamp without an offset has no defined meaning. Guessing its zone is a decision to be wrong twice a year.
- Separate event time from ingest time. Event time is when the thing happened; ingest time is when the pipeline saw it. Recording both is what makes lateness measurable.
import datetime as dt
def to_utc(value: str) -> str:
"""Parse an ISO-8601 timestamp and normalize to UTC. Raise if naive."""
parsed = dt.datetime.fromisoformat(value.strip())
if parsed.tzinfo is None:
raise ValueError(f"timestamp lacks timezone offset: {value!r}")
return parsed.astimezone(dt.timezone.utc).isoformat()
def now_utc() -> str:
"""Ingest timestamp. Never use datetime.utcnow()."""
return dt.datetime.now(dt.timezone.utc).isoformat()Caution
datetime.utcnow() returns a naive datetime holding UTC values. Comparing it to an aware datetime raises TypeError; passing it to .astimezone() silently reinterprets it as local time, shifting every record by the machine's offset. It is deprecated as of Python 3.12. Use datetime.now(dt.timezone.utc).
import datetime as dt
from zoneinfo import ZoneInfo
# Epoch seconds โ ALWAYS pass tz, never rely on local interpretation
dt.datetime.fromtimestamp(1757606400, tz=dt.timezone.utc)
# Epoch milliseconds (common in JavaScript and many APIs)
dt.datetime.fromtimestamp(1757606400000 / 1000, tz=dt.timezone.utc)
# Known-zone local time from a source that documents its zone
naive = dt.datetime.strptime("2026-09-11 14:30:00", "%Y-%m-%d %H:%M:%S")
aware = naive.replace(tzinfo=ZoneInfo("America/Los_Angeles"))
utc = aware.astimezone(dt.timezone.utc)Use zoneinfo (standard library since 3.9) rather than fixed offsets: ZoneInfo("America/Los_Angeles") accounts for daylight saving transitions, whereas a hardcoded -08:00 is wrong for eight months of the year.
Note
Before Python 3.11, fromisoformat parsed only the exact format produced by isoformat() and rejected a trailing Z. On 3.11 and later it accepts the full common ISO-8601 range including Z. If code must run on 3.10, normalize first: value.replace("Z", "+00:00").
During a daylight-saving fall-back, a local wall-clock time occurs twice. ZoneInfo defaults to the first occurrence via the fold attribute. If the source does not record an offset, that hour is genuinely ambiguous and roughly one hour of data per year is unorderable. The only real fix is upstream: require offsets in the contract. See Data Quality & Schema Contracts.
print() cannot be filtered, routed, timestamped, or turned off. Use logging from the first version of the script.
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s %(message)s",
)
log = logging.getLogger(__name__)| Level | Meaning | Example |
|---|---|---|
DEBUG |
Per-record detail, off in normal operation | Field values after transformation |
INFO |
Run boundaries and counts | read=7 accepted=4 quarantined=3 |
WARNING |
Handled problem, run continues | A record was quarantined |
ERROR |
The run failed or produced incomplete output | Destination unreachable |
CRITICAL |
Data integrity is at risk | Reconciliation mismatch after commit |
# 1. Lazy formatting โ the string is only built if the level is enabled
log.info("processed %d records from %s", count, path) # correct
log.info(f"processed {count} records from {path}") # builds the string always
# 2. Log exceptions with the traceback attached
try:
load(path)
except Exception:
log.exception("load failed for %s", path) # includes traceback
# 3. Emit one machine-readable summary per run
log.info("run_summary %s", json.dumps({
"source": str(path), "read": read, "accepted": accepted,
"quarantined": quarantined, "duration_s": round(elapsed, 2),
}, sort_keys=True))That final summary line is what makes a pipeline observable without extra infrastructure: it can be grepped, shipped to a log aggregator, or parsed into freshness and error-rate metrics. See Log Aggregation & Visibility for the collection side.
Caution
Never log credentials, tokens, session identifiers, or unredacted personal data. Logs are frequently retained longer and read more widely than the data itself. When a record must be logged for diagnosis, log its identifier rather than its contents.
A single script becomes unmaintainable at roughly the point where it needs tests. This layout scales from a hundred lines to a real project and is directly installable.
assetpipe/
โโโ pyproject.toml
โโโ README.md
โโโ src/
โ โโโ assetpipe/
โ โโโ __init__.py
โ โโโ config.py # paths and settings, no logic
โ โโโ readers.py # source input, one function per format
โ โโโ transform.py # pure functions: value in, value out
โ โโโ sinks.py # destination writes
โ โโโ cli.py # argument parsing and wiring only
โโโ tests/
โโโ fixtures/
โ โโโ assets_small.csv
โโโ test_transform.py
Without it, import assetpipe may resolve to the working-directory copy rather than the installed package, so tests can pass against code that would fail once installed. The src layout makes that impossible.
[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"
[project]
name = "assetpipe"
version = "0.1.0"
description = "Asset inventory ingestion pipeline"
requires-python = ">=3.10"
dependencies = []
[project.scripts]
assetpipe = "assetpipe.cli:main"
[tool.setuptools.packages.find]
where = ["src"]python3 -m venv .venv && source .venv/bin/activate
pip install -e .
assetpipe --helpThe editable install (-e) points at the source tree, so edits take effect without reinstalling. The [project.scripts] entry creates the assetpipe command.
Keep transform.py free of file and database access. Pure functions โ value in, value out โ need no fixtures, no temporary directories, and no mocks. Push all input/output to the edges in readers.py, sinks.py, and cli.py. This single rule is the difference between a test suite that runs in milliseconds and one nobody runs.
Test the cases that actually break pipelines, not the happy path that already works.
| Category | Test |
|---|---|
| Boundary | Empty file, header only, single record, final line without newline |
| Encoding | UTF-8 with BOM, non-ASCII names, undecodable byte |
| Structure | Too few fields, too many fields, reordered header |
| Values | Empty string versus absent key, leading zeros, whitespace padding |
| Time | Naive timestamp, Z suffix, non-UTC offset, epoch seconds |
| Idempotence | Same input twice produces the same destination state |
import datetime as dt
import unittest
from assetpipe.transform import to_utc, normalize_serial
class TestTimestamps(unittest.TestCase):
def test_offset_is_converted_to_utc(self):
self.assertEqual(
to_utc("2026-09-11T07:00:00-07:00"),
"2026-09-11T14:00:00+00:00",
)
def test_naive_timestamp_is_rejected(self):
with self.assertRaises(ValueError):
to_utc("2026-09-11T14:00:00")
def test_already_utc_is_unchanged(self):
self.assertEqual(
to_utc("2026-09-11T14:00:00+00:00"),
"2026-09-11T14:00:00+00:00",
)
class TestSerials(unittest.TestCase):
def test_case_and_whitespace_normalized(self):
self.assertEqual(normalize_serial(" sn-1001 "), "SN-1001")
def test_empty_serial_rejected(self):
with self.assertRaises(ValueError):
normalize_serial(" ")
if __name__ == "__main__":
unittest.main()python3 -m unittest discover -s tests -vTip
Keep fixture files tiny โ five to ten rows containing every defect you care about. A fixture that must be scrolled is a fixture nobody reads, and a test suite that takes minutes is one that gets skipped before a deadline. pytest offers a more concise syntax and is worth adopting, but unittest requires no dependency, which matters for tooling that must run on locked-down hosts.
Assemble the pieces into a working, installable, tested package.
mkdir -p ~/assetpipe/src/assetpipe ~/assetpipe/tests/fixtures
cd ~/assetpipe"""Pure transformation functions. No file or database access belongs here."""
from __future__ import annotations
import datetime as dt
import hashlib
import unicodedata
REQUIRED_FIELDS = ("serial", "hostname", "site", "os", "last_seen")
def normkey(value: str) -> str:
return unicodedata.normalize("NFKC", value).strip().casefold()
def normalize_serial(value: str) -> str:
cleaned = unicodedata.normalize("NFKC", value or "").strip().upper()
if not cleaned:
raise ValueError("serial is empty")
return cleaned
def to_utc(value: str) -> str:
parsed = dt.datetime.fromisoformat((value or "").strip().replace("Z", "+00:00"))
if parsed.tzinfo is None:
raise ValueError(f"timestamp lacks timezone offset: {value!r}")
return parsed.astimezone(dt.timezone.utc).isoformat()
def asset_id(serial: str) -> str:
return hashlib.sha256(normkey(serial).encode("utf-8")).hexdigest()[:32]
def transform_row(row: dict[str, str]) -> dict[str, str]:
"""Return a normalized record or raise ValueError describing the problem."""
for field in REQUIRED_FIELDS:
if not (row.get(field) or "").strip():
raise ValueError(f"missing required field: {field}")
serial = normalize_serial(row["serial"])
return {
"asset_id": asset_id(serial),
"serial": serial,
"hostname": row["hostname"].strip().lower(),
"site": row["site"].strip().lower(),
"os": row["os"].strip(),
"last_seen": to_utc(row["last_seen"]),
}"""Source input. Bounded memory, explicit encoding, structural checks."""
from __future__ import annotations
import csv
from collections.abc import Iterator
from pathlib import Path
from .transform import REQUIRED_FIELDS
def read_assets_csv(path: Path) -> Iterator[tuple[int, dict[str, str]]]:
with path.open(newline="", encoding="utf-8-sig") as fh:
reader = csv.DictReader(fh, restkey="__extra__", restval=None)
header = tuple(reader.fieldnames or ())
if header != REQUIRED_FIELDS:
raise ValueError(f"header mismatch: expected {REQUIRED_FIELDS}, got {header}")
for row in reader:
if row.get("__extra__") is not None:
raise ValueError(f"line {reader.line_num}: too many fields")
if any(row.get(f) is None for f in REQUIRED_FIELDS):
raise ValueError(f"line {reader.line_num}: too few fields")
yield reader.line_num, row"""Argument parsing and wiring. No business logic belongs here."""
from __future__ import annotations
import argparse
import json
import logging
import sys
from pathlib import Path
from .readers import read_assets_csv
from .transform import transform_row
log = logging.getLogger("assetpipe")
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(prog="assetpipe", description="Normalize an asset CSV export")
parser.add_argument("source", type=Path, help="input CSV path")
parser.add_argument("--out", type=Path, required=True, help="accepted JSON Lines output")
parser.add_argument("--quarantine", type=Path, required=True, help="rejected JSON Lines output")
parser.add_argument("--verbose", action="store_true")
args = parser.parse_args(argv)
logging.basicConfig(
level=logging.DEBUG if args.verbose else logging.INFO,
format="%(asctime)s %(levelname)s %(name)s %(message)s",
)
read = accepted = quarantined = 0
args.out.parent.mkdir(parents=True, exist_ok=True)
args.quarantine.parent.mkdir(parents=True, exist_ok=True)
try:
with args.out.open("w", encoding="utf-8") as out, \
args.quarantine.open("w", encoding="utf-8") as bad:
for line_no, row in read_assets_csv(args.source):
read += 1
try:
record = transform_row(row)
except ValueError as exc:
quarantined += 1
log.warning("line %d quarantined: %s", line_no, exc)
bad.write(json.dumps(
{"line": line_no, "reason": str(exc), "raw": row},
sort_keys=True, ensure_ascii=False) + "\n")
continue
accepted += 1
out.write(json.dumps(record, sort_keys=True, ensure_ascii=False) + "\n")
except (OSError, ValueError):
log.exception("run failed for %s", args.source)
return 1
log.info("run_summary %s", json.dumps(
{"source": str(args.source), "read": read,
"accepted": accepted, "quarantined": quarantined}, sort_keys=True))
return 0 if read == accepted + quarantined else 1
if __name__ == "__main__":
sys.exit(main())Create an empty src/assetpipe/__init__.py, then add the pyproject.toml from ยง8.
tests/fixtures/assets_small.csv:
serial,hostname,site,os,last_seen
SN-1001,ws-acct-01,vancouver,Windows 11,2026-09-10T07:03:00-07:00
sn-1002, WS-ACCT-02 ,Vancouver,Windows 11,2026-09-10T14:05:00Z
SN-1003,,portland,Ubuntu 24.04,2026-09-10T14:07:00+00:00
SN-1004,ws-ops-07,seattle,Windows 11,2026-09-10 14:11:00
Use the test module from ยง9, extended for the fixture, then run:
python3 -m venv .venv && source .venv/bin/activate
pip install -e .
python3 -m unittest discover -s tests -v
assetpipe tests/fixtures/assets_small.csv --out /tmp/ok.jsonl --quarantine /tmp/bad.jsonlExpected summary:
INFO assetpipe run_summary {"accepted": 2, "quarantined": 2, "read": 4, "source": "tests/fixtures/assets_small.csv"}
Two records are accepted with sn-1002 normalized to SN-1002 and its Z suffix converted to +00:00; two are quarantined โ one for the missing hostname and one for the timestamp with no offset. Inspect both output files to confirm.
- Why does
newline=""matter when opening a CSV file, and what breaks without it? - What is the difference between
encoding="utf-8"andencoding="utf-8-sig", and which source produces the difference? - Why is
datetime.utcnow()unsafe, and what replaces it? - When does
itertools.groupbyproduce wrong aggregates? - Why does
log.info("count %d", n)differ fromlog.info(f"count {n}")in a hot loop? - What does the
src/layout prevent that a flat layout does not? - A record has
"last_seen": "2026-09-11T14:00:00". Why is quarantining it better than assuming UTC? - Which functions in the lab package can be tested with no filesystem access, and why is that the design goal?
| Area | Verification performed | Limitation |
|---|---|---|
to_utc() |
Executed; 2026-09-11T07:00:00-07:00 โ 2026-09-11T14:00:00+00:00; naive input raised ValueError; Z suffix accepted |
Does not cover leap seconds or pre-1970 dates |
normalize_serial() / asset_id()
|
Executed; whitespace and case normalization confirmed; identifier stable across runs | 32-character truncation is for readability, not a cryptographic commitment |
read_assets_csv() |
Executed against the fixture; header mismatch, too-few-fields, and too-many-fields paths each raised as documented | UTF-8/BOM inputs tested; other encodings not exercised |
assetpipe CLI |
Executed against the fixture; observed read=4 accepted=2 quarantined=2 and inspected both output files |
Run on the fixture only |
| Package install |
pip install -e . completed in a virtual environment and the assetpipe console script resolved |
setuptools backend only; no wheel publication tested |
| Test suite |
python3 -m unittest discover -s tests -v passed |
pytest alternative mentioned but not executed |
batched() |
Executed on a range input; final short chunk returned correctly | Python 3.12 itertools.batched not used in the shown implementation |
Local checks used Python 3.12.3 on Ubuntu with setuptools from the active virtual environment. These identify the verification environment and are not a recommendation to pin to those versions.
Submission Guidelines:
- Prefer standard-library examples; state and justify any added dependency.
- Include a failing case alongside each new pattern โ the defect it prevents.
- State the minimum supported Python version for any version-dependent behavior.
- Keep transformation examples free of file and database access.
- Report what you executed and the exact output observed.
- Update the section index when adding a guide.
| Area | Official References |
|---|---|
| ๐ CSV | csv module ยท RFC 4180 |
| ๐งพ JSON | json module ยท JSON Lines |
| ๐ Time | datetime ยท zoneinfo ยท RFC 3339 |
| ๐ค Encoding | Unicode HOWTO ยท codecs |
| ๐ Iteration | itertools |
| ๐ Logging | Logging HOWTO ยท Logging Cookbook |
| ๐ฆ Packaging | Packaging User Guide ยท pyproject.toml specification |
| ๐งช Testing | unittest ยท pytest |
- ๐๏ธ Data Engineering Section Index
- ๐งฑ Data Engineering Fundamentals
- ๐๏ธ SQL & Data Modeling
- ๐ ETL & ELT Pipeline Design
- โ Data Quality & Schema Contracts
- ๐ก๏ธ Secure Data Pipelines
- ๐ Log Aggregation & Visibility
- ๐ Repository Glossary
| Item | Details |
|---|---|
| ๐ฏ Focus | Correct, bounded-memory, testable Python for data processing |
| ๐งฐ Core Technologies | Python standard library: csv, json, itertools, datetime, zoneinfo, logging, unittest |
| ๐ Format | Reference guide with a complete installable lab package |
| ๐งช Validation Status | Examples and lab executed locally; results and limitations documented above |
| ๐ Location | Data-Engineering/Phase1/python_data_processing.md |
| ๐ Content Review Date | September 11, 2026 |
๐ Stream the Data. State the Encoding. Test the Edges.
Code that holds one record at a time, names its assumptions, and fails loudly outlives code that is merely clever.
Repository: ULTIMATE CYBERSECURITY MASTER GUIDE
Pacific Northwest Computers: PNWC on GitHub
๐ Master Index | ๐ฏ Role Navigation | ๐ Table of Contents | ๐ Legal Notice
โญ Star the repository if you find it useful! โญ
๐ Source: Data-Engineering/Phase1/python_data_processing.md | โ๏ธ Suggest an edit | โฌ
๏ธ Back to Data Engineering
- ๐บ๏ธ Browse by Topic
- ๐ All Repository Files
- ๐งญ Learning Paths
- ๐ Master Guide
- ๐ค Glossary
- โ๏ธ Legal & Disclaimer
๐ Foundations & Learning
๐ก๏ธ Defense & Investigations
- Incident Response
- Open Source Intelligence (OSINT)
- Reverse Engineering
- Operations Security (OPSEC)
- Compliance
โ๏ธ Platforms & Infrastructure
๐ก Hardware, Wireless & Field Projects
- PNWC Cyberdeck
- Hardware Testing
- Hardware Hacking
- Software-Defined Radio
- Wardriving
- uConsole
- Space Security
๐งฐ Assessment & Field Reference
๐ฆ Downloads & Project Resources
๐๏ธ Additional Topics