LakeScan v0.1.2

Release 0.1.2: dashboard coverage rounding fix

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

LakeScan v0.1.1

LakeScan finds personal data (PII) in S3 data lakes by reading under
10% of each Parquet file. It produces a per-table inventory showing
which columns hold emails, phone numbers, Indian PAN and Aadhaar
numbers, card numbers and IP addresses — with confidence levels,
file counts, and documented suppressions. The HTML report is an
interactive offline dashboard that works in any browser with no
network requests.

The binary is 6 MB with no runtime dependencies. Needs s3:ListBucket
and s3:GetObject. Licensed under FSL-1.1-ALv2: free to use inside
your own organisation.

---

What's in v0.1.1

Preflight checks (new in M14)

LakeScan now validates credentials and bucket access before the scan
starts, so a misconfigured IAM policy or expired key gives you a
clear error immediately rather than a confusing failure mid-scan.

• Wrong or expired credentials → exit 3 with instructions to run
  aws configure
• No access to the bucket → exit 3 naming the exact IAM action
  needed (s3:ListBucket)
• Bucket in the wrong region → exit 2 naming the correct region
• Bucket not found → exit 2 with a check-the-name message
• --pii runs → one-line hint listing the additional permissions
  needed (s3:GetObject, kms:Decrypt)

sts:GetCallerIdentity is now required and is in the sample IAM
policy in README.

Detection improvements (M12)

• Person-name and date-of-birth column hints now recognise real-world
  spellings: FIRST, LAST, MIDDLE, MAIDEN, family, given, fname,
  lname, forename, bdate, birth_dt. Non-person qualifiers
  (product_name, station_name, drug_name, multipleBirthInteger) no
  longer fire.
• US (NANP), UK and EU phone formats accepted with separators.
  Bare digit runs only confirm as phone when the column name
  includes a phone/contact word — prevents US NPI identifiers
  from being flagged as phones.
• Repeated (list and map) columns are now read and named for their
  hints by the leaf field — fixes missed emails and names in FHIR
  and similar nested data.
• Placeholder cleaning: example.com/org/net emails, masked values
  (****1234), and sentinel dates (1900-01-01, 1970-01-01) are
  dropped before judging a column.
• A column with zero cleaned values never produces a finding,
  even on a name hint.

Coverage honesty (M11.1 + M12)

• Columns that could hold personal data but were not examined
  (repeated types, unsupported types, float/decimal columns named
  like PII) are recorded per table as columns_not_examined.
• A table with such columns never reports "no PII found" — it
  reports what was and wasn't checked.

Other fixes

• Empty or mistyped prefix prints a warning to stderr and exits 0
  rather than silently succeeding.
• Wrong-region redirect names the correct region in the error
  message.
• Fixture emails moved to the reserved .example TLD.

Binary sizes

Target	Size
aarch64-apple-darwin	5.8 MB
x86_64-apple-darwin	6.6 MB
x86_64-unknown-linux-gnu	7.0 MB
aarch64-unknown-linux-gnu	6.3 MB

---

Full changelog: CHANGELOG.md
What a scan can miss: README — "What a scan can miss"
Required IAM permissions: README — "Permissions"

LakeScan v0.1.0

The first release: an S3 bucket audit from the object listing, and, with
--pii, a scan of Parquet files for personal data, run in your own account
with read-only access. A discovery aid, not a compliance assessment: no
finding does not prove there is no personal data.

Bucket audit (no object contents read)

• lakescan scan s3://bucket/prefix pages through the listing with fixed
  memory, however large the bucket.
• Six rules: secret-looking keys (sensitive-key), deprecated
  REDUCED_REDUNDANCY storage, stale STANDARD data, small objects in cold
  tiers, empty objects, and unrecognized storage classes.
• A summary as text or JSON: inventory by storage class, findings by rule
  and severity, examples, the largest objects, size and age histograms, and
  an estimate of duplicated content.
• --findings: one JSON line per finding as it is found, keys untruncated.
• Frozen exit codes for CI: 0 clean, 1 findings at or above
  --fail-on, and distinct codes for usage, credential, network, report
  and coverage failures.
• Suppressions (--config): accept a finding by key or prefix, with a
  reason and an optional expiry date. Expired and unused entries are
  reported.
• Hostile key names (terminal escapes, right-to-left overrides) are escaped
  in text output and round-trip exactly in JSON.

PII scanning of Parquet files (--pii)

• Finds emails, phone numbers (E.164 and Indian mobiles), Indian PAN and
  Aadhaar numbers, card numbers and IP addresses from the values, and
  people's names and dates of birth from column names.
• Reads only what reveals a column's contents: the footer, the first data
  page of up to three row groups, and dictionary pages. Small files are
  read whole, in one request. On large files that is 0.8-8.1% of the data
  in tests.
• Messy data is cleaned first (nulls, blanks, placeholders, padding,
  dominant default values), and every verdict comes with a confidence:
  confirmed, likely or possible.
• Snappy, ZSTD, LZ4, GZIP and Brotli pages. GZIP and Brotli are decoded by
  LakeScan within each page's declared size, so a hostile file cannot
  exhaust memory.
• Never reports a value: only the column, the rule, and how many sampled
  values matched.
• A per-table PII inventory: partitioned and Iceberg-style keys are
  grouped into tables, with each table's PII columns and whether each is
  documented by a suppression. Also as CSV (--inventory).
• Coverage fails closed: a Parquet file that could not be scanned
  (unreadable, encrypted, access denied, too wide, and so on) fails the run
  with exit code 6 until a suppression documents it.
• Parquet files without an extension (as Athena CTAS and Trino write them)
  are found by sniffing: one 8-byte ranged GET per object with no
  extension, or one not known to be another format, checks the PAR1
  magic (--pii-sniff unknown, the default; none turns it off).
• Every output says what it could not say: inspection is sampled in every
  mode (a whole-file download is still sampled); objects out of scope are
  counted by extension, with sniffed non-Parquet objects and empty ones; a
  table with nothing found reads "No structured PII detected in sampled
  data from N files"; and a truncated inventory is flagged in the text, the
  HTML report, on stderr and in JSON.
• Column-name hints for people's names and dates of birth no longer fire
  on product_name, file_name, birth_place and the like.
• Each table shows the date range of its files; a documented location
  shows when its suppression expires; the undocumented count is split by
  confidence.
• Concurrent scanning (--concurrency) within a memory budget
  (--memory-budget, 512 MiB by default, at most 64 GiB), with nearby page
  reads coalesced into one request (--coalesce-gap). Output is identical
  at any concurrency. A file that waits more than 10 minutes for memory is
  content-not-scanned with reason budget-timeout.

Reports and release

• --html: a self-contained, offline HTML report with every name escaped
  and no personal-data values.
• Every report records the scan's settings, and a --config file by its
  SHA-256 and entry count, never its path.
• Output files are written to a temporary file and renamed into place only
  when the run completes, so a failed run never leaves a partial report.
• JSON schema_version 1 is now stable, under an additive-only
  compatibility policy. The last breaking change before the freeze:
  largest_objects entries no longer carry severity, which was always
  null.
• Release binaries for macOS (Apple silicon and Intel) and Linux (x86_64
  and arm64, glibc 2.17 or newer, so Amazon Linux 2023 and Ubuntu 22.04 and
  later), each with a SHA256SUMS entry.
• Licensed under FSL-1.1-ALv2: free to use, modify and run inside your own
  organization; each version becomes Apache-2.0 two years after its
  release.