Skip to content

LakeScan v0.1.1

Choose a tag to compare

View all tags
@github-actions github-actions released this 18 Oct 06:17
· 12 commits to main since this release
v0.1.1
be8f805

LakeScan finds personal data (PII) in S3 data lakes by reading under
10% of each Parquet file. It produces a per-table inventory showing
which columns hold emails, phone numbers, Indian PAN and Aadhaar
numbers, card numbers and IP addresses — with confidence levels,
file counts, and documented suppressions. The HTML report is an
interactive offline dashboard that works in any browser with no
network requests.

The binary is 6 MB with no runtime dependencies. Needs s3:ListBucket
and s3:GetObject. Licensed under FSL-1.1-ALv2: free to use inside
your own organisation.

What's in v0.1.1

Preflight checks (new in M14)

LakeScan now validates credentials and bucket access before the scan
starts, so a misconfigured IAM policy or expired key gives you a
clear error immediately rather than a confusing failure mid-scan.

  • Wrong or expired credentials → exit 3 with instructions to run
    aws configure
  • No access to the bucket → exit 3 naming the exact IAM action
    needed (s3:ListBucket)
  • Bucket in the wrong region → exit 2 naming the correct region
  • Bucket not found → exit 2 with a check-the-name message
  • --pii runs → one-line hint listing the additional permissions
    needed (s3:GetObject, kms:Decrypt)

sts:GetCallerIdentity is now required and is in the sample IAM
policy in README.

Detection improvements (M12)

  • Person-name and date-of-birth column hints now recognise real-world
    spellings: FIRST, LAST, MIDDLE, MAIDEN, family, given, fname,
    lname, forename, bdate, birth_dt. Non-person qualifiers
    (product_name, station_name, drug_name, multipleBirthInteger) no
    longer fire.
  • US (NANP), UK and EU phone formats accepted with separators.
    Bare digit runs only confirm as phone when the column name
    includes a phone/contact word — prevents US NPI identifiers
    from being flagged as phones.
  • Repeated (list and map) columns are now read and named for their
    hints by the leaf field — fixes missed emails and names in FHIR
    and similar nested data.
  • Placeholder cleaning: example.com/org/net emails, masked values
    (****1234), and sentinel dates (1900-01-01, 1970-01-01) are
    dropped before judging a column.
  • A column with zero cleaned values never produces a finding,
    even on a name hint.

Coverage honesty (M11.1 + M12)

  • Columns that could hold personal data but were not examined
    (repeated types, unsupported types, float/decimal columns named
    like PII) are recorded per table as columns_not_examined.
  • A table with such columns never reports "no PII found" — it
    reports what was and wasn't checked.

Other fixes

  • Empty or mistyped prefix prints a warning to stderr and exits 0
    rather than silently succeeding.
  • Wrong-region redirect names the correct region in the error
    message.
  • Fixture emails moved to the reserved .example TLD.

Binary sizes

Target Size
aarch64-apple-darwin 5.8 MB
x86_64-apple-darwin 6.6 MB
x86_64-unknown-linux-gnu 7.0 MB
aarch64-unknown-linux-gnu 6.3 MB

Full changelog: CHANGELOG.md
What a scan can miss: README — "What a scan can miss"
Required IAM permissions: README — "Permissions"

Assets 7
Loading