Skip to content
Go to file

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

CT Map/Reduce Tooling


  1. Install the python dependencies: pip install -r python/requirements.txt
  2. Build the CT-to-Disk scraper: go get


  1. Create a configuration file:
cat > ~/.ct-fetch.conf <<EOF
# Available directives:
# certPath = Path under which to store full DER-encoded certificates
# issuerCNFilter = Prefixes to match for CNs for permitted issuers, comma delimited
# runForever = Run forever, pausing `pollingDelay` between runs
# pollingDelayMean = Mean polling delay duration
# pollingDelayStdDev = A standard deviation, like 100, or 1000.
# logExpiredEntries = Add expired entries to the database
# numThreads = Use this many threads per CPU
# logList = URLs of the CT Logs, comma delimited
# cacheSize = Size of internal cache in entries, default is probably fine
# Examples
# Only accept certificates for Let's Encrypt's intermediates or the ISRG roots
issuerCNFilter = Let's Encrypt, ISRG
# Update the CT log list as you like, comma-delimited. Not currently tested with more than one log.
logList =
# Choose if this should complete when it catches up to the CT logs, or be a daemon
runForever = false

# Optionally, a path with plenty of disk space if you want to save PEM files
# certPath = /ct

# Redis cache server
redisHost =
redisTimeout = 2s



You'll also need to configure your Redis instance with maxmemory_policy:noeviction, which is checked programmatically and a warning will go to the logs if not set correctly.

IAM for Google Cloud

Permissions that seem necessary are:

Cloud Datastore Owner
Cloud Datastore User
Monitoring Metric Writer
Cloud Memorystore Redis Editor
Storage Admin
Storage Object Admin

Populating your storage and Redis with CT certificates

ct-fetch -config ~/.ct-fetch.conf

Note: Consider using --offset X to start from the Xth log entry. Also, --limit Y will stop after processing Y certificates.


my_ip=$(ipconfig getifaddr en0) # macOS
docker run redis:4
# for some reason `docker run -p 6379:7000 redis:4 --port 7000` is needed for me

RedisHost=${my_ip}:6379 go test -v ./...


Map/Reduce functions for processing Certificate Transparency. Used for



No packages published


You can’t perform that action at this time.