Skip to content

feat: add CRM people sync cache and human audit event ingestion #34

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
michaelmwu merged 3 commits into from
Feb 21, 2026
Merged

feat: add CRM people sync cache and human audit event ingestion #34

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3e3a88a
feat: add CRM people sync and human audit event pipeline
michaelmwu Feb 21, 2026
e65e2e6
fix: persist webhook jobs before returning 202
michaelmwu Feb 21, 2026
e316926
Merge remote-tracking branch 'origin/main' into michaelmwu/audit-log-…
michaelmwu Feb 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion .env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# Shared runtime (optional defaults)
LOG_LEVEL=INFO
RUNTIME_ENV=local

# Queue transport (optional defaults)
REDIS_URL=redis://redis:6379/0
Expand All @@ -12,7 +13,9 @@ REDIS_SOCKET_TIMEOUT=5.0
POSTGRES_URL=postgresql://postgres:postgres@postgres:5432/workflows
POSTGRES_DB=workflows
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres # change in production
# Change in production
POSTGRES_PASSWORD=postgres
POSTGRES_HOST_BIND=127.0.0.1
# Optional: expose postgres on host for local debugging
POSTGRES_PORT=5432

Expand All @@ -26,6 +29,7 @@ JOB_RESULT_TTL_SECONDS=3600
# Internal transfer storage (optional defaults for local compose)
MINIO_ENDPOINT=http://minio:9000
MINIO_ROOT_USER=internal
# Change in production
MINIO_ROOT_PASSWORD=change-me
MINIO_INTERNAL_BUCKET=internal-transfers
MINIO_HOST_BIND=127.0.0.1
Expand All @@ -49,12 +53,17 @@ RESUME_KEYWORDS=resume,cv,curriculum
OPENAI_API_KEY=
OPENAI_BASE_URL=
OPENAI_MODEL=gpt-4o-mini
CRM_SYNC_ENABLED=true
CRM_SYNC_INTERVAL_SECONDS=900
CRM_SYNC_PAGE_SIZE=200

# Discord bot (required for bot runtime)
DISCORD_BOT_TOKEN=your_bot_token_here
HEALTHCHECK_PORT=3000
DISCORD_SENDMSG_CHARACTER_LIMIT=2000
CHECK_EMAIL_WAIT=2
AUDIT_API_BASE_URL=
AUDIT_API_TIMEOUT_SECONDS=2.0
Comment on lines +65 to +66
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Feb 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

AUDIT_API_BASE_URL is blank with no guidance comment.

Other empty required values (e.g., WEBHOOK_SHARED_SECRET, OPENAI_API_KEY) have comments like "Required: …" or descriptive placeholders. Consider adding a comment and a sensible local default (e.g., http://worker:8090) so developers know what to set and local-compose works out of the box.

Proposed fix 
-AUDIT_API_BASE_URL=
-AUDIT_API_TIMEOUT_SECONDS=2.0
+# Required for Discord bot audit logging; points to the worker ingest API
+AUDIT_API_BASE_URL=http://worker:8090
+AUDIT_API_TIMEOUT_SECONDS=2.0
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
AUDIT_API_BASE_URL=
AUDIT_API_TIMEOUT_SECONDS=2.0
# Required for Discord bot audit logging; points to the worker ingest API
AUDIT_API_BASE_URL=http://worker:8090
AUDIT_API_TIMEOUT_SECONDS=2.0
🧰 Tools
🪛 dotenv-linter (4.0.0)

[warning] 65-65: [UnorderedKey] The AUDIT_API_BASE_URL key should go before the CHECK_EMAIL_WAIT key

(UnorderedKey)

[warning] 66-66: [UnorderedKey] The AUDIT_API_TIMEOUT_SECONDS key should go before the CHECK_EMAIL_WAIT key

(UnorderedKey)

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.env.example around lines 65 - 66, AUDIT_API_BASE_URL is empty and lacks the
guidance comments used for other required envs; update the .env.example to set a
sensible local default (e.g., AUDIT_API_BASE_URL=http://worker:8090) and add a
short comment like "Required: URL of the Audit API (local dev default shown)"
similar to the existing comments for WEBHOOK_SHARED_SECRET and OPENAI_API_KEY so
developers know what to set and local-compose works out of the box; leave
AUDIT_API_TIMEOUT_SECONDS unchanged unless you want to document its units in a
one-line comment.

# Required for Discord bot commands that post to channels
CHANNEL_ID=1391742724666822798

Expand Down
7 changes: 7 additions & 0 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,16 @@ This repo contains multiple services:
- `apps/worker`: webhook ingest API and queue consumer
- `docker-compose.yml`: stack orchestration with Redis, Postgres, and MinIO

4. Human audit logging
- Human-triggered CRM actions from Discord should write to the worker audit ingest endpoint.
- Audit logging must be best effort: command flows should never fail if audit writes fail.
- Keep reusable audit-write logic outside individual cogs.

## Common Paths

- Bot core: `apps/discord_bot/src/five08/discord_bot/bot.py`
- Bot config: `apps/discord_bot/src/five08/discord_bot/config.py`
- Bot audit helper: `apps/discord_bot/src/five08/discord_bot/utils/audit.py`
- Worker API: `apps/worker/src/five08/worker/api.py`
- Worker consumer: `apps/worker/src/five08/worker/consumer.py`
- Shared settings: `packages/shared/src/five08/settings.py`
Expand Down Expand Up @@ -87,6 +93,7 @@ async def setup(bot: commands.Bot) -> None:
- Add shared env/config in `packages/shared/src/five08/settings.py`.
- Add service-specific settings in local service `config.py` by subclassing shared settings.
- Keep secrets in env vars, not code.
- For Discord CRM audit writes, use `AUDIT_API_BASE_URL` and shared `WEBHOOK_SHARED_SECRET`.

## Agent Guidelines

Expand Down
8 changes: 8 additions & 0 deletions DEVELOPMENT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,13 +109,21 @@ async def setup(bot: commands.Bot) -> None:
- extract skills (LLM when configured, heuristic fallback otherwise)
- update contact skills field in EspoCRM
- Manual queueing is available via `POST /process-contact/{contact_id}`.
- Human action audit ingest is available at `POST /audit/events`.

## Discord CRM Audit Flow

- CRM slash commands in `apps/discord_bot/src/five08/discord_bot/cogs/crm.py` emit best-effort audit events for human actions.
- Audit writing is centralized in `apps/discord_bot/src/five08/discord_bot/utils/audit.py`.
- Audit writes must never break command execution; failures are logged as warnings only.

## Environment Variables

Use `.env.example` as source of truth. Key categories:

- Shared queue/runtime: `REDIS_URL`, `REDIS_QUEUE_NAME`, `POSTGRES_URL`, `JOB_MAX_ATTEMPTS`, `JOB_RETRY_BASE_SECONDS`, `JOB_RETRY_MAX_SECONDS`, `LOG_LEVEL`, webhook settings
- Bot credentials/integrations: Discord, email, Espo, Kimai
- Discord CRM audit writer: `AUDIT_API_BASE_URL`, `AUDIT_API_TIMEOUT_SECONDS` (plus shared `WEBHOOK_SHARED_SECRET`)
- Worker controls: `WORKER_NAME`, `WORKER_QUEUE_NAMES`, `WORKER_BURST`
- Worker CRM processing: `MAX_ATTACHMENTS_PER_CONTACT`, `MAX_FILE_SIZE_MB`, `ALLOWED_FILE_TYPES`, `RESUME_KEYWORDS`, `OPENAI_API_KEY`, `OPENAI_BASE_URL`, `OPENAI_MODEL`

Expand Down
140 changes: 90 additions & 50 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,13 +41,18 @@ Migrations:
- Job states: `queued`, `running`, `succeeded`, `failed`, `dead`, `canceled`.
- Idempotency key is unique and optional.
- Attempts are stored with `run_after`/retry state so delivery failures are never lost.
- Human audit events are persisted in `audit_events`.
- CRM identity cache is persisted in `people`.

### Worker API Endpoints

- `GET /health`: Redis/Postgres/worker health check.
- `POST /webhooks/{source}`: Generic webhook enqueue endpoint.
- `POST /webhooks/espocrm`: EspoCRM webhook endpoint (expects array payload).
- `POST /webhooks/espocrm/people-sync`: EspoCRM contact-change webhook for people cache sync.
- `POST /process-contact/{contact_id}`: Manually enqueue one contact skills job.
- `POST /sync/people`: Manually enqueue a full CRM->people cache sync.
- `POST /audit/events`: Persist one human audit event (`discord` or `admin_dashboard`).

## Local Development

Expand Down Expand Up @@ -87,59 +92,94 @@ docker compose up --build

## Environment Variables

### Shared (bot + worker)

- `REDIS_URL` (default: `redis://redis:6379/0`)
- `REDIS_QUEUE_NAME` (default: `jobs.default`)
- `REDIS_KEY_PREFIX` (default: `jobs`)
- `ESPO_BASE_URL` (required by both bot and worker)
- `ESPO_API_KEY` (required by both bot and worker)
- `JOB_TIMEOUT_SECONDS` (default: `600`)
- `JOB_RESULT_TTL_SECONDS` (default: `3600`)
- `WEBHOOK_SHARED_SECRET` (required; requests are rejected when unset)
- `POSTGRES_URL` (default: `postgresql://postgres:postgres@postgres:5432/workflows`)
- `POSTGRES_DB` (default: `workflows`)
- `POSTGRES_USER` (default: `postgres`)
- `POSTGRES_PASSWORD` (default: `postgres`)
- `JOB_MAX_ATTEMPTS` (default: `8`)
- `JOB_RETRY_BASE_SECONDS` (default: `5`)
- `JOB_RETRY_MAX_SECONDS` (default: `300`)
- `WEBHOOK_INGEST_HOST` (default: `0.0.0.0`)
- `WEBHOOK_INGEST_PORT` (default: `8090`)
- `LOG_LEVEL` (default: `INFO`)
- `MINIO_ENDPOINT` (default: `http://minio:9000`)
- `MINIO_INTERNAL_BUCKET` (default: `internal-transfers`)
- `MINIO_ROOT_USER` (default: `internal`)
- `MINIO_ROOT_PASSWORD`
- `MINIO_HOST_BIND` (default: `127.0.0.1`; set to `0.0.0.0` to expose MinIO)
- `MINIO_API_PORT` (default: `9000`)
- `MINIO_CONSOLE_PORT` (default: `9001`)
- `MINIO_ACCESS_KEY` / `MINIO_SECRET_KEY` (compatibility aliases; use `MINIO_ROOT_USER` and `MINIO_ROOT_PASSWORD` by default for internal transfers)

### Discord Bot

- `DISCORD_BOT_TOKEN`
- `CHANNEL_ID`
- `EMAIL_USERNAME`
- `EMAIL_PASSWORD`
- `IMAP_SERVER`
- `SMTP_SERVER`
- `KIMAI_BASE_URL`
- `KIMAI_API_TOKEN`
- Optional: `CHECK_EMAIL_WAIT`, `DISCORD_SENDMSG_CHARACTER_LIMIT`, `HEALTHCHECK_PORT`
Use `.env.example` as the source of truth for defaults.

### Core Runtime (Bot + Worker)

- `Required`: `ESPO_BASE_URL`, `ESPO_API_KEY`
- `Optional`: `LOG_LEVEL` (default: `INFO`)
- `Optional`: `RUNTIME_ENV` (default: `local`; non-local values require explicit `POSTGRES_URL` and `MINIO_ROOT_PASSWORD`)

### Queue + Job Runtime

- `Optional`: `REDIS_URL` (default: `redis://redis:6379/0`)
- `Optional`: `REDIS_QUEUE_NAME` (default: `jobs.default`)
- `Optional`: `REDIS_KEY_PREFIX` (default: `jobs`)
- `Optional`: `JOB_TIMEOUT_SECONDS` (default: `600`)
- `Optional`: `JOB_RESULT_TTL_SECONDS` (default: `3600`)
- `Optional`: `JOB_MAX_ATTEMPTS` (default: `8`)
- `Optional`: `JOB_RETRY_BASE_SECONDS` (default: `5`)
- `Optional`: `JOB_RETRY_MAX_SECONDS` (default: `300`)

### Postgres + Compose Exposure

- `Optional`: `POSTGRES_URL` (default: `postgresql://postgres@postgres:5432/workflows`)
- `Optional` (Compose DB container): `POSTGRES_DB` (default: `workflows`)
- `Optional` (Compose DB container): `POSTGRES_USER` (default: `postgres`)
- `Optional` (Compose DB container): `POSTGRES_PASSWORD` (default: `postgres`)
- `Optional` (Compose host bind): `POSTGRES_HOST_BIND` (default: `127.0.0.1`)
- `Optional` (Compose host port): `POSTGRES_PORT` (default: `5432`)

### MinIO + Internal Transfers

- `Required` in non-local environments: `MINIO_ROOT_PASSWORD`
- `Optional`: `MINIO_ENDPOINT` (default: `http://minio:9000`)
- `Optional`: `MINIO_INTERNAL_BUCKET` (default: `internal-transfers`)
- `Optional`: `MINIO_ROOT_USER` (default: `internal`)
- `Optional`: `MINIO_HOST_BIND` (default: `127.0.0.1`; set `0.0.0.0` to expose externally)
- `Optional`: `MINIO_API_PORT` (default: `9000`)
- `Optional`: `MINIO_CONSOLE_PORT` (default: `9001`)
- Note: `MINIO_ACCESS_KEY` / `MINIO_SECRET_KEY` are `SharedSettings` alias properties (`minio_access_key`, `minio_secret_key`) and are not env-loaded fields.
- Note: use `MINIO_ROOT_USER` and `MINIO_ROOT_PASSWORD` as the actual env vars.

### Worker API Ingest

- `Required` for protected endpoints: `WEBHOOK_SHARED_SECRET` (ingest requests are rejected when unset)
- `Optional`: `WEBHOOK_INGEST_HOST` (default: `0.0.0.0`)
- `Optional`: `WEBHOOK_INGEST_PORT` (default: `8090`)

### Worker Consumer

- `WORKER_NAME` (default: `integrations-worker`)
- `WORKER_QUEUE_NAMES` (default: `jobs.default`, comma-separated)
- `WORKER_BURST` (default: `false`)
- `MAX_ATTACHMENTS_PER_CONTACT` (default: `3`)
- `MAX_FILE_SIZE_MB` (default: `10`)
- `ALLOWED_FILE_TYPES` (default: `pdf,doc,docx,txt`)
- `RESUME_KEYWORDS` (default: `resume,cv,curriculum`)
- `OPENAI_API_KEY` (optional; if unset, heuristic extraction is used)
- `OPENAI_BASE_URL` (optional)
- `OPENAI_MODEL` (default: `gpt-4o-mini`)
- `Optional`: `WORKER_NAME` (default: `integrations-worker`)
- `Optional`: `WORKER_QUEUE_NAMES` (default: `jobs.default`, comma-separated)
- `Optional`: `WORKER_BURST` (default: `false`)

### Worker CRM Sync + Skills Extraction

- `Optional`: `CRM_SYNC_ENABLED` (default: `true`)
- `Optional`: `CRM_SYNC_INTERVAL_SECONDS` (default: `900`)
- `Optional`: `CRM_SYNC_PAGE_SIZE` (default: `200`)
- `Optional`: `MAX_ATTACHMENTS_PER_CONTACT` (default: `3`)
- `Optional`: `MAX_FILE_SIZE_MB` (default: `10`)
- `Optional`: `ALLOWED_FILE_TYPES` (default: `pdf,doc,docx,txt`)
- `Optional`: `RESUME_KEYWORDS` (default: `resume,cv,curriculum`)
- `Optional`: `OPENAI_API_KEY` (if unset, heuristic extraction is used)
- `Optional`: `OPENAI_BASE_URL`
- `Optional`: `OPENAI_MODEL` (default: `gpt-4o-mini`)

### Discord Bot Core

- `Required`: `DISCORD_BOT_TOKEN`
- `Required`: `CHANNEL_ID`
- `Optional`: `HEALTHCHECK_PORT` (default: `3000`)
- `Optional`: `DISCORD_SENDMSG_CHARACTER_LIMIT` (default: `2000`)
- `Optional`: `CHECK_EMAIL_WAIT` (default: `2`)

### Discord Email Monitoring

- `Required`: `EMAIL_USERNAME`
- `Required`: `EMAIL_PASSWORD`
- `Required`: `IMAP_SERVER`
- `Required`: `SMTP_SERVER`

### Discord CRM Audit Logging (Best Effort)

- `Optional`: `AUDIT_API_BASE_URL` (when set with `WEBHOOK_SHARED_SECRET`, CRM commands emit best-effort audit events)
- `Optional`: `AUDIT_API_TIMEOUT_SECONDS` (default: `2.0`)

### Kimai (Legacy/Deprecating)

- `Currently required by config model`: `KIMAI_BASE_URL`, `KIMAI_API_TOKEN`

## Commands

Expand Down
1 change: 1 addition & 0 deletions apps/discord_bot/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# Pinned from ghcr.io/astral-sh/uv:python3.12-bookworm-slim (verified 2026-02-21).
FROM ghcr.io/astral-sh/uv@sha256:e5b65587bce7de595f299855d7385fe7fca39b8a74baa261ba1b7147afa78e58
Comment on lines +1 to 2
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Feb 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify the digest matches the claimed tag and check for current tags

IMAGE="ghcr.io/astral-sh/uv"
DIGEST="sha256:e5b65587bce7de595f299855d7385fe7fca39b8a74baa261ba1b7147afa78e58"

echo "=== Fetching manifest for the pinned digest ==="
curl -fsSL \
  -H "Accept: application/vnd.oci.image.manifest.v1+json" \
  "https://ghcr.io/v2/astral-sh/uv/manifests/${DIGEST}" \
  2>/dev/null | jq '{mediaType, config: .config.digest}' || echo "Manifest fetch failed or digest not found"

echo ""
echo "=== Checking available tags for python3.12-bookworm variant ==="
# Query the Docker registry API to list tags
curl -fsSL \
  "https://ghcr.io/v2/astral-sh/uv/tags/list" \
  2>/dev/null | jq '.tags[] | select(contains("python3.12") and contains("bookworm"))' | head -20

Repository: 508-dev/508-workflows

Length of output: 177

🌐 Web query:

astral-sh uv Docker image python3.12-bookworm-slim ghcr.io releases

💡 Result:

  • Image / registry: ghcr.io/astral-sh/uv (Astral’s official GHCR package). The canonical place to browse all tags (“releases”) and digests is the GitHub Container package page (and its “Versions” list). [2]

  • Tag scheme (how releases are published):

    • Distroless / base tags (e.g. latest, 0.10.4, 0.10) and
    • Derived “Python base” images are tagged as:
      ghcr.io/astral-sh/uv:{major}.{minor}.{patch}-{base} and ghcr.io/astral-sh/uv:{major}.{minor}-{base}. [1]

  • For the specific base you asked about (python3.12-bookworm-slim):

    • You should expect version-pinned tags like:
      ghcr.io/astral-sh/uv:<UV_VERSION>-python3.12-bookworm-slim (example known tag: 0.5.26-python3.12-bookworm-slim). [4]
    • There was also discussion that the published tag name is python3.12-bookworm-slim (not python3.12-slim-bookworm). [3]

🌐 Web query:

uv package manager Docker image latest tags 2026

💡 Result:

As of Feb 21, 2026, the official uv Docker images are published as ghcr.io/astral-sh/uv and also mirrored on Docker Hub as docker.io/astral/uv. [1][2]

“Latest” uv version (what :latest points to)

  • The latest released uv version is 0.10.2 (released 2026-02-10), so ghcr.io/astral-sh/uv:latest / astral/uv:latest should correspond to that release. [2][3]

Tag schemes you can use (2026)

Distroless (just the uv binaries):

  • :latest
  • :{major}.{minor}.{patch} (example shown in docs: :0.10.2)
  • :{major}.{minor} (tracks latest patch in that minor line, e.g. :0.8) [2]

Derived OS / Python base images (uv preinstalled):

  • Alpine bases: :alpine, :alpine3.23, :alpine3.22, etc. [2]
  • Debian bases (current in docs): :debian, :debian-slim, :trixie, :trixie-slim [2]
  • Python bases (examples): :python3.14-alpine, :python3.13-alpine, :python3.14-trixie, :python3.14-trixie-slim, etc. [2]

Version-pinned derived images (recommended for reproducibility):

  • :{major}.{minor}.{patch}-{base} (example: :0.10.2-alpine)
  • :{major}.{minor}-{base} (example: :0.10-alpine) [2]

Clarify the uv version in the pinned image comment.

The digest pinning is correct for security. However, the referenced tag ghcr.io/astral-sh/uv:python3.12-bookworm-slim should include the uv version number for reproducibility and clarity. According to the uv Docker image tagging scheme, this should be formatted as ghcr.io/astral-sh/uv:{version}-python3.12-bookworm-slim (example: 0.10.2-python3.12-bookworm-slim for the latest release as of Feb 21, 2026). Update the comment to specify the exact version this digest corresponds to.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@apps/discord_bot/Dockerfile` around lines 1 - 2, Update the Dockerfile
comment to state the exact uv image version that corresponds to the pinned
digest: replace the current tag reference
"ghcr.io/astral-sh/uv:python3.12-bookworm-slim" with the fully qualified tag
including the uv release (e.g.,
"ghcr.io/astral-sh/uv:0.10.2-python3.12-bookworm-slim") so the comment next to
the FROM line accurately documents which uv version the sha256 digest
represents.


WORKDIR /app
Expand Down
5 changes: 0 additions & 5 deletions apps/discord_bot/src/five08/__init__.py
View file
Open in desktop

This file was deleted.

Loading