Add Pub/Sub schema validation using Protocol Buffers

[jonaskay]

Add Pub/Sub schema validation using Protocol Buffers

Background

The webhook service currently publishes raw Slack event JSON bytes to Pub/Sub. There is no schema enforcement: any payload — valid or not — is accepted by the topic, and downstream consumers have no contract to rely on.

Google Cloud Pub/Sub supports native schema validation for topics. When a schema is attached to a topic, the service rejects publishes that don't conform before they reach any subscriber.

Goal

Define a Protocol Buffer schema that mirrors the Slack event envelope, register it with the Pub/Sub topic, and make the webhook publish messages that conform to it. Wire format is JSON-encoded proto (readable during development, still schema-validated server-side).

Scope

1. Proto source

• New file: proto/threadops/v1/slack_event.proto
• syntax = "proto3", package threadops.v1
• go_package = "github.com/jonaskay/threadops/internal/gen/threadops/v1;threadopsv1"
• Messages mirror the Slack event envelope structs in services/webhook/types.go:
  • SlackEvent (root, equivalent to the current OuterEvent)
  • InnerEvent
  • Authorization
• Same field names and types as the Go structs. event_time stays int64.

2. Code generation with protoc

• Generated Go into internal/gen/threadops/v1/slack_event.pb.go
• Add Makefile target generate invoking protoc --go_out=...
• Commit the generated .pb.go so day-to-day builds don't require protoc
• README: document prerequisites (protoc, protoc-gen-go) and the make generate step

3. Publisher

• Change internal/pubsub/publisher.go:18 so Publish takes a proto.Message instead of []byte
• Marshal to JSON via google.golang.org/protobuf/encoding/protojson inside the publisher
• Promotes google.golang.org/protobuf to a direct dependency (already indirect)

4. Webhook handler

• services/webhook/handler.go: after Slack signature verification, parse the request body with protojson.UnmarshalOptions{DiscardUnknown: true} into a SlackEvent and hand the message to the publisher (replacing the raw pub.Publish(ctx, body) at services/webhook/handler.go:33)
  • DiscardUnknown: true is required — Slack sends many fields we don't model
• On JSON parse error: respond 400 Bad Request
• Delete services/webhook/types.go — replaced by the generated proto types

5. Tests

• services/webhook/handler_test.go: the request body becomes a real Slack JSON payload; the fake publisher's Publish takes proto.Message; assert on message fields (e.g. TeamID, Event.Type) instead of byte equality
• Add a new case: malformed JSON → 400

6. Terraform

• New terraform/ directory:
  • google_pubsub_schema.slack_event with type = "PROTOCOL_BUFFER" and definition = file("../proto/threadops/v1/slack_event.proto")
  • google_pubsub_topic.slack_events with schema_settings { schema = ..., encoding = "JSON" }
  • variables.tf (at minimum project_id), versions.tf pinning the google provider
  • No backend config — leave to the operator
• README: section on applying the Terraform

Acceptance criteria

• proto/threadops/v1/slack_event.proto exists and matches the fields currently in services/webhook/types.go
• make generate regenerates internal/gen/threadops/v1/slack_event.pb.go
• Publisher interface accepts proto.Message and publishes JSON-encoded proto
• Webhook handler parses Slack JSON into SlackEvent and republishes it via the typed publisher
• Handler returns 400 for malformed JSON
• Unit tests cover: happy path, verify failure, publish failure, malformed JSON
• terraform apply against a real GCP project creates the schema and a topic bound to it
• Publishing a non-conforming message to the live topic is rejected by Pub/Sub
• README documents proto prerequisites, codegen, and Terraform apply

Out of scope / notes

• E2E tests: e2e/docker-compose.yml uses the Pub/Sub emulator, which does not enforce schemas. E2E tests will keep passing unchanged but won't exercise schema validation — real validation only happens against the live GCP topic. Consider documenting this caveat in e2e/README.md.
• Domain event abstraction: This issue deliberately mirrors the Slack envelope 1:1 rather than introducing a ThreadOps-native domain event type. Decoupling from Slack's shape can be a follow-up once a real processor exists.
• Processor service: services/processor is still a placeholder. Updating it to consume the typed messages is a separate task.

References

• Publisher: internal/pubsub/publisher.go:13
• Publish call site: services/webhook/handler.go:33
• Slack event Go structs: services/webhook/types.go:3
• Topic configuration: PUBSUB_TOPIC env var read in services/webhook/main.go:34