feat: add isolated Discord archive data plane

[GrantTheAssistant]

Adds the isolated per-tenant GCE archive service, private authenticated API, bounded Firestore projection, backup and restore tooling, hardened host isolation, deployment verification, and full tests. Production rollout keeps each tenant isolated and follows disabled, shadow, then authoritative cutover.

[socket-security]

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Action	Severity	Alert  (click "▶" to expand/collapse)
Warn		Obfuscated code: golang github.com/envoyproxy/go-control-plane/envoy is 90.0% likely obfuscated Confidence: 0.90 Location: Package overview From: ? → golang/github.com/envoyproxy/go-control-plane/envoy@v1.37.0 ℹ Read more on: This package | This alert | What is obfuscated code? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore golang/github.com/envoyproxy/go-control-plane/envoy@v1.37.0. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		Obfuscated code: golang github.com/envoyproxy/go-control-plane/envoy is 90.0% likely obfuscated Confidence: 0.90 Location: Package overview From: ? → golang/github.com/envoyproxy/go-control-plane/envoy@v1.37.0 ℹ Read more on: This package | This alert | What is obfuscated code? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore golang/github.com/envoyproxy/go-control-plane/envoy@v1.37.0. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		Obfuscated code: golang github.com/envoyproxy/go-control-plane/envoy is 90.0% likely obfuscated Confidence: 0.90 Location: Package overview From: ? → golang/github.com/envoyproxy/go-control-plane/envoy@v1.37.0 ℹ Read more on: This package | This alert | What is obfuscated code? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore golang/github.com/envoyproxy/go-control-plane/envoy@v1.37.0. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		Obfuscated code: golang github.com/envoyproxy/go-control-plane/envoy is 90.0% likely obfuscated Confidence: 0.90 Location: Package overview From: ? → golang/github.com/envoyproxy/go-control-plane/envoy@v1.37.0 ℹ Read more on: This package | This alert | What is obfuscated code? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore golang/github.com/envoyproxy/go-control-plane/envoy@v1.37.0. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		Obfuscated code: golang github.com/envoyproxy/go-control-plane/envoy is 90.0% likely obfuscated Confidence: 0.90 Location: Package overview From: ? → golang/github.com/envoyproxy/go-control-plane/envoy@v1.37.0 ℹ Read more on: This package | This alert | What is obfuscated code? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore golang/github.com/envoyproxy/go-control-plane/envoy@v1.37.0. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		Obfuscated code: golang github.com/envoyproxy/go-control-plane/envoy is 90.0% likely obfuscated Confidence: 0.90 Location: Package overview From: ? → golang/github.com/envoyproxy/go-control-plane/envoy@v1.37.0 ℹ Read more on: This package | This alert | What is obfuscated code? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore golang/github.com/envoyproxy/go-control-plane/envoy@v1.37.0. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		Obfuscated code: golang github.com/planetscale/vtprotobuf is 90.0% likely obfuscated Confidence: 0.90 Location: Package overview From: ? → golang/github.com/planetscale/vtprotobuf@v0.6.1-0.20240319094008-0393e58bdf10 ℹ Read more on: This package | This alert | What is obfuscated code? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore golang/github.com/planetscale/vtprotobuf@v0.6.1-0.20240319094008-0393e58bdf10. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		Obfuscated code: golang google.golang.org/appengine/v2 is 90.0% likely obfuscated Confidence: 0.90 Location: Package overview From: ? → golang/google.golang.org/appengine/v2@v2.0.6 ℹ Read more on: This package | This alert | What is obfuscated code? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore golang/google.golang.org/appengine/v2@v2.0.6. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

View full report

[clawsweeper]

Codex review: needs real behavior proof before merge. Reviewed July 15, 2026, 10:12 PM ET / July 16, 2026, 02:12 UTC.

Summary
Adds a per-tenant GCE Discord archive service with an authenticated API, Firebase projection, SQLite tombstones, deployment automation, backup/restore tooling, release packaging, and security hardening.

Reproducibility: not applicable. This PR proposes a new production deployment architecture rather than fixing broken behavior with an established current contract.

Review metrics: 5 noteworthy metrics.

• Patch surface: 48 files, +6,290/-59. The review unit combines API, storage, Discord ingestion, cloud deployment, release packaging, and supply-chain changes.
• Persistent schema: version 4 → 5. The migration affects existing archives and requires explicit upgrade and restore validation.
• Runtime binaries: 1 added, 1 existing changed in packaging. Release archives and Docker images now include a second network-facing executable.
• Cloud dependencies: 52 go.mod additions, 140 go.sum additions. The new production service substantially expands dependency and supply-chain exposure.
• Deployment assets: 12 GCE assets added. Maintainers would own provisioning, systemd, backup, restore, verification, and lifecycle behavior.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🦐 gold shrimp
Result: blocked until real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

• Obtain explicit maintainer sponsorship for either a second GCE/Firebase data plane or a migration toward the existing crawl-remote architecture.
• [P1] Add redacted live proof covering provisioning, allowed and denied API access, projection, backup, restore, and disabled/shadow/authoritative cutover.
• Run go mod tidy and provide fresh-install plus v4-to-v5 upgrade validation with rollback expectations.

Proof guidance:

• [P1] Needs real behavior proof before merge: No after-fix evidence shows a real GCE deployment, authorized and rejected API calls, Firebase projection, backup/restore, or staged cutover; add redacted terminal output or logs, update the PR body to trigger review, and ask a maintainer for @clawsweeper re-review if it does not rerun automatically.

Risk before merge

• [P1] Merging would commit the project to operating and supporting two materially different remote archive architectures without a documented ownership or deprecation relationship.
• [P1] Existing archives would undergo a schema migration, while the supplied evidence does not demonstrate fresh-install, upgrade, rollback, or restored-database behavior in a real environment.
• [P1] The new private API, service-account authentication, source-CIDR enforcement, Firebase writes, metadata-server restrictions, IAM automation, and secret staging create security boundaries that unit tests cannot fully validate.
• [P1] The new systemd services, projection loop, backup/restore scripts, and deployment automation could make an archive unavailable or inconsistent if production assumptions differ from the mocked and static test environment.
• [P1] The added Google Cloud dependency graph and Socket alert require an explicit supply-chain acceptance decision before the new binary is shipped.

Maintainer options:

1. Confirm direction and prove production behavior (recommended)
   Require explicit sponsorship plus redacted fresh-install, v4-to-v5 upgrade, auth-boundary, projection, backup, restore, and cutover evidence before merge.
2. Converge on crawl-remote
   Replace the parallel GCE service with focused extensions to the existing remote archive architecture and preserve only reusable hardening from this branch.
3. Pause the provider-specific feature
   Close this implementation if the long-term operational and support cost of a second data plane is not justified.

Next step before merge

• [P1] A maintainer must choose the permanent remote-archive architecture and review contributor-owned deployment proof; the primary blocker is not a safe automated code repair.

Maintainer decision needed

• Question: Should discrawl permanently support this GCE/Firebase archive data plane alongside the existing separately deployed Cloudflare crawl-remote service?
• Rationale: The branch is technically and operationally broad, introduces a parallel source of remote-archive behavior, and commits maintainers to provider-specific infrastructure; repository evidence cannot determine that product ownership choice automatically.
• Likely owner: steipete — The documented remote architecture and original Discrawl product direction make this the strongest available routing candidate.
• Options:
  • Keep one remote architecture (recommended): Decline the parallel service and implement any missing behavior through the existing crawl-remote boundary or provider-neutral APIs.
  • Sponsor the GCE plane: Accept GCE/Firebase as an additional supported deployment model after defining ownership, compatibility, release, and operational proof requirements.
  • Extract reusable hardening: Pause the data-plane feature and submit narrow PRs for independently useful tombstone, bulk-delete, attachment, and migration hardening.

Security
Needs attention: The new network, cloud IAM, Firebase, host-isolation, secret, and dependency boundaries require live security validation and explicit supply-chain acceptance.

Review findings

• [P1] Use the existing remote archive boundary — deploy/gce/README.md:1
• [P2] Classify directly imported cloud modules as direct dependencies — go.mod:16

Review details

Best possible solution:

Choose one canonical remote-archive architecture; either extend crawl-remote through provider-neutral interfaces, or explicitly sponsor this GCE plane and split generally useful SQLite, deletion, and attachment hardening into narrow changes while requiring live deployment and upgrade evidence for the provider-specific service.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR proposes a new production deployment architecture rather than fixing broken behavior with an established current contract.

Is this the best way to solve the issue?

No. The branch duplicates the supported remote-archive responsibility without first proving that the existing crawl-remote boundary is insufficient; a provider-neutral extension or explicitly sponsored second deployment model would be more maintainable.

Full review comments:

• [P1] Use the existing remote archive boundary — deploy/gce/README.md:1
  The repository already documents openclaw/crawl-remote as the separately deployed remote archive service, but this patch adds a second full API, auth, projection, and deployment stack without showing that the supported path is insufficient. Please extend the existing boundary or obtain explicit approval for two competing implementations; otherwise their auth, query, deletion, and operational behavior will drift.
  Confidence: 0.92
• [P2] Classify directly imported cloud modules as direct dependencies — go.mod:16
  The new source imports Firestore, Firebase, and Google API packages directly, but the corresponding modules are added under the indirect dependency block. Running go mod tidy should promote direct imports, so commit the tidy result to keep module metadata reproducible.
  Confidence: 0.86

Overall correctness: patch is incorrect
Overall confidence: 0.82

AGENTS.md: unclear because the file could not be read completely.

Codex review notes: model internal, reasoning high; reviewed against 0dc6a87cad9a.

Label changes

Label changes:

• add P2: This is a substantial optional feature with meaningful architectural and operational impact, but no evidence of an urgent current-user regression.
• add merge-risk: 🚨 compatibility: The branch changes the persistent SQLite schema and release artifact composition used by existing installations.
• add merge-risk: 🚨 security-boundary: The diff introduces an authenticated network service, Google IAM identities, secret handling, Firebase access, metadata restrictions, and private-network trust rules.
• add merge-risk: 🚨 availability: The new systemd services, projection process, backup/restore path, and provisioning automation can affect archive startup, consistency, and recovery.
• add rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦐 gold shrimp.
• add status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: No after-fix evidence shows a real GCE deployment, authorized and rejected API calls, Firebase projection, backup/restore, or staged cutover; add redacted terminal output or logs, update the PR body to trigger review, and ask a maintainer for @clawsweeper re-review if it does not rerun automatically.

Label justifications:

• P2: This is a substantial optional feature with meaningful architectural and operational impact, but no evidence of an urgent current-user regression.
• merge-risk: 🚨 compatibility: The branch changes the persistent SQLite schema and release artifact composition used by existing installations.
• merge-risk: 🚨 security-boundary: The diff introduces an authenticated network service, Google IAM identities, secret handling, Firebase access, metadata restrictions, and private-network trust rules.
• merge-risk: 🚨 availability: The new systemd services, projection process, backup/restore path, and provisioning automation can affect archive startup, consistency, and recovery.
• rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦐 gold shrimp.
• status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: No after-fix evidence shows a real GCE deployment, authorized and rejected API calls, Firebase projection, backup/restore, or staged cutover; add redacted terminal output or logs, update the PR body to trigger review, and ask a maintainer for @clawsweeper re-review if it does not rerun automatically.

Evidence reviewed

Security concerns:

• [medium] Validate the new dependency chain — go.mod:16
  Socket flagged github.com/envoyproxy/go-control-plane/envoy@v1.37.0, introduced through the new Google Cloud/Firebase dependency graph; verify its generated-code provenance and necessity before including it in shipped artifacts.
  Confidence: 0.9
• [high] Prove the private API boundary in GCE — internal/archiveapi/server.go:1
  Unit tests cannot demonstrate that audience, caller service account, source CIDR, Direct VPC routing, firewall rules, metadata denial, and host isolation compose correctly in a deployed tenant project.
  Confidence: 0.94
• [high] Review provisioning credential exposure — deploy/gce/provision.sh:1
  The provisioning and host-install paths stage bot tokens, service configuration, release archives, and IAM changes; a live review should confirm temporary files, command output, instance metadata, logs, and failure cleanup do not disclose secrets or broaden access.
  Confidence: 0.86

What I checked:

• Existing remote archive path: Current documentation says the remote service is deployed separately in openclaw/crawl-remote using Cloudflare Workers/D1, while discrawl stores its endpoint and calls that service; the PR does not establish why this supported architecture cannot satisfy the requested deployment. (pkg.go.dev) (README.md, 0dc6a87cad9a)
• Parallel provider-specific architecture: The branch introduces a second remote data plane tied directly to GCE, Google IAM, Firebase, Firestore, RTDB, Cloud NAT, systemd, and tenant-specific infrastructure. (deploy/gce/README.md:1, 608fc011f0f4)
• Persistent upgrade surface: The branch raises the SQLite schema from version 4 to 5 and adds message tombstones plus projection indexes, making existing-database upgrade behavior part of the merge contract. (internal/store/store.go:21, 608fc011f0f4)
• Release and runtime expansion: The PR packages a second discrawl-api executable in GoReleaser archives and the Docker image, expanding the supported release and operational surface. (.goreleaser.yaml:29, 608fc011f0f4)
• Supply-chain review signal: The branch adds a substantial Google Cloud/Firebase dependency graph, including github.com/envoyproxy/go-control-plane/envoy@v1.37.0, which Socket flagged for maintainer review. (go.mod:16, 608fc011f0f4)
• Release provenance: The latest published module documentation identifies v0.11.5 and continues to describe the Cloudflare remote service rather than this GCE data plane, so the PR capability is not already shipped. (pkg.go.dev) (CHANGELOG.md:1, 6dbb9e2c5e8e)

Likely related people:

• steipete: The repository and release history identify Peter Steinberger with Discrawl’s original product direction, and the current documented remote archive architecture is central to this decision. (role: feature owner; confidence: high; commits: 6dbb9e2c5e8e; files: README.md, CHANGELOG.md)
• jeanmonet: The current changelog credits recent work on the local-first maintainer archive workflow, making this person relevant to archive operational boundaries and deployment expectations. (role: recent adjacent contributor; confidence: medium; files: CHANGELOG.md, README.md)
• agent-eli: The current changelog credits recent attachment-storage preservation work, which is adjacent to this PR’s archive, attachment, and persistence changes. (role: recent storage contributor; confidence: medium; files: internal/store, internal/media, CHANGELOG.md)

What the crustacean ranks mean

• 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
• 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
• 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
• 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
• 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
• 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
• 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

[steipete]

Thanks for the thoughtful work here, @GrantTheAssistant. A second remote-archive backend is a crawlkit-family architecture decision that needs a design conversation first. Discrawl is keeping one canonical remote boundary until that direction is settled, so we are closing this implementation rather than taking on parallel GCE/Firebase behavior.

The generic SQLite, tombstone, and attachment hardening is valuable. We are extracting those pieces into narrow maintainer-owned PRs with your credit preserved. Thank you for identifying and implementing those improvements.