Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ load("@gazelle//:def.bzl", "gazelle")
# Resolve protobuf import ambiguities - use the actual protopb packages, not the proto aliases
# gazelle:resolve go github.com/uber/submitqueue/gateway/protopb //gateway/protopb
# gazelle:resolve go github.com/uber/submitqueue/orchestrator/protopb //orchestrator/protopb
# gazelle:resolve go github.com/uber/submitqueue/stovepipe/protopb //stovepipe/protopb
# gazelle:resolve go github.com/uber/submitqueue/stovepipe/gateway/protopb //stovepipe/gateway/protopb

# Export marker files for test data dependencies (used by FindRepoRoot in tests)
exports_files(
Expand Down
132 changes: 71 additions & 61 deletions Makefile

Large diffs are not rendered by default.

8 changes: 8 additions & 0 deletions core/extension/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
load("@rules_go//go:def.bzl", "go_library")

go_library(
name = "extension",
srcs = ["extension.go"],
importpath = "github.com/uber/submitqueue/core/extension",
visibility = ["//visibility:public"],
)
17 changes: 17 additions & 0 deletions core/extension/extension.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
// Copyright (c) 2025 Uber Technologies, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

// Package extension holds extension interfaces and implementations shared
// across SubmitQueue, Stovepipe, and other repo-local services.
package extension
50 changes: 39 additions & 11 deletions example/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,11 @@ Example gRPC servers and clients for running the submitqueue services locally.

## Services

- **Gateway** (port 8081) — entry point for land requests. Exposes `Ping` and `Land` RPCs.
- **Orchestrator** (port 8082) — coordinates the pipeline. Exposes `Ping` RPC and consumes queue messages across 9 pipeline topics.
- **SubmitQueue Gateway** (port 8081) — entry point for land requests. Exposes `Ping` and `Land` RPCs.
- **SubmitQueue Orchestrator** (port 8082) — coordinates the pipeline. Exposes `Ping` RPC and consumes queue messages across 9 pipeline topics.
- **Stovepipe Gateway** (port 8083) - entry point for commit deployment verification requests. Exposes `Ping` RPC.

Both services require MySQL (app database + queue database). Docker Compose handles this automatically.
Services require MySQL (app database + queue database). Docker Compose handles this automatically.

## Directory Structure

Expand All @@ -23,48 +24,62 @@ example/
│ ├── main.go # Orchestrator server entry point
│ ├── Dockerfile
│ └── docker-compose.yml # Orchestrator-only stack
└── client/
├── gateway/main.go # Gateway ping client
└── orchestrator/main.go # Orchestrator ping client
├── client/
│ ├── gateway/main.go # Gateway ping client
│ └── orchestrator/main.go # Orchestrator ping client
└── stovepipe/
└── gateway/
├── server/
│ ├── main.go # Stovepipe gateway gRPC server (Docker: :8080; go run default :8083)
│ ├── Dockerfile
│ └── docker-compose.yml # Gateway-only stack
└── client/main.go # Stovepipe gateway ping client
```

## Running

### Docker Compose (Recommended)
### Docker Compose (recommended)


```bash
# Start full stack (Gateway + Orchestrator + MySQL)
# Start full SubmitQueue stack (Gateway + Orchestrator + MySQL)
make local-start

# Start individual services
make local-gateway-start
make local-orchestrator-start

# Start Stovepipe gateway (Gateway + 2x MySQL)
make local-stovepipe-gateway-start

# View logs and status
make local-logs
make local-ps

# Stop
# Stop (SubmitQueue + Stovepipe default projects)
make local-stop
```

For Docker, `make build-stovepipe-gateway-linux` copies a Linux binary to `.docker-bin/stovepipe-gateway` so it does not overwrite SubmitQueue’s `.docker-bin/gateway`. Stovepipe `make local-stovepipe-gateway-start` applies **only the queue schema** on `mysql-queue` (`make local-init-stovepipe-queue-schema`); SubmitQueue storage/counter schemas on `mysql-app` are skipped until Stovepipe has its own app schema. `make local-stop` stops the SubmitQueue stack and runs `docker compose down` on the Stovepipe gateway compose file for **`STOVEPIPE_LOCAL_PROJECT`** (default `stovepipe`). SubmitQueue examples use project **`submitqueue`** by default (`make SUBMITQUEUE_LOCAL_PROJECT=myname ...`). Stovepipe containers are named like `stovepipe-mysql-app-1`, not `submitqueue-*`.

### Bazel

```bash
# Build servers
bazel build //example/server/gateway:gateway
bazel build //example/server/orchestrator:orchestrator
bazel build //example/stovepipe/gateway/server:gateway

# Build clients
bazel build //example/client/gateway:gateway
bazel build //example/client/orchestrator:orchestrator
bazel build //example/stovepipe/gateway/client:gateway
```

### Go

```bash
go run example/server/gateway/main.go
go run example/server/orchestrator/main.go
go run example/stovepipe/gateway/server/main.go
```

## Testing with Clients
Expand All @@ -73,6 +88,7 @@ go run example/server/orchestrator/main.go
# Go clients
go run example/client/gateway/main.go -addr localhost:8081 -message "hello"
go run example/client/orchestrator/main.go -addr localhost:8082 -message "hello"
go run example/stovepipe/gateway/client/main.go -addr localhost:8083 -message "hello"
```

Client flags:
Expand All @@ -93,14 +109,17 @@ go install github.com/fullstorydev/grpcurl/cmd/grpcurl@latest
# Ping
grpcurl -plaintext -d '{"message": "hello"}' localhost:8081 uber.submitqueue.gateway.SubmitQueueGateway/Ping
grpcurl -plaintext -d '{"message": "hello"}' localhost:8082 uber.submitqueue.orchestrator.SubmitQueueOrchestrator/Ping
grpcurl -plaintext -d '{"message": "hello"}' localhost:8083 uber.submitqueue.stovepipe.StovepipeGateway/Ping

# List services
grpcurl -plaintext localhost:8081 list
grpcurl -plaintext localhost:8082 list
grpcurl -plaintext localhost:8083 list

# Describe a service
grpcurl -plaintext localhost:8081 describe uber.submitqueue.gateway.SubmitQueueGateway
grpcurl -plaintext localhost:8082 describe uber.submitqueue.orchestrator.SubmitQueueOrchestrator
grpcurl -plaintext localhost:8083 describe uber.submitqueue.stovepipe.StovepipeGateway
```

## API Reference
Expand All @@ -123,3 +142,12 @@ grpcurl -plaintext localhost:8082 describe uber.submitqueue.orchestrator.SubmitQ
| Method | Description |
|--------|-------------|
| `Ping` | Health check, returns service name and timestamp |

### Stovepipe Gateway

**Service**: `uber.submitqueue.stovepipe.StovepipeGateway`
**Proto**: `stovepipe/gateway/proto/gateway.proto`

| Method | Description |
|--------|-------------|
| `Ping` | Health check |
12 changes: 0 additions & 12 deletions example/server/stovepipe/Dockerfile

This file was deleted.

19 changes: 0 additions & 19 deletions example/server/stovepipe/docker-compose.yml

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
load("@rules_go//go:def.bzl", "go_binary", "go_library")

go_library(
name = "stovepipe_lib",
name = "gateway_lib",
srcs = ["main.go"],
importpath = "github.com/uber/submitqueue/example/client/stovepipe",
importpath = "github.com/uber/submitqueue/example/stovepipe/gateway/client",
visibility = ["//visibility:private"],
deps = [
"//stovepipe/protopb",
"//stovepipe/gateway/protopb",
"@org_golang_google_grpc//:grpc",
"@org_golang_google_grpc//credentials/insecure",
],
)

go_binary(
name = "stovepipe",
embed = [":stovepipe_lib"],
name = "gateway",
embed = [":gateway_lib"],
visibility = ["//visibility:public"],
)
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,13 @@ import (
"os"
"time"

pb "github.com/uber/submitqueue/stovepipe/protopb"
pb "github.com/uber/submitqueue/stovepipe/gateway/protopb"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)

func main() {
addr := flag.String("addr", "localhost:8083", "stovepipe server address")
addr := flag.String("addr", "localhost:8083", "stovepipe gateway server address")
message := flag.String("message", "", "message to send in ping request")
timeout := flag.Duration("timeout", 5*time.Second, "request timeout")
flag.Parse()
Expand All @@ -50,7 +50,7 @@ func run(addr, message string, timeout time.Duration) error {
defer conn.Close()

// Create a client
client := pb.NewSubmitQueueStovepipeClient(conn)
client := pb.NewStovepipeGatewayClient(conn)

// Create context with timeout
ctx, cancel := context.WithTimeout(context.Background(), timeout)
Expand All @@ -61,7 +61,7 @@ func run(addr, message string, timeout time.Duration) error {
Message: message,
}

fmt.Printf("Sending ping to stovepipe at %s...\n", addr)
fmt.Printf("Sending ping to stovepipe gateway at %s...\n", addr)
resp, err := client.Ping(ctx, req)
if err != nil {
return fmt.Errorf("ping failed: %w", err)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,13 @@ exports_files(
)

go_library(
name = "stovepipe_lib",
name = "gateway_server_lib",
srcs = ["main.go"],
importpath = "github.com/uber/submitqueue/example/server/stovepipe",
importpath = "github.com/uber/submitqueue/example/stovepipe/gateway/server",
visibility = ["//visibility:private"],
deps = [
"//stovepipe/controller",
"//stovepipe/protopb",
"//stovepipe/gateway/controller",
"//stovepipe/gateway/protopb",
"@com_github_uber_go_tally_v4//:tally",
"@org_golang_google_grpc//:grpc",
"@org_golang_google_grpc//reflection",
Expand All @@ -21,7 +21,7 @@ go_library(
)

go_binary(
name = "stovepipe",
embed = [":stovepipe_lib"],
name = "gateway",
embed = [":gateway_server_lib"],
visibility = ["//visibility:public"],
)
15 changes: 15 additions & 0 deletions example/stovepipe/gateway/server/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
FROM debian:bookworm-slim

RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/*
WORKDIR /root/

# Copy pre-built Linux binary (host path is stovepipe-* to avoid clobbering main .docker-bin/gateway).
# Built via: make build-stovepipe-gateway-linux
COPY .docker-bin/stovepipe-gateway ./gateway

# Reuse SubmitQueue sample queue YAML; compose sets QUEUE_CONFIG_PATH for parity with main gateway image.
COPY example/server/gateway/queues.yaml ./queues.yaml

EXPOSE 8080

CMD ["./gateway"]
68 changes: 68 additions & 0 deletions example/stovepipe/gateway/server/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Docker Compose for Stovepipe gateway manual testing
#
# Mirrors example/server/gateway/docker-compose.yml: same MySQL pair, healthchecks,
# env wiring, and startup ordering. The Stovepipe gateway binary is Ping-only today
# and does not open MySQL yet; variables are set so future work matches SubmitQueue.
#
# IMPORTANT: Before running compose, build the Linux binary:
# make build-stovepipe-gateway-linux
# OR
# bazel build --platforms=@rules_go//go/toolchain:linux_amd64 //example/stovepipe/gateway/server:gateway
#
# Quick start:
# make local-stovepipe-gateway-start
#
# After `up`, only the queue schema is applied (`local-init-stovepipe-queue-schema`).

services:
# Application Database - Stores business data (requests, counters, etc.)
mysql-app:
image: mysql:8.0
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_DATABASE: submitqueue
ports:
- "3306" # Random ephemeral port to avoid conflicts
healthcheck:
# Use 127.0.0.1 (TCP) instead of localhost (Unix socket). MySQL treats
# "localhost" as a socket connection, which can be ready before the TCP
# listener — causing dependent services that connect over TCP to fail.
test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "-proot"]
interval: 5s
timeout: 5s
retries: 10

# Queue Database - Messaging infrastructure (messages, offsets, partition leases)
# Separate from app DB to demonstrate queue is pluggable infrastructure
mysql-queue:
image: mysql:8.0
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_DATABASE: submitqueue
ports:
- "3306" # Random ephemeral port to avoid conflicts
healthcheck:
test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "-proot"]
interval: 5s
timeout: 5s
retries: 10

stovepipe-service:
build:
context: ${REPO_ROOT}
dockerfile: example/stovepipe/gateway/server/Dockerfile
ports:
- "8080" # Random ephemeral port to avoid conflicts
environment:
- PORT=:8080
# Application database connection
- MYSQL_DSN=root:root@tcp(mysql-app:3306)/submitqueue?parseTime=true
# Queue infrastructure connection (separate database)
- QUEUE_MYSQL_DSN=root:root@tcp(mysql-queue:3306)/submitqueue?parseTime=true
# Path to YAML queue configuration baked into the image
- QUEUE_CONFIG_PATH=/root/queues.yaml
depends_on:
mysql-app:
condition: service_healthy
mysql-queue:
condition: service_healthy
Loading
Loading