Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: configurable data retention period #693

Merged
merged 3 commits into from
Jul 6, 2024
Merged

feat: configurable data retention period #693

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
785b324
feat: data retention for tenants
abelanger5 Jul 4, 2024
35920a0
chore: generate and docs
abelanger5 Jul 5, 2024
f7cbdb7
chore: lint
abelanger5 Jul 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion Taskfile.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,6 @@ tasks:
- task: generate-sqlc
seed-dev:
cmds:
- sh ./hack/dev/run-go-with-env.sh run github.com/steebchen/prisma-client-go migrate dev --skip-generate
- SEED_DEVELOPMENT=true sh ./hack/dev/run-go-with-env.sh run ./cmd/hatchet-admin seed
start-dev:
deps:
Expand Down
4 changes: 4 additions & 0 deletions api/v1/server/handlers/tenants/create.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,10 @@ func (t *TenantService) TenantCreate(ctx echo.Context, request gen.TenantCreateR
Name: request.Body.Name,
}

if t.config.Runtime.Limits.DefaultTenantRetentionPeriod != "" {
createOpts.DataRetentionPeriod = &t.config.Runtime.Limits.DefaultTenantRetentionPeriod
}

// write the user to the db
tenant, err := t.config.APIRepository.Tenant().CreateTenant(createOpts)

Expand Down
39 changes: 20 additions & 19 deletions frontend/docs/pages/self-hosting/_meta.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,21 +1,22 @@
{
"index": "Introduction",
"-- Docker": {
"type": "separator",
"title": "Docker"
},
"hatchet-lite": "Hatchet Lite",
"docker-compose": "Docker Compose",
"-- Kubernetes": {
"type": "separator",
"title": "Kubernetes"
},
"kubernetes-quickstart": "Quickstart",
"kubernetes-glasskube": "Installing with Glasskube",
"networking": "Networking",
"-- Managing Hatchet": {
"type": "separator",
"title": "Managing Hatchet"
},
"configuration-options": "Configuration Options"
"index": "Introduction",
"-- Docker": {
"type": "separator",
"title": "Docker"
},
"hatchet-lite": "Hatchet Lite",
"docker-compose": "Docker Compose",
"-- Kubernetes": {
"type": "separator",
"title": "Kubernetes"
},
"kubernetes-quickstart": "Quickstart",
"kubernetes-glasskube": "Installing with Glasskube",
"networking": "Networking",
"-- Managing Hatchet": {
"type": "separator",
"title": "Managing Hatchet"
},
"configuration-options": "Configuration Options",
"data-retention": "Data Retention"
}
106 changes: 76 additions & 30 deletions frontend/docs/pages/self-hosting/configuration-options.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
# Configuration Options

The Hatchet server and engine can be configured via `HATCHET_SERVER` environment variables. This document contains a list of all available options.

This document outlines the environment variables used to configure the server. These variables are grouped based on the configuration sections they belong to.
The Hatchet server and engine can be configured via `SERVER` and `DATABASE` environment variables. This document contains a list of all available options.

## Runtime Configuration

Expand All @@ -14,13 +12,59 @@ This document outlines the environment variables used to configure the server. T
| `SERVER_GRPC_BIND_ADDRESS` | GRPC server bind address | `127.0.0.1` |
| `SERVER_GRPC_BROADCAST_ADDRESS` | GRPC server broadcast address | `127.0.0.1:7070` |
| `SERVER_GRPC_INSECURE` | Controls if the GRPC server is insecure | `false` |
| `SERVER_WORKER_ENABLED` | Whether the internal worker is enabled | `false` |

## Services Configuration

| Variable | Description | Default Value |
| ----------------- | ------------------------ | ------------------------------------------------------------------------------------------------ |
| `SERVER_SERVICES` | List of enabled services | `["health", "ticker", "grpc", "eventscontroller", "queue", "webhookscontroller", "heartbeater"]` |
| `SERVER_SHUTDOWN_WAIT` | Shutdown wait duration | `20s` |
| `SERVER_ENFORCE_LIMITS` | Enforce tenant limits | `false` |
| `SERVER_ALLOW_SIGNUP` | Allow new tenant signups | `true` |
| `SERVER_ALLOW_INVITES` | Allow new invites | `true` |
| `SERVER_ALLOW_CREATE_TENANT` | Allow tenant creation | `true` |
| `SERVER_ALLOW_CHANGE_PASSWORD` | Allow password changes | `true` |

## Database Configuration

| Variable | Description | Default Value |
| ---------------------------- | ------------------------ | ------------- |
| `DATABASE_POSTGRES_HOST` | PostgreSQL host | `127.0.0.1` |
| `DATABASE_POSTGRES_PORT` | PostgreSQL port | `5431` |
| `DATABASE_POSTGRES_USERNAME` | PostgreSQL username | `hatchet` |
| `DATABASE_POSTGRES_PASSWORD` | PostgreSQL password | `hatchet` |
| `DATABASE_POSTGRES_DB_NAME` | PostgreSQL database name | `hatchet` |
| `DATABASE_POSTGRES_SSL_MODE` | PostgreSQL SSL mode | `disable` |
| `DATABASE_MAX_CONNS` | Max database connections | `5` |
| `DATABASE_LOG_QUERIES` | Log database queries | `false` |
| `CACHE_DURATION` | Cache duration | `60s` |

## Security Check Configuration

| Variable | Description | Default Value |
| -------------------------------- | ----------------------- | ------------------------------ |
| `SERVER_SECURITY_CHECK_ENABLED` | Enable security check | `true` |
| `SERVER_SECURITY_CHECK_ENDPOINT` | Security check endpoint | `https://security.hatchet.run` |

## Limit Configuration

| Variable | Description | Default Value |
| ------------------------------------------------ | -------------------------------- | ------------- |
| `SERVER_LIMITS_DEFAULT_TENANT_RETENTION_PERIOD` | Default tenant retention period | `720h` |
| `SERVER_LIMITS_DEFAULT_WORKFLOW_RUN_LIMIT` | Default workflow run limit | `1000` |
| `SERVER_LIMITS_DEFAULT_WORKFLOW_RUN_ALARM_LIMIT` | Default workflow run alarm limit | `750` |
| `SERVER_LIMITS_DEFAULT_WORKFLOW_RUN_WINDOW` | Default workflow run window | `24h` |
| `SERVER_LIMITS_DEFAULT_WORKER_LIMIT` | Default worker limit | `4` |
| `SERVER_LIMITS_DEFAULT_WORKER_ALARM_LIMIT` | Default worker alarm limit | `2` |
| `SERVER_LIMITS_DEFAULT_EVENT_LIMIT` | Default event limit | `1000` |
| `SERVER_LIMITS_DEFAULT_EVENT_ALARM_LIMIT` | Default event alarm limit | `750` |
| `SERVER_LIMITS_DEFAULT_EVENT_WINDOW` | Default event window | `24h` |
| `SERVER_LIMITS_DEFAULT_CRON_LIMIT` | Default cron limit | `5` |
| `SERVER_LIMITS_DEFAULT_CRON_ALARM_LIMIT` | Default cron alarm limit | `2` |
| `SERVER_LIMITS_DEFAULT_SCHEDULE_LIMIT` | Default schedule limit | `1000` |
| `SERVER_LIMITS_DEFAULT_SCHEDULE_ALARM_LIMIT` | Default schedule alarm limit | `750` |

## Alerting Configuration

| Variable | Description | Default Value |
| ------------------------------------ | -------------------------- | ------------- |
| `SERVER_ALERTING_SENTRY_ENABLED` | Enable Sentry for alerting | |
| `SERVER_ALERTING_SENTRY_DSN` | Sentry DSN | |
| `SERVER_ALERTING_SENTRY_ENVIRONMENT` | Sentry environment | `development` |

## Encryption Configuration

Expand Down Expand Up @@ -51,12 +95,17 @@ This document outlines the environment variables used to configure the server. T
| `SERVER_AUTH_GOOGLE_CLIENT_ID` | Google auth client ID | |
| `SERVER_AUTH_GOOGLE_CLIENT_SECRET` | Google auth client secret | |
| `SERVER_AUTH_GOOGLE_SCOPES` | Google auth scopes | `["openid", "profile", "email"]` |
| `SERVER_AUTH_GITHUB_ENABLED` | Whether GitHub auth is enabled | `false` |
| `SERVER_AUTH_GITHUB_CLIENT_ID` | GitHub auth client ID | |
| `SERVER_AUTH_GITHUB_CLIENT_SECRET` | GitHub auth client secret | |
| `SERVER_AUTH_GITHUB_SCOPES` | GitHub auth scopes | `["read:user", "user:email"]` |

## Task Queue Configuration

| Variable | Description | Default Value |
| ------------------------------- | ------------ | -------------------------------------- |
| `SERVER_TASKQUEUE_RABBITMQ_URL` | RabbitMQ URL | `amqp://user:password@localhost:5672/` |
| Variable | Description | Default Value |
| ------------------------------ | ------------------ | -------------------------------------- |
| `SERVER_MSGQUEUE_KIND` | Message queue kind | |
| `SERVER_MSGQUEUE_RABBITMQ_URL` | RabbitMQ URL | `amqp://user:password@localhost:5672/` |

## TLS Configuration

Expand All @@ -73,10 +122,12 @@ This document outlines the environment variables used to configure the server. T

## Logging Configuration

| Variable | Description | Default Value |
| ---------------------- | ------------- | ------------- |
| `SERVER_LOGGER_LEVEL` | Logger level | |
| `SERVER_LOGGER_FORMAT` | Logger format | |
| Variable | Description | Default Value |
| ------------------------ | ------------- | ------------- |
| `SERVER_LOGGER_LEVEL` | Logger level | |
| `SERVER_LOGGER_FORMAT` | Logger format | |
| `DATABASE_LOGGER_LEVEL` | Logger level | |
| `DATABASE_LOGGER_FORMAT` | Logger format | |

## OpenTelemetry Configuration

Expand All @@ -85,16 +136,11 @@ This document outlines the environment variables used to configure the server. T
| `SERVER_OTEL_SERVICE_NAME` | Service name for OpenTelemetry | |
| `SERVER_OTEL_COLLECTOR_URL` | Collector URL for OpenTelemetry | |

## Version Control System (VCS) Configuration

| Variable | Description | Default Value |
| -------------------------------------- | ----------------------------- | ------------- |
| `SERVER_VCS_KIND` | Type of VCS | |
| `SERVER_VCS_GITHUB_ENABLED` | Whether GitHub is enabled | |
| `SERVER_VCS_GITHUB_APP_CLIENT_ID` | GitHub app client ID | |
| `SERVER_VCS_GITHUB_APP_CLIENT_SECRET` | GitHub app client secret | |
| `SERVER_VCS_GITHUB_APP_NAME` | GitHub app name | |
| `SERVER_VCS_GITHUB_APP_WEBHOOK_SECRET` | GitHub app webhook secret | |
| `SERVER_VCS_GITHUB_APP_WEBHOOK_URL` | GitHub app webhook URL | |
| `SERVER_VCS_GITHUB_APP_ID` | GitHub app ID | |
| `SERVER_VCS_GITHUB_APP_SECRET_PATH` | Path to the GitHub app secret | |
## Tenant Alerting Configuration

| Variable | Description | Default Value |
| -------------------------------------------- | -------------------------------- | ---------------------- |
| `SERVER_TENANT_ALERTING_SLACK_ENABLED` | Enable Slack for tenant alerting | |
| `SERVER_TENANT_ALERTING_SLACK_CLIENT_ID` | Slack client ID | |
| `SERVER_TENANT_ALERTING_SLACK_CLIENT_SECRET` | Slack client secret | |
| `SERVER_TENANT_ALERTING_SLACK_SCOPES` | Slack scopes | `["incoming-webhook"]` |
9 changes: 9 additions & 0 deletions frontend/docs/pages/self-hosting/data-retention.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Data Retention

In Hatchet engine version `0.36.0` and above, you can configure the default data retention per tenant for workflow runs and events. The default value is set to 30 days, which means that all workflow runs which were created over 30 days ago and are in a final state (i.e. completed or failed), and all events which were created over 30 days ago, will be deleted.

This can be configured by setting the following environment variable to a Go duration string:

```sh
SERVER_LIMITS_DEFAULT_TENANT_RETENTION_PERIOD=720h # 30 days
```
2 changes: 1 addition & 1 deletion internal/services/controllers/jobs/controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -685,7 +685,7 @@ func (jc *JobsControllerImpl) runStepRunReassign(ctx context.Context, startedAt
err = g.Wait()

if err != nil {
jc.l.Err(err).Msg("could not run step run requeue")
jc.l.Err(err).Msg("could not run step run reassign")
}
}
}
Expand Down
24 changes: 24 additions & 0 deletions internal/services/controllers/workflows/controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,30 @@ func (wc *WorkflowsControllerImpl) Start() (func() error, error) {
return nil, fmt.Errorf("could not schedule get group key run reassign: %w", err)
}

_, err = wc.s.NewJob(
gocron.DurationJob(time.Second*60),
gocron.NewTask(
wc.runDeleteExpiredWorkflowRuns(ctx),
),
)

if err != nil {
cancel()
return nil, fmt.Errorf("could not delete expired workflow runs: %w", err)
}

_, err = wc.s.NewJob(
gocron.DurationJob(time.Second*60),
gocron.NewTask(
wc.runDeleteExpiredEvents(ctx),
),
)

if err != nil {
cancel()
return nil, fmt.Errorf("could not delete expired events: %w", err)
}

wc.s.Start()

f := func(task *msgqueue.Message) error {
Expand Down
Loading
Loading