Repository navigation
A Note From The Maintainer
This release contains many changes. The goal was to modernize many of the components in this repo to the year 2026. The scheduler got a massive overhaul to address many long standing issues. TES API support has not changed at all. As part of this release a pass was taken on the https://github.com/calypr/funnel/issues list and around 90 issues were addressed. This release is feature rich. If you have any issues with this release at all feel free to create a new issue in https://github.com/calypr/funnel/issues
Funnel v0.12.3
Funnel can now run GPU tasks, upload outputs over HTTP, save structured events to files, and advertise the backend settings a deployment accepts. Kubernetes deployments gain native storage configuration and mounted S3 routes.
This release also changes task scheduling and recovery, strengthens access to files and credentials, and separates the public Go client and plugin APIs from the server's implementation.
These notes cover changes since the last published stable release, v0.11.14.
Existing deployments need an upgrade plan. This release includes configuration changes, database migration requirements, and Go import changes. Read the upgrade guide before replacing a running server.
Before you upgrade
- Stop older Funnel servers, schedulers, and nodes before starting the new versions. Upgrade the participating components and authorization plugins together. Older schedulers do not understand the new task ownership records.
- Back up the task database, configuration, custom templates, and batch-scheduler submission records. Check the upgrade and rollback using a restored copy.
- If you use standalone MongoDB, convert it to a replica set or sharded cluster. Changing the connection string alone is not enough. A single-node replica set is sufficient for development.
- If your Kubernetes Jobs contain storage plans with version 1, 2, or 3, finish those tasks and clean up their Jobs with a compatible older Funnel binary first.
- Remove the
Elasticconfiguration section, even if you do not select Elasticsearch as your database. Elasticsearch support has been removed, and the configuration parser rejects that field. - For HPC batch systems such as Slurm or PBS, cancel jobs submitted without trusted submission records using the scheduler's own command before upgrading.
- If you build Go applications or plugins against Funnel, update their imports and rebuild them. Old net/rpc plugin binaries cannot be mixed with the new host.
The guide also covers retired Kubernetes settings, FTP configuration, and the private state directory now required by HPC backends.
GPU tasks and backend-specific settings
Funnel can now pass GPU requests to Kubernetes, AWS Batch, GCP Batch, local Docker, and Slurm deployments configured for GPU jobs. This includes making the assigned devices available inside the task container, not only asking the scheduler for them.
Tasks put these settings in TES resources.backend_parameters. For example, a local Docker or Kubernetes task can request one GPU with:
{
"resources": {
"backend_parameters_strict": true,
"backend_parameters": {
"gpu_count": "1"
}
}
}Use backend_parameters_strict: true when the task depends on these settings. Funnel rejects unsupported required parameters. Unsupported optional parameters produce a server warning and are removed from the stored task. Parameter names are case-insensitive, but sending the same name twice with different casing is an error.
GET /v1/service-info reports the deployment's supported keys in tesResources_backend_parameters. Clients can check what a server accepts before submitting work.
How each backend handles GPUs
- Kubernetes:
gpu_countsets the GPU request and limit on the executor container.gpu_resourceselects the device resource and defaults tonvidia.com/gpu.gpu_node_selectorselects administrator-defined node labels, such asaccelerator=a100. Worker and sidecar containers do not receive the executor's GPU allocation. Conflicting selectors or GPU settings in a custom executor template are rejected. - AWS Batch:
gpu_countrequests GPU resources from Batch. Tasks can also selectjob_queueandjob_definition, subject to the server's AWS permissions. Funnel forwards the exact devices Batch assigned to the worker throughNVIDIA_VISIBLE_DEVICES; it does not pick different host GPUs. The queue's compute environments determine which hardware is available. Naming a queue after a GPU model does not itself enforce that model. - GCP Batch: request
gpu_typeandgpu_counttogether. Optionalmachine_typeselects a compatible VM, andinstall_gpu_driversdefaults totruefor GPU jobs. Funnel passes the accelerator settings to GCP and enables GPU access for each executor container. GCP validates the hardware, machine type, region, and quota combination. - Slurm: operators can allow native submission parameters such as
gresandpartitionthroughSlurm.BackendParameters. Each value must match its configured regular expression, and the submission template must use it. A site allocation hook suppliesFUNNEL_GPU_DEVICESwith the allocated physical GPU IDs or UUIDs for Docker. Slurm's task-local device numbering must not be copied blindly into that variable. - Local Docker: tasks can request a count through
gpu_countor specific physical GPU indices or UUIDs throughgpu_devices. When both are supplied, the count must match the device list. Local execution does not reserve GPUs between concurrent tasks, so operators must manage sharing or exclusive access.
Set up the deployment first
The GPU drivers, device plugins, container toolkit, and compatible images still need to exist on the deployment. Funnel does not install a GPU stack on a local host or Kubernetes cluster. Provider-specific driver installation settings apply to GCP Batch as documented.
Custom Docker run templates must include {{.GPUArgs}}. Existing AWS Batch worker definitions must start the worker with --Compute aws-batch. Slurm workers must load the Slurm backend, and their allocation hook must provide the physical device allocation.
PBS, Grid Engine, HTCondor, and Funnel's manual scheduler do not support these GPU backend parameters. Use strict parameters to reject a task that requires unsupported GPU settings rather than silently losing its request.
The tests check request validation, provider requests, and forwarding allocated device IDs. They do not prove that a deployment has usable GPU hardware. Run a task such as nvidia-smi -L on the deployment and check the visible devices before relying on it for GPU workloads.
See the GPU task guide, AWS Batch GPU setup, and GCP Batch GPU setup.
Scheduling and recovery
Task ownership and resource accounting are stored together
Funnel now records which node owns a task and how much CPU and memory that task reserved in one database transaction. A failed assignment cannot leave only half of that change committed.
The scheduler improvements cover BoltDB, PostgreSQL, and MongoDB:
- Competing schedulers cannot both reserve the same task through the assignment path.
- A completed, failed, or canceled task releases the resources reserved when it was assigned. Repeated completion messages do not release those resources twice.
- Routine node heartbeats no longer rewrite task reservations or repeatedly block scheduling.
- The scheduler scans queued work in limited batches, so a task that cannot currently run does not hide runnable work farther down the queue.
- Node resource totals and assigned-task lists are read consistently. Idle nodes recover their available capacity after the last task releases its reservation, and startup repairs stale idle usage left by earlier accounting bugs.
- Node drain requests are published immediately. Work already running can continue while the node stops accepting new work.
At the first startup after upgrading, Funnel imports existing active assignments into the new storage format. Conflicting owners or inconsistent reservations stop startup so an operator can investigate. Do not run an older scheduler alongside the upgraded one to work around a migration error.
Worker failures leave clearer task results
A worker claims a task before preparing files or launching an executor. Duplicate workers observe the existing task instead of launching another copy through that claim path. Kubernetes retains the configured Job retry limit for failures before a worker claims the task. A failed worker container can be retired for retry even when a sidecar keeps the pod alive.
Tasks remain INITIALIZING while the worker prepares them. The worker reports RUNNING just before dispatching the first executor. For Kubernetes tasks with a recorded owner pod, Funnel detects when that owner exits, disappears, or is replaced and records SYSTEM_ERROR rather than replaying the task under the same ID.
Funnel also retains failed Kubernetes Jobs until it has recorded the task result and diagnostics. Failed database writes and submission timeouts have recovery paths instead of silently losing the failure report.
This is not an exactly-once execution guarantee. A canceled or unreachable task may still have a process running outside Funnel's reach. Lost execution claims do not automatically move to another worker. Submit a new task when a failed task needs to be retried.
Large node lists are paginated
GET /v1/nodes returns up to 500 nodes by default. Clients can request page_size values from 1 through 1000 and follow next_page_token using page_token.
Funnel's scheduler, metrics, and dashboard follow the pages. External clients that make one request must be updated if they need the complete node list. Older gRPC node clients need updated generated bindings to use the pagination fields.
See scheduler ownership and upgrades.
Kubernetes storage
Configure storage using Kubernetes objects
Kubernetes.Storage supports existing persistent volume claims, claims provisioned through a StorageClass, and explicit persistent volume/claim pairs. A persistent volume claim, or PVC, is Kubernetes' request for persistent storage.
Funnel applies the resolved workspace storage to the worker and executor pods. It supports selecting the intended containers when a Job template has sidecars. A PVC template with the logical name workspace produces a claim named <taskID>-workspace.
The submitted Job records its storage plan. Later cleanup uses that plan instead of guessing from a configuration file that may have changed. Funnel checks resource ownership and preserves operator-owned claims. Invalid or conflicting cleanup records are reported instead of guessing which storage to delete. Interrupted provisioning can still leave a persistent volume requiring cleanup; creation and cancellation are not one atomic operation. It also checks storage permissions before creating task Jobs and reports storage errors before the task reaches its final state.
Cleanup now honors DisableJobCleanup consistently. Startup backlog cleanup runs in the background instead of making server startup wait for it. Missing-resource races during cancellation no longer turn an already removed Kubernetes resource into an HTTP 500. Recovery also distinguishes a lost worker Job from a Job still being created.
Existing PVTemplate, PVCTemplate, and JobTemplate configurations remain a fallback when native Storage is absent. Their presence alone does not require conversion. Configurations using retired settings do require migration.
See Kubernetes storage configuration and migration.
Read and write selected S3 paths through mounted storage
Kubernetes workers can route selected S3 inputs and outputs through configured filesystem mounts. Tasks still use ordinary S3 URLs. Mounted routes are recorded when the Job is submitted, so a later worker configuration change does not silently reroute an existing task. Operators choose the matching endpoint, bucket, prefix, and volume in Funnel's configuration.
For mounted inputs, Funnel copies files into the task workspace. Writing to a staged input does not change the mounted source through that copy. Prefix matching respects directory boundaries, and Funnel rejects paths that escape the selected mount.
For mounted outputs, Funnel writes a temporary file and replaces the destination after the copy finishes. It then waits until S3 reports an object with the expected size before reporting successful publication. Cancellation stops that wait. Outputs outside the selected route use the normal storage API path.
The remote check compares size, not contents. An old object with the same size can satisfy it. This does not provide checksum or object-version verification, and the storage driver's later synchronization to S3 has its own behavior and limits.
Retired settings include S3FilesFilesystemId, PVCMode, StorageClassName, SharedPVCName, Bindings, and MountedInputs. Mounted read/write settings now select a Volume and, when necessary, a SubPath. Use the migration instructions rather than renaming fields without checking their meaning.
Older Chicago binary configurations need to be exported with their original schema before conversion. Their old filesystem-ID field number conflicts with a different setting in the canonical schema. See the upgrade guide before loading those files.
See mounted S3 inputs and outputs.
File transfers and task outputs
Upload outputs to HTTP and HTTPS URLs
Funnel can now send output files with HTTP PUT. This supports presigned upload URLs, so the service submitting a task can grant access to a specific object without giving the worker general storage credentials.
- Uploads send the file's length, including for empty files.
- Uploads follow 307 and 308 redirects and resend the file to the destination.
- Uploads reject 301, 302, and 303 redirects, which could otherwise turn a PUT into a GET and falsely look successful.
- File names containing spaces,
#,?, or%retain their intended names when Funnel builds output URLs. - Canceling a task stops its HTTP transfers.
A presigned URL normally covers one exact FILE output. Directory and wildcard outputs need a service that can authorize or sign each resulting file upload. By default, HTTP transfers have no overall transfer deadline; operators can configure one. Connection and response-header waits still have limits.
See HTTP storage and presigned URLs.
Safer S3 downloads and uploads
Failed source downloads no longer truncate an existing destination before a complete replacement is available. On ordinary Unix filesystems, Funnel stages the replacement and renames it into place. Mountpoint S3 uses a separate download path because it does not support the same filesystem operations; a failure after publication into that mount begins cannot be rolled back atomically.
S3 uploads return the successful upload's receipt and acknowledged size. Directory detection respects the slash boundary between keys, so a sibling such as results-old/file.txt does not make results look like a directory. Directory listings also retain endpoint-qualified URLs needed for safe staging.
S3 uploads can explicitly request AES256 server-side encryption. Conflicting encryption settings are rejected.
More predictable output collection
- Fixes prevent task volumes from being nested incorrectly or hiding paths the task needs. Sibling paths such as
/outand/out2are treated as separate paths. Kubernetes task validation also checks the configured protected system paths. The default Kubernetes list includes/dev,/proc,/sys,/run, and/var/run; operators can configureKubernetes.ForbiddenPathPrefixes. This setting is specific to Kubernetes. - Replacing a local file with a shorter or empty copy no longer leaves trailing bytes from the old contents.
- Nested outputs and stdout/stderr file outputs keep their intended paths.
- An explicit
FILEoutput pointing to a directory is rejected. An omitted type retains automatic detection. - Wildcard outputs retain literal file names across storage backends. A pattern matching no files produces a warning and is skipped.
- Available outputs can still be published when another output or preparation step fails. A successful partial upload does not turn the whole failed task into
COMPLETE. - Overlapping output paths reuse completed output copies instead of letting cleanup remove files needed by another output.
- Upload errors identify the task output path that failed.
Execution and cloud backends
- Default Docker execution provides a writable, executor-local
/tmpwhile retaining a read-only container root. An explicit task mount for/tmpor a parent path suppresses that temporary mount. Worker.Container.EnforceResourcescan enforce requested Docker CPU and memory limits, with swap disabled. Enforcement is optional.- Executors can read
stdinfrom the staged input file. Missing files and escaping symbolic links fail instead of reading an unintended host path. Worker.Container.EnableTagscontrols forwarding task tags as container labels. Enabled label values remain literal data, including spaces, quotes, and text resembling shell commands.- Cancellation handles races with executor completion and gives stop operations a time limit. Reported exit codes survive cancellation and Kubernetes log-stream failures more consistently.
- Kubernetes tasks can set the
_IMAGE_PULL_POLICYtag toAlways,IfNotPresent, orNever. Missing or empty values retainAlways. Custom executor templates must use{{.ImagePullPolicy}}to honor the setting. - Kubernetes tracking uses the selected executor container for results and logs. Sidecars or init containers do not determine the executor's exit result. Pod watches reconnect after server timeouts for long-running tasks.
- AWS Batch uses resource requirements for CPU and memory overrides and new job definitions. When available, EC2 instance type metadata is recorded through Amazon's instance metadata service.
- CPU, memory, and disk detection are handled independently, including on Apple Silicon hosts.
Event files, logs, and monitoring
Save structured events to a file
The new file event writer saves task events as newline-delimited JSON: one structured JSON event per line. This is separate from the human-readable diagnostic log.
EventWriters:
- log
- file
EventFile: /var/log/funnel/events.ndjsonThe parent directory must already exist and be writable. Funnel appends to an existing file and creates new files with permissions 0600. --EventFile can also set the path.
Give each process its own event file. Funnel coordinates writes within one writer, not between processes sharing a file, and does not rotate the files automatically. Event selection and log filtering still apply. The file format is useful for inspection and processing, but is not a database backup or an automatic replay system.
More reliable delivery and shutdown
- A fix prevents fast executor output from changing earlier log chunks when an output buffer is reused.
- Log-tail delivery retries transient write failures and stops retrying after persistent failures. Cancellation releases blocked log writers instead of leaving them waiting on a stopped consumer.
- Prometheus counters
funnel_logging_events_totalandfunnel_logging_events_dropped_totalreport processed and dropped stdout/stderr log events. They give operators a way to spot log-delivery pressure. - The Pub/Sub event reader reports decoding, destination-write, and acknowledgment errors. It does not acknowledge a message whose destination write failed. Pub/Sub publishing remains asynchronous.
- Server shutdown waits for task submissions and closes owned services. Listener startup failures are returned instead of leaving the other listener running unnoticed. Event pipelines and their background routines are closed more consistently.
Security and authentication
This release fixes access-control, file-access, credential-handling, and runtime-argument problems. The changes include:
- Preserve the caller's identity through task reads and event writes, and reject duplicate task replacement.
- Restrict local storage to configured roots and reject output and executor-log paths that escape their task workspace, including escaping symbolic links.
- Copy staged inputs without widening the source file's permissions.
- Keep task-supplied environment and tag values from becoming extra options to the container or batch runtime.
- Tie HPC cancellation to trusted submission records and cloud cancellation to the authorized task.
- Bind configured FTP credentials to the intended endpoint.
- Validate OIDC login state, token audience, and expiry, and strengthen request handling at the HTTP gateway.
- Wait for the initial OIDC signing keys before validating tokens, avoiding failures caused by the first key fetch still being in progress.
- Restrict authenticated dashboard requests to supported same-origin API routes, encode task and node identifiers, and reject redirects that could send credentials elsewhere.
- Reject named pipes and unsupported special files as file-transfer endpoints.
- Keep panic recovery around the full API interceptor chain, including authentication and audit handling. Executor panic handling and internal task/build diagnostics were improved without exposing raw panic values to API callers.
API audit records include the method, user, status, and task ID when available. They omit request bodies, tokens, raw errors, and panic values. Configuration diagnostics redact sensitive values.
The HTTP task client and CLI support bearer tokens through FUNNEL_SERVER_TOKEN. A bearer token takes precedence over Basic credentials. Existing username/password environment variables remain supported.
Authenticated remote plaintext connections are rejected by default. Use HTTPS or a secure tunnel. Trusted deployments that deliberately permit plaintext can set FUNNEL_ALLOW_INSECURE_CREDENTIALS=true in the connecting processes. This opt-in also permits authenticated HTTPS-to-HTTP redirects. Loopback HTTP remains supported.
OIDC deployments must set Server.OidcAuth.RequireAudience to the audience expected by their API. Tokens must contain that audience and an expiry. See OIDC authentication configuration.
HPC deployments must configure an absolute, persistent HPCStateDir with mode 0700. Keep it outside task work directories and enabled local-storage roots, and back it up with the task database. FTP deployments using configured credentials must set FTPStorage.Endpoint.
Custom runtime templates that use index .Env or index .Tags need updating. Use .GetEnvArgs, .GetTagArgs, or the supported range and eq patterns described in the deployment guide.
See the deployment upgrade instructions and remote server connection guide.
Public Go packages and authorization plugins
The public integration packages now have specific jobs. The TES client is separate from the task models, and server implementation packages live under internal/.
Paths below use github.com/calypr/funnel as the module prefix:
| Previous import or use | New location |
|---|---|
tes request, response, and enum types |
api/tes |
tes.Client and tes.NewClient |
client/tes |
Configuration schema types in config |
api/config |
| Logger configuration schema types | api/logger |
plugins/proto and plugins/shared contracts |
api/plugin |
| Build version information | version |
External Go applications cannot import the implementation packages under internal/. The public config and logger packages contain schema types, not the old application defaults, YAML loaders, runtime logger, or plugin process manager. Old imports do not have forwarding packages.
The REST client accepts a caller-supplied HTTP client through WithHTTPClient, including caller-chosen timeouts. It preserves a server URL prefix and escaped task identifiers, which matters when Funnel sits behind a reverse proxy at a subpath.
The former TES marshaler globals and helpers are removed. Go consumers can use protobuf's protojson functions. Generated server gRPC and gateway bindings are internal; direct gRPC integrations can generate bindings in their own module.
Rebuild net/rpc plugins with the host
The net/rpc transport now carries complete protobuf messages inside its encoding. This preserves explicit zero values and optional fields, including an output explicitly declared as FILE. Previously that value could disappear and look unspecified.
The encoding differs from older net/rpc binaries. Rebuild and deploy the host and plugins together. Messages exchanged over gRPC keep the same format, although Go import changes still apply.
Plugins can clear inherited settings
Plugin configuration can now replace an inherited setting with false, zero, an empty string, or an empty collection.
A plugin must return the complete edited host configuration. Returning a small partial configuration can erase required settings. Clone the supplied configuration, change the fields the plugin owns, and return that copy. Kubernetes plugin overrides cannot change the backend's Jobs namespace.
See the Go client and package guide and plugin migration guide.
Task API limits and service discovery
Task-list pages now account for response size as well as the requested number of tasks. A page can contain fewer tasks than requested because the next task would exceed the byte limit. Always follow next_page_token until it is empty. Funnel does not truncate a task to squeeze it into a page.
The default gRPC response limit is 4 MiB. Server.MaxResponseBytes changes it, and 0 selects the default. The setting also configures Funnel's HTTP-to-gRPC gateway. External gRPC clients need a compatible receive limit of their own.
If a single task is too large, Funnel returns RESOURCE_EXHAUSTED with the task ID and suggests MINIMAL or a larger response limit. Requesting smaller pages cannot shrink an individual task. This is a protobuf transport limit, not a guarantee about JSON size or server memory use.
PostgreSQL and MongoDB continuation tokens use creation time and task ID for a stable order. New tokens can resume even if the last returned task was deleted. Older ID-only tokens remain accepted while their referenced task still exists.
/v1/service-info remains public when authentication is enabled, including when a request supplies missing or invalid credentials. It reports the Funnel build version separately from the TES API version, which is 1.1.0, and lists supported backend parameters. Task endpoints still require the configured authentication.
Task read and cancellation errors now map missing tasks and denied access to the appropriate API status instead of reporting every failure as an internal server error.
See task response sizes.
Database compatibility and fixes
- MongoDB retains authentication settings supplied through its connection URI instead of overwriting them with empty separate settings.
- Older MongoDB tasks whose output sizes were stored as numbers remain readable alongside current records. This covers task get/list requests and their BASIC and FULL views.
- MongoDB task-name prefix filters treat punctuation as literal text instead of interpreting it as a regular expression.
- PostgreSQL task-event persistence and task-view handling were corrected. Events and requested views remain consistent through the API path.
- DynamoDB validates stored execution-log indexes and returns an error for invalid values instead of using them as unchecked indexes.
The MongoDB transaction requirement and Elasticsearch removal still require the deployment changes described above. These compatibility fixes do not remove those migration requirements.
CLI, dashboard, and configuration
FUNNEL_CONFIGsupplies the default configuration file path. An explicit--configwins.FUNNEL_LOG_LEVELoverrides the configured log level. An explicit--Logger.Levelwins.FUNNEL_SERVER,FUNNEL_SERVER_USER,FUNNEL_SERVER_PASSWORD,FUNNEL_SERVER_TOKEN, andFUNNEL_ALLOW_INSECURE_CREDENTIALSremain supported for their documented client and CLI uses.funnel debugprints version information and redacted configuration for troubleshooting.- Malformed task JSON reports a line and column. Task connection errors include a server-address hint and clearer HTTP error descriptions.
- Freshly created tasks without logs can be fetched and listed safely. Executors that never ran no longer produce placeholder execution logs.
- Task get/list requests apply
MINIMAL,BASIC, andFULLviews consistently, including with PostgreSQL. Invalid or repeated HTTP view parameters return 400. funnel task list --allprints pages as they arrive while preserving JSON output.- Web task details refresh while the task is active and show executor details even when a failed task has missing or empty execution logs.
- The dashboard shows visible authentication/access errors, preserves multiline details, and handles quoted log values and equals signs correctly.
- Active node displays include available and total CPU, memory, and disk.
- Database initialization logs make startup waits visible. PostgreSQL's existing five-minute initialization budget now cancels stalled network operations.
- Swift errors distinguish an initial authentication failure from a bucket lookup failure. RPC credential examples now place credentials under
RPCClient.Credential. - Backend and worker configuration is checked at startup. Missing GCP project/location values, malformed HPC templates, invalid reconciliation intervals, and missing or nonpositive worker polling intervals fail with the relevant configuration field instead of failing later during work. A disabled reconciler does not require a reconciliation interval.
- A leading
~inWorker.WorkDiris rejected with an explanation instead of being treated as an expanded home directory.
funnel dashboard, the terminal dashboard, also has broader updates:
- Task sorting, filtering, expanded details, and page navigation were refreshed.
- Node and recorded-worker views show where work is running, including recorded workers when no manual scheduler is available.
- Task cancellation asks for confirmation and shows cancellation errors.
- Paging no longer loses queued navigation when a refresh overlaps it. A failed page request keeps the current page and continuation state.
- The dashboard can recover after the Funnel server restarts.
The task model still cannot distinguish every missing executor exit code from zero. Use the task's state to decide whether work completed successfully, rather than interpreting a displayed zero alone as success.
Removed support and deployment files
Elasticsearch is no longer a database or event-writer backend. Existing deployments must migrate their data to a supported backend and remove the Elastic section from configuration before starting this version.
Bundled GCE provisioning scripts, VM installers, and systemd service examples have been removed. Operators need to maintain their infrastructure setup and service definitions. Runtime compute backends and container images remain available.
Badger remains available as an embedded task database. It does not implement the scheduler service required by the manual compute backend. Its new database guide explains configuration, backups, and that limitation.
Development, testing, and releases
- Handwritten documentation lives in
docs/; Hugo's website configuration remains inwebsite/. - Protocol generation is ordered and repeatable, and dependency setup initializes the required schema submodule.
- Protocol definitions and Buf generation configuration live under
api/. Usemake prototo generate code. Direct proto import paths changed even where message fields and service identities stayed the same. - Integration and end-to-end tests are organized by purpose. Generated databases and work directories are removed by default.
-funnel-keep-artifactsretains files for debugging while still stopping services and closing handles. - Duplicate legacy integration workflows were removed, and automated dead-code checks were added.
- Integration fixtures use pinned versions and verify that Kubernetes and Gen3 tests run the candidate image. Setup uses limited retries for dependency and image downloads, with retained diagnostics for failures.
- CI covers scheduler behavior and crash recovery across BoltDB, PostgreSQL, and MongoDB, plus Docker execution, Kubernetes storage, Nextflow, authentication, and Gen3 workflows.
- AWS integrations moved to the AWS Go SDK v2, with related Google Cloud, MongoDB, container, logging, and other dependency updates.
- Dashboard dependencies and build tooling were updated, including the move to Vite. The source build requires Go 1.27.3, as declared in
go.mod. Prebuilt binaries do not require a Go installation. - Release builds use a pinned GoReleaser executable. Archives include the MIT license and upgrade guide.
- Packaging validation is available as a manual workflow. It does not run as a normal PR job. The publishing workflow supports manual dry runs and verifies that the selected tag matches its checkout.
- Container publications include branch/version tags and tags containing publication time and commit identity. Stable version tags update
latest; prerelease tags do not. - The installer supports destination paths containing spaces, extracts into a temporary directory, and reports HTTP download failures with a limited number of retries.
The merged release source passed all 28 post-merge checks before the tag was pushed. Manual packaging validation also passed. A local test of the packaged binary started a server, submitted a task through the CLI, ran its Docker executor, and verified the completed task's published output.
Install v0.12.3
Install the binary with the version's own installer:
curl -fsSL https://raw.githubusercontent.com/calypr/funnel/v0.12.3/install.sh | bash -s -- v0.12.3The default destination is $HOME/.local/bin. Archives and SHA-256 checksums are available for Linux and macOS on amd64 and arm64 from this release's assets.
The container image is:
docker pull quay.io/ohsu-comp-bio/funnel:v0.12.3Before starting an existing deployment, complete the upgrade steps.
Full change list: v0.11.14...v0.12.3.