v0.4.0

proTES now processes tasks with FOCA's plugin engine instead of its own
middlewares, on FOCA 0.19: plugins check or change tasks, choose the TES
instances and their order, and record the provenance of each task, which
users read through the API. The image ships the routing plugins of
foca-plugins (tes.routing.distance and generic.routing.random), so that,
as in 0.3.0, tasks go to the TES instance closest to their input data, with
random order as the fallback. The release is built from locked, hash-checked
dependencies, and removes forwarded_to from task logs. Several changes are
breaking: read the upgrade
steps
first. See the documentation for
details.

Breaking changes

• Plugins replace the middlewares: new tasks pass through a pipeline of FOCA
  plugins (plugins.pipeline), and custom.middlewares is refused at
  startup. The middleware management API and its collection are removed, and
  custom middlewares must be ported to plugins. Tasks that no TES instance
  accepts fail with 503 (was 500), requests that already passed proTES
  with 508 Loop Detected (if server.public_url or plugins.gateway_id
  is set), and tasks with tags under gateway. other than
  those of the clients' choices with 400 (see
  Plugins)
  (#333, #346; #315, #320; #316, #317)
• proTES runs on FOCA 0.19 (was 0.17): custom configs must apply the config
  changes of FOCA 0.18 and 0.19, and the image and the Helm chart of this
  release must be deployed together (see FOCA 0.18 and
  0.19)
  (#331, #338; #359, #360)
• TES instances are listed in custom.tes.backends (in the Helm chart,
  tes.backends) instead of custom.tes.service_list and
  custom.tes.discovery (see TES instance
  list)
  (#332, #340)
• API: logs[].metadata.forwarded_to is removed without a deprecation
  period, also from tasks stored by earlier versions; clients read the TES
  instance and the remote identifier from the gateway tags gateway.backend
  and gateway.backend_task_id. custom.storeLogs is refused at startup
  (see Forwards in task
  logs)
  (#324, #374)
• Access control: custom lists of security.access_control.policies must
  add the rules users-list-plugins, users-get-plugins,
  owners-get-provenance and admins-get-provenance, or users get 403
  (#322, #352; #218, #357)
• Database: proTES uses the new collections plugin_builds, pipelines,
  failed_runs and plugin_store, and creates indexes on startup: its
  database user needs the right createIndex (see Task
  provenance)
  (#334, #349)

Features

• The image ships the routing plugins tes.routing.distance and
  generic.routing.random of foca-plugins, pinned and hash-checked in
  uv.lock, and the default pipeline is distance routing with random order
  as the fallback again, as in 0.3.0. With the default settings, the hosts of
  inputs are looked up at DB-IP and Nominatim; set provider: offline and a
  GeoIP database, or FOCA's foca.routing.ordered, to avoid third parties
  (see
  Routing)
  (#379, #381)
• Task provenance: proTES stores the provenance of each task's plugin
  pipeline run and serves it with GET /tasks/{id}/provenance to the task's
  owner and to admins; new tasks have FOCA's gateway tags (gateway.*)
  that summarize it (#334, #349; #322, #352)
• Clients may choose the plugins and the TES instances of a task with the
  tags gateway.pipeline (if the admin enables it) and gateway.backends,
  and list what they may choose with GET /plugins; each choice applies to
  its task only (#218, #357)
• Helm chart: values for the plugins (plugins) and for the settings of the
  identity provider's issuer (#335, #348; #337, #341)

Security

• Requests to TES instances discovered in service registries go through
  FOCA's guarded HTTP client, which checks the address of each connection,
  so that a host whose addresses change after discovery can't lead requests
  with proTES's token to private addresses (#339, #347)
• .env files, PEM, key and PKCS#12 files are excluded from the Docker
  build context everywhere, also below pro_tes/ (#361, #367)
• The dev-image push job, which holds the Docker Hub credentials, now derives the version with build requirements installed from uv.lock with hash checks, exported by a digest-pinned uv image (#376, #378)

Fixes

• The forwarding of a task, with its state request, takes at most 55 seconds
  with the default settings (plugins.defaults.forward_timeout: 20, was
  FOCA's 60), within the 60 seconds of common proxies' read timeouts; other
  work, e.g., registry reads, token requests and database writes, comes on
  top. Tokens of service registries are no longer logged (#364, #369)
• Middlewares kept the order of the TES instances, and the changes of failed
  middlewares were undone, before the plugins replaced the middlewares
  (#314, #319; #313, #318)

Documentation

• The upgrade path from 0.3.0 (#362, #363, #368)
• A recipe for images derived from the image of proTES with other plugins,
  pinned by hash, tested in CI with a test plugin (#323, #355)
• The org-wide community files replace those of the repository, including
  the contributing guide (#325, #326; #370, #372)
• The README logo, the Zenodo DOI and the docs badge (#350, #351; #307,
  #309, #311)

Maintenance

• CI's lint, test, chart and docs jobs install the hash-checked versions from a new uv.lock (uv sync --locked), and Dependabot updates pyproject.toml and the lock together (#375, #377)
• CI passes in forks, uses the org's shared actions for the CI gate and for
  scanning images, checks pull request titles under the gate, and checks the
  chart before signing anything; workflow and job names are aligned and
  prepared for the repository name protes (#327, #328; #330, #342; #329,
  #343; #371, #373; #353, #354; #310, #312; #356, #358)
• Sourcery skips rule-test fixtures (#306)
• Dependabot updates of the pre-commit hooks and of the GitHub Actions (#344,
  #345)

Commits

• ci: let Sourcery skip rule-test fixtures by @uniqueg in #306
• docs: add the Zenodo DOI and badge by @uniqueg in #307
• docs: fix the docs badge URL by @uniqueg in #309
• docs: use Shields.io for the docs badge by @uniqueg in #311
• ci: align workflow and job names by @uniqueg in #312
• refactor: remove the middleware management API by @uniqueg in #317
• fix: keep the backend order in middlewares by @uniqueg in #319
• fix: undo changes of failed middlewares by @uniqueg in #318
• refactor!: pass tasks to middlewares, not requests by @uniqueg in #320
• docs: use the org-wide community files by @uniqueg in #326
• ci: make CI pass in forks by @uniqueg in #328
• refactor!: use FOCA 0.18 features by @uniqueg in #338
• refactor!: list TES backends with FOCA by @uniqueg in #340
• feat(chart): accept issuer settings by @uniqueg in #341
• ci: use the shared CI gate action by @uniqueg in #342
• ci: use the shared scan-image action by @uniqueg in #343
• refactor!: process tasks with FOCA plugins by @uniqueg in #346
• fix: check connections to discovered backends by @uniqueg in #347
• feat(chart): add plugin values by @uniqueg in #348
• feat: store the provenance of tasks by @uniqueg in #349
• docs: replace the README logo by @uniqueg in #351
• feat: serve task provenance by @uniqueg in #352
• docs: add a derived-image recipe for plugins by @uniqueg in #355
• ci: check chart before signing anything by @uniqueg in #354
• feat: select plugins and backends per task by @uniqueg in #357
• ci: prepare for the repository name protes by @uniqueg in #358
• build: use FOCA 0.19 by @uniqueg in #360
• fix: keep .env files out of the image build by @uniqueg in #367
• docs: add the upgrade path from 0.3.0 by @uniqueg in #368
• build(deps): bump the github-actions group across 1 directory with 2 updates by @dependabot[bot] in #345
• build(deps): bump the pre-commit group across 1 directory with 2 updates by @dependabot[bot] in #344
• fix: bound task creation, log registry tokens by @uniqueg in https://github.c...

v0.3.0

The first release, with the app image, the Helm chart and these notes
published together. It changes almost everything since the last deployed
version (images tagged by date, FOCA 0.12) and the legacy charts (2.x and
3.0.0): proTES now requires FOCA 0.17.0 (or a later 0.17 release) and Python
3.13 or later, requires access tokens, gives each task to the user who
created it, calls TES instances with credentials of its own, can discover
them in a service registry, forwards each task exactly once, tracks tasks
without blocking workers, and has a hardened chart and image. Many of the
changes are breaking: read the upgrade
steps first. See the
documentation for details.

Known limitations of this release:

• per-user limits on stored or unfinished tasks are not available yet
  (#285); limit requests at the ingress controller (see "Rate limits" on the
  Configuration page of the documentation);
• finished tasks are kept until they are deleted by hand: retention is not
  available yet (#286);
• each task adds two owner rules to the access control database, and FOCA
  0.17 loads all rules for each request, so requests slow down as the number
  of tasks grows; FOCA 0.18 will cache the policy enforcer;
• once CANCELED is stored, the resume sweep of the workers ignores the
  task: if its tracking is lost afterwards, its final logs may be missing
  (see "Task tracking" on the Operations page of the documentation).

Breaking changes

• API: all TES endpoints except GET /service-info require an access token
  of a trusted issuer (security.auth.issuers) for a configured audience
  (security.auth.audience), and a role from the groups of the identity
  provider (security.access_control.claim_groups, group_mapping);
  before, anyone could create, list, read and cancel all tasks. HTTP Basic
  credentials of callers are no longer passed through to TES instances (see
  Authentication)
  (#212, #240)
• API: bearer tokens must be access tokens as per RFC 9068 (typ header
  at+jwt); other tokens, such as ID tokens, are rejected (401). The new
  option security.auth.clients limits the accepted tokens to those of
  certain clients (see
  Authentication)
  (#214, #245)
• API: each task belongs to the user who created it: users list, read and
  cancel only their own tasks, and admins can no longer see tasks; they only
  manage the access control rules. Tasks of others and unknown tasks get
  403 (was 404 for unknown tasks). The rule users-get-cancel-tasks is
  replaced by owners-get-cancel-tasks, and the access control model has
  owner roles and no admin bypass. Tasks stored by earlier versions have no
  owner, so nobody sees them until operators give them to their users (see
  Task
  ownership)
  (#153, #215, #250)
• API: errors are problem details (title, status, detail, sent as
  application/json) instead of objects with message and code; request
  bodies are limited to 1 MiB (413) (see
  Errors)
  (#203, #219)
• API: times of tasks (creation_time, and start_time and end_time of
  the logs) are UTC RFC 3339 date-times instead of local-time strings;
  proTES sets creation_time and rejects tasks that set it (400), as the
  TES specification marks it read-only (see Task times and remote
  identifiers)
  (#208, #238)
• API: GET /tasks returns an empty next_page_token on the last page (it
  returned the last task's identifier, so clients kept paging) and an empty
  page if no task matches; invalid page tokens and page sizes are rejected
  (400); pages have 256 tasks by default (was 5) and at most 2047;
  name_prefix is taken literally, not as a regular expression (see Task
  listing)
  (#165, #208, #238)
• API: pages of the view FULL have at most 64 tasks
  (custom.controllers.list_tasks.max_page_size_full), also by default, so
  clients that list full tasks must follow next_page_token; inline
  contents of inputs (inputs[].content) over 128 KiB
  (custom.controllers.post_task.max_inline_content) are rejected with
  413 Content Too Large (see
  Task listing)
  (#284, #292)
• API: tasks that no TES instance accepts (500), and tasks that fail
  validation after the middlewares (400), are deleted with their owner's
  roles instead of being kept as SYSTEM_ERROR (see Stored
  tasks)
  (#284, #292)
• API: proTES sends each task to a TES instance with a single request, to
  <url>/ga4gh/tes/v1 only (earlier versions also tried <url>/v1 and
  <url>), without following redirects, and no longer fails over to the
  next path or TES instance when the outcome is unknown (read timeouts,
  connections lost after sending, 5xx): such tasks fail with
  SYSTEM_ERROR and 502 Bad Gateway, as the TES instance may run them
  (see TES instances and their
  credentials)
  (#274, #282)
• API: unfinished tasks whose identifier at their TES instance is unknown
  can't be canceled (409 Conflict; was 500, or 200 without effect for
  tasks in state UNKNOWN); tasks in state UNKNOWN are now canceled at
  their TES instance. New tasks are stored with the state that their TES
  instance reports, instead of UNKNOWN (see
  Task times and remote identifiers)
  (#275, #291)
• API: tasks with credentials in the URLs of their inputs or outputs
  (https://user:password@host/...) are rejected (400), for all TES
  instances; custom.controllers.post_task.url_credentials: strip removes
  the credentials instead (see
  TES instances and their credentials)
  (#213, #251)
• API: logs[].metadata.forwarded_to is a list of forwards (id, url,
  forward_time) instead of a nested tesNextTes object; tasks stored by
  earlier versions are served with their forwards as lists (see
  Stored tasks)
  (#173, #219)
• API: the middleware management API is no longer registered, as its
  controllers don't exist yet (see
  Middleware management API)
  (#196; #203, #219)
• Configuration: custom configs must apply the config changes of FOCA 0.14
  to 0.17 (e.g., exceptions.mapping; see the FOCA release notes and
  migration guides). The API specification (tes) and the access control
  rules are named entries, which overlays change by name; the service info
  is configured in service_info.service and no longer stored in the
  database (see
  Configuration)
  (#203, #219; #214, #245)
• Configuration: the identity provider is deployment configuration, without
  defaults: proTES refuses to start without trusted issuers, an audience and
  a group of the role admin (unless
  custom.access_control.require_admin is false), without enforced access
  control, or with service clients, which it doesn't support yet (see
  Authentication)
  (#212,
  #240; #215, #250)
• Configuration: unknown keys in custom are errors: the app and the Celery
  worker refuse to start, naming them, instead of ignoring them. Remove the
  keys that had no effect from config overlays: custom.celery,
  custom.controllers.celery and custom.controllers.post_task.timeout
  (post, poll, job); requests to TES instances time out after
  custom.tes.timeout seconds instead (default 15, was a hard-coded 5)
  (see
  Configuration)
  (#276, #281; #274, #282)
• Configuration: entries of custom.tes.service_list are objects with the
  URL of the TES instance (url) and how proTES authenticates at it
  (auth: client_credentials by default, basic or none), instead of
  URLs; with client_credentials, the app and the Celery worker need
  PROTES_CLIENT_ID and PROTES_CLIENT_SECRET and refuse to start without
  them. URLs of TES instances must be https URLs without credentials
  (unless custom.tes.url_schemes allows http); tasks are only forwarded
  to configured (or discovered) TES instances. Custom middlewares that read
  custom.tes.service_list get these entries instead of URLs (see
  TES instances and their credentials)
  (#213, #251)
• Configuration: the base config lists no TES instances (it listed public
  ones), and the default service info describes the app with placeholders
  on reserved example domains (RFC 2606; id is org.example.protes);
  deployments set their own, and the app warns on startup while the
  placeholders are set (see
  TES instances and their credentials)
  (#229, #251, #263; #205, #220)
• Configuration: custom.controllers.post_task.polling.wait and attempts
  must be positive, and wait and the new max_wait below 1800 seconds;
  custom.controllers.list_tasks.default_page_size must not exceed
  max_page_size (see
  [Task tr...