Skip to content

v0.3.0

Pre-release
Pre-release

Choose a tag to compare

View all tags
@github-actions github-actions released this 24 Oct 13:55
· 46 commits to dev since this release
v0.3.0
This tag was signed with the committer’s verified signature.
uniqueg Alex Kanitz
SSH Key Fingerprint: 9NjZS21iWrTwyzKfdaTSze0mbHhsu+9cNOFOXtblYdk
Verified
Learn about vigilant mode.
8af9c36
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

The first release, with the app image, the Helm chart and these notes
published together. It changes almost everything since the last deployed
version (images tagged by date, FOCA 0.12) and the legacy charts (2.x and
3.0.0): proTES now requires FOCA 0.17.0 (or a later 0.17 release) and Python
3.13 or later, requires access tokens, gives each task to the user who
created it, calls TES instances with credentials of its own, can discover
them in a service registry, forwards each task exactly once, tracks tasks
without blocking workers, and has a hardened chart and image. Many of the
changes are breaking: read the upgrade
steps first. See the
documentation for details.

Known limitations of this release:

  • per-user limits on stored or unfinished tasks are not available yet
    (#285); limit requests at the ingress controller (see "Rate limits" on the
    Configuration page of the documentation);
  • finished tasks are kept until they are deleted by hand: retention is not
    available yet (#286);
  • each task adds two owner rules to the access control database, and FOCA
    0.17 loads all rules for each request, so requests slow down as the number
    of tasks grows; FOCA 0.18 will cache the policy enforcer;
  • once CANCELED is stored, the resume sweep of the workers ignores the
    task: if its tracking is lost afterwards, its final logs may be missing
    (see "Task tracking" on the Operations page of the documentation).

Breaking changes

  • API: all TES endpoints except GET /service-info require an access token
    of a trusted issuer (security.auth.issuers) for a configured audience
    (security.auth.audience), and a role from the groups of the identity
    provider (security.access_control.claim_groups, group_mapping);
    before, anyone could create, list, read and cancel all tasks. HTTP Basic
    credentials of callers are no longer passed through to TES instances (see
    Authentication)
    (#212, #240)
  • API: bearer tokens must be access tokens as per RFC 9068 (typ header
    at+jwt); other tokens, such as ID tokens, are rejected (401). The new
    option security.auth.clients limits the accepted tokens to those of
    certain clients (see
    Authentication)
    (#214, #245)
  • API: each task belongs to the user who created it: users list, read and
    cancel only their own tasks, and admins can no longer see tasks; they only
    manage the access control rules. Tasks of others and unknown tasks get
    403 (was 404 for unknown tasks). The rule users-get-cancel-tasks is
    replaced by owners-get-cancel-tasks, and the access control model has
    owner roles and no admin bypass. Tasks stored by earlier versions have no
    owner, so nobody sees them until operators give them to their users (see
    Task
    ownership    )
    (#153, #215, #250)
  • API: errors are problem details (title, status, detail, sent as
    application/json) instead of objects with message and code; request
    bodies are limited to 1 MiB (413) (see
    Errors)
    (#203, #219)
  • API: times of tasks (creation_time, and start_time and end_time of
    the logs) are UTC RFC 3339 date-times instead of local-time strings;
    proTES sets creation_time and rejects tasks that set it (400), as the
    TES specification marks it read-only (see Task times and remote
    identifiers    )
    (#208, #238)
  • API: GET /tasks returns an empty next_page_token on the last page (it
    returned the last task's identifier, so clients kept paging) and an empty
    page if no task matches; invalid page tokens and page sizes are rejected
    (400); pages have 256 tasks by default (was 5) and at most 2047;
    name_prefix is taken literally, not as a regular expression (see Task
    listing    )
    (#165, #208, #238)
  • API: pages of the view FULL have at most 64 tasks
    (custom.controllers.list_tasks.max_page_size_full), also by default, so
    clients that list full tasks must follow next_page_token; inline
    contents of inputs (inputs[].content) over 128 KiB
    (custom.controllers.post_task.max_inline_content) are rejected with
    413 Content Too Large (see
    Task listing)
    (#284, #292)
  • API: tasks that no TES instance accepts (500), and tasks that fail
    validation after the middlewares (400), are deleted with their owner's
    roles instead of being kept as SYSTEM_ERROR (see Stored
    tasks    )
    (#284, #292)
  • API: proTES sends each task to a TES instance with a single request, to
    <url>/ga4gh/tes/v1 only (earlier versions also tried <url>/v1 and
    <url>), without following redirects, and no longer fails over to the
    next path or TES instance when the outcome is unknown (read timeouts,
    connections lost after sending, 5xx): such tasks fail with
    SYSTEM_ERROR and 502 Bad Gateway, as the TES instance may run them
    (see TES instances and their
    credentials    )
    (#274, #282)
  • API: unfinished tasks whose identifier at their TES instance is unknown
    can't be canceled (409 Conflict; was 500, or 200 without effect for
    tasks in state UNKNOWN); tasks in state UNKNOWN are now canceled at
    their TES instance. New tasks are stored with the state that their TES
    instance reports, instead of UNKNOWN (see
    Task times and remote identifiers)
    (#275, #291)
  • API: tasks with credentials in the URLs of their inputs or outputs
    (https://user:password@host/...) are rejected (400), for all TES
    instances; custom.controllers.post_task.url_credentials: strip removes
    the credentials instead (see
    TES instances and their credentials)
    (#213, #251)
  • API: logs[].metadata.forwarded_to is a list of forwards (id, url,
    forward_time) instead of a nested tesNextTes object; tasks stored by
    earlier versions are served with their forwards as lists (see
    Stored tasks)
    (#173, #219)
  • API: the middleware management API is no longer registered, as its
    controllers don't exist yet (see
    Middleware management API)
    (#196; #203, #219)
  • Configuration: custom configs must apply the config changes of FOCA 0.14
    to 0.17 (e.g., exceptions.mapping; see the FOCA release notes and
    migration guides). The API specification (tes) and the access control
    rules are named entries, which overlays change by name; the service info
    is configured in service_info.service and no longer stored in the
    database (see
    Configuration)
    (#203, #219; #214, #245)
  • Configuration: the identity provider is deployment configuration, without
    defaults: proTES refuses to start without trusted issuers, an audience and
    a group of the role admin (unless
    custom.access_control.require_admin is false), without enforced access
    control, or with service clients, which it doesn't support yet (see
    Authentication)
    (#212,
    #240; #215, #250)
  • Configuration: unknown keys in custom are errors: the app and the Celery
    worker refuse to start, naming them, instead of ignoring them. Remove the
    keys that had no effect from config overlays: custom.celery,
    custom.controllers.celery and custom.controllers.post_task.timeout
    (post, poll, job); requests to TES instances time out after
    custom.tes.timeout seconds instead (default 15, was a hard-coded 5)
    (see
    Configuration)
    (#276, #281; #274, #282)
  • Configuration: entries of custom.tes.service_list are objects with the
    URL of the TES instance (url) and how proTES authenticates at it
    (auth: client_credentials by default, basic or none), instead of
    URLs; with client_credentials, the app and the Celery worker need
    PROTES_CLIENT_ID and PROTES_CLIENT_SECRET and refuse to start without
    them. URLs of TES instances must be https URLs without credentials
    (unless custom.tes.url_schemes allows http); tasks are only forwarded
    to configured (or discovered) TES instances. Custom middlewares that read
    custom.tes.service_list get these entries instead of URLs (see
    TES instances and their credentials)
    (#213, #251)
  • Configuration: the base config lists no TES instances (it listed public
    ones), and the default service info describes the app with placeholders
    on reserved example domains (RFC 2606; id is org.example.protes);
    deployments set their own, and the app warns on startup while the
    placeholders are set (see
    TES instances and their credentials)
    (#229, #251, #263; #205, #220)
  • Configuration: custom.controllers.post_task.polling.wait and attempts
    must be positive, and wait and the new max_wait below 1800 seconds;
    custom.controllers.list_tasks.default_page_size must not exceed
    max_page_size (see
    Task tracking)
    (#277, #278, #279, #288; #208, #238)
  • Database: task identifiers are unique: on startup, proTES creates a unique
    index on task.id and drops the index of earlier versions; it refuses to
    start while stored tasks share identifiers, and names them (see Task
    identifiers    )
    (#248, #258)
  • Database: on startup, proTES removes the HTTP Basic credentials of callers
    that earlier versions stored with each task (basic_auth), and
    credentials in the URLs of inputs and outputs of stored tasks (see
    TES instances and their credentials)
    (#213, #251)
  • Celery: the task that tracks tasks no longer takes credentials, and tasks
    are tracked by short polls that schedule the next one, so the new worker
    rejects tracking jobs that earlier versions queued: upgrade while no tasks
    are being submitted. Unfinished tasks stored by earlier versions keep
    their last state (see Task
    tracking    )
    (#213, #251; #277, #278, #279, #288)
  • Celery: after polling.attempts consecutive failed polls, a task is
    UNKNOWN until its TES instance answers again, instead of failing with a
    final SYSTEM_ERROR; workers no longer set INITIALIZING when they start
    tracking a task (see
    Task tracking)
    (#278, #288)
  • Helm chart: the chart is replaced by a thin chart over FOCA's library
    chart foca-common 0.2.0, with camelCase values (e.g., hostName instead
    of applicationDomain), other resource names and selectors, Secrets of
    the deployment for MongoDB, the broker and Flower instead of credentials
    in the values (the broker's user was guest:guest, Flower's
    admin:admin), Flower off by default, pinned images, NetworkPolicies, and
    an Ingress for all paths (was /ga4gh/tes/v1 only). Releases of the
    legacy charts can't be upgraded: uninstall them and install the new chart
    (see Helm
    chart    )
    (#216, #252; #246, #270)
  • Helm chart: the chart version follows the app version, so chart versions
    restart below 3.0.0; the chart attached to a release deploys the image of
    that release (protes.tag defaults to the chart's appVersion), and the
    chart of this release needs images of this release (see
    Image)
    (#44, #230)
  • Image: latest is the highest stable release (not alpha, beta or release
    candidate) on main; images of commits on dev are tagged dev and
    sha-<commit>. Date-tagged images are no longer pushed (see
    Image)
    (#44, #230)
  • Image: the image installs proTES as a package, runs as a non-root user,
    and serves the app with gunicorn --config python:pro_tes.gunicorn_config 'pro_tes.app:init_app()'; /app/pro_tes, gunicorn.py and wsgi.py are
    gone, and GUNICORN_PROCESSES and GUNICORN_THREADS are replaced by
    GUNICORN_CMD_ARGS. The Celery worker runs celery --app pro_tes.celery_worker worker (see
    Image)
    (#204,
    #223)
  • Docker Compose: MongoDB 7.0 (was 3.6, which can't be upgraded to 7.0
    directly), named volumes instead of PROTES_DATA_DIR, whose data is not
    migrated, ports published on 127.0.0.1 only, and Flower opt-in (see
    Docker
    Compose    )
    (#171, #219; #204, #223)
  • Python: proTES requires Python 3.13 or later, the Python version of the
    image (based on elixircloud/foca:0.17.0-py3.13), the only one it is
    tested with (see
    Configuration)
    (#214, #245; #246, #270)
  • Middlewares: the distance-based middleware no longer uses the unmaintained
    package ip2geotools: it looks up IP addresses with DB-IP's API over
    HTTPS (was HTTP) and geocodes places with Nominatim, times out after 10
    seconds (was 62), and no longer locates addresses that are not global, nor
    places without city or country, for which it falls back to the next
    middleware (see
    Distance-based task distribution)
    (#214, #245)

Features

  • Task ownership: users with the role user create tasks and become their
    owners (FOCA's owner roles); admins manage the access control rules via
    FOCA's permissions API (#153, #215, #250)
  • Credentials of proTES for TES instances: an access token of proTES's own
    client (OAuth 2.0 client credentials grant, from
    custom.tes.client_credentials, cached and refreshed), HTTP Basic
    credentials from environment variables, or none, per TES instance (#213,
    #251)
  • Discovery of TES instances in a service registry with verified services
    (GA4GH Service Registry API, e.g., Cloud Registry): custom.tes.discovery,
    off by default, without a default registry. Only verified TES services
    with public https addresses are used, and the active TES instances are
    listed in the service info (backends) (#202, #260)
  • Task tracking with short polls that schedule the next one, so that a few
    worker processes track many tasks; polls are acknowledged after they ran,
    and workers resume the tracking of stored unfinished tasks when they start
    and periodically, so restarts and upgrades of the workers no longer stop
    the tracking (#277, #278, #279, #288)
  • Helm chart: Gateway API HTTPRoute (with an optional redirect to HTTPS),
    OpenShift Route, Ingress with an optional cert-manager certificate,
    external MongoDB, protes.extraEnv for the app and the worker (e.g., the
    client secret), Gunicorn settings, probes, and checks that fail rendering
    for values of the legacy chart and for missing security settings (#216,
    #252)
  • ignore_error of executors is relayed to TES instances (#176, #219)
  • Configuration: local overrides (config.local.yaml,
    docker-compose.override.yaml) are ignored by Git (#266, #269)

Security

  • Signed images and chart: images (releases, dev and sha-<commit>) come
    with SBOM and provenance attestations, a GitHub artifact attestation and a
    keyless Sigstore signature; the chart attached to the release has a GitHub
    artifact attestation and a Sigstore bundle
    (protes-0.3.0.tgz.sigstore.json). The published images are the scanned
    ones: images are scanned for known vulnerabilities with Trivy in CI,
    before a release is published, and weekly (latest). See "Verifying
    images and charts" on the Operations page of the documentation (#234,
    #256)
  • Task identifiers can't be shared: a request that draws an identifier in
    use gets another one, and never gets a role on another user's task (#248,
    #258)
  • Discovered TES instances must resolve to public addresses only, checked
    on discovery and before each request; requests to them follow no
    redirects (#202, #260)

Fixes

  • Tasks that their TES instance reports as PREEMPTED are final (they were
    polled forever), and CANCELING tasks are polled until they are canceled
    (#261, #262)
  • Tasks are sent to TES instances exactly once (#274, #282)
  • New tasks are stored with the state that their TES instance reports, and
    a cancel request while a task is being forwarded keeps it CANCELED
    (#275, #291)
  • A cancel request no longer replaces a final state that the tracking
    stored while the request was sent to the TES instance (#296, #301)
  • Tasks whose create request was cut off (e.g., by a rolling upgrade) no
    longer stay UNKNOWN for good: the resume sweep of the workers sets them
    to SYSTEM_ERROR after custom.controllers.post_task.max_create_time
    seconds (new, default 3600, at most a year). On upgrade, this also applies
    to tasks of earlier versions: UNKNOWN tasks without any remote
    identifier that are older than that become SYSTEM_ERROR (see Task
    times and remote
    identifiers    )
    (#297, #303)
  • Failed polls no longer end the tracking of tasks that are still running,
    and the tracking survives worker restarts (#277, #278, #279, #288)
  • Task listings iterate the database cursor instead of loading all matching
    tasks, and the view BASIC leaves out the logs it should (#284, #292;
    #208, #238)
  • Each TES instance gets its own copy of the task; the remote identifier is
    stored in tes_endpoint.task_id, so that tasks can also be canceled with
    custom.storeLogs.execution_trace: False (#208, #238)
  • Startup no longer loops forever with jsonschema > 3.2.0, and no longer
    fetches the GA4GH service info schemas from GitHub (#173, #203, #219)
  • Output sizes reported by TES instances as integers no longer fail the
    tracking of finished tasks (#203, #219)
  • The tracking task uses the database named by MONGO_DBNAME, as the app
    does (#203, #219)
  • The examples of TES instances in the docs and the chart use the current
    config format (#229, #263)
  • Release workflow: the previous release is found on first releases, and
    releases are published after their assets are uploaded, as immutable
    releases require (#272, #273; #280, #287)

Documentation

  • A documentation site on Read the Docs (MkDocs Material), with pages on
    getting started, configuration, the API, access control, middlewares,
    deployment, operations, development and upgrading (#209, #228; #210,
    #221)
  • A code of conduct, a contributing guide, a security policy, CODEOWNERS and
    citation metadata (#211, #232, #243; #233, #242; #235, #244; #247, #249)
  • Rate limits: how to limit requests at the ingress controller (#284, #292)
  • The served API specification declares the 409 (cancel) and 413
    (create) responses of proTES as extensions (#298, #302)
  • The size of FULL pages: a page can reach about 64 MiB by default, so
    memory-limited deployments (the chart's API pod has 256 MiB) should lower
    custom.controllers.list_tasks.max_page_size_full (#298, #302)

Maintenance

  • Releases are published from signed version tags by a release workflow,
    with checks of the tag, the citation metadata, the release notes and the
    pinned dependencies, also before tagging; the notes of the first release
    list all its pull requests (#44, #230; #264, #267; #265, #268; #289,
    #293)
  • The release version comes from one source, the tag (and, before tagging,
    the citation metadata): the docs and the chart's README get it from
    there, and a test fails if it is hard-coded elsewhere (#290, #299)
  • Packaging with pyproject.toml and setuptools-scm, and FOCA 0.17.0 and
    foca-common 0.2.0 releases pinned (#204, #223; #246, #270)
  • Checks with pre-commit (ruff, mypy, semgrep, ast-grep, gitleaks,
    actionlint), Google-style docstrings and complete type hints, and tests
    against a real MongoDB at 100% coverage (#227; #206, #231; #142, #149,
    #168, #207, #236; #159, #239)
  • Pull requests target dev by default, also those of Dependabot (#257,
    #259)
  • Ported fixes of cloud-registry's first release: Gunicorn keepalive 100 s,
    generic-repository and docs-URL tests, Dependabot groups, unused Swagger 2
    specs removed (#266, #269)
  • Citation metadata are set safely (file mode kept) (#253, #255)

Commits

New Contributors

Full Changelog: https://github.com/elixir-cloud-aai/proTES/commits/v0.3.0

Contributors

lvarin, uniqueg, and 12 other contributors
Assets 4
Loading