Release 2023-01-12 - (expected chart version 4.30.0)

[zebot]

[2023-01-12] (Chart Release 4.30.0)

Release notes

• This realease migrates data from galley.member_client to galley.mls_group_member_client. When upgrading wire-server no manual steps are required. (Use MLS member table indexed by group id #2859)

• Upgrade webapp version to 2022-12-19-production.0-v0.31.9-0-6b2f2bf (Update webapp version in Helm chart [skip ci] #2302)

API changes

• • The endpoints POST /conversations/list and GET /conversations have been removed. Use POST /conversations/list-ids followed by POST /conversations/list instead.
  • The endpoint PUT /conversations/:id/access has been removed. Use its qualified counterpart instead.
  • The field access_role_v2 in the Conversation type, in the request body of POST /conversations, and in the request body of PUT /conversations/:domain/:id/access has been removed. Its content is now contained in the access_role field instead. It replaces the legacy access role, previously contained in the access_role field.
  • Clients implementing the V3 API must be prepared to handle a change in the format of the conversation.access_update event. Namely, the field access_role_v2 has become optional. When missing, its value is to be found in the field access_role. (Versioned combinators and access_role changes in V3 #2841)

• Added a domain parameter to the typing indicator status update API ([FS-1188] Enable the typing indicator in federated environments. #2892)

• Support MLS self-conversations via a new endpoint GET /conversations/mls-self. This removes the PUT counterpart introduced in [FS-925] Add endpoint to create MLS self-conversation #2730 ([FS-925] Lazily create an MLS Self-conversation #2839)

• List the MLS self-conversation automatically without needing to call GET /conversations/mls-self first ([FS-1191] List the MLS Self-conversation Automatically #2856)

• Fail early in galley when the MLS removal key is not configured (Fail on MLS endpoints in galley when MLS is not enabled #2899)

• Introduce a flag in brig to enable MLS explicitly. When this flag is set to false or absent, MLS functionality is completely disabled and all MLS endpoints fail immediately. (MLS flag in brig #2913)

• Conversation events may have a "subconv" field for events that originate in a MLS subconversation (Add optional subconversation field to conversation events #2933)

• GET /system/settings/unauthorized returns a curated set of system settings from brig. The endpoint is reachable without authentication/authorization. It's meant to be used by apps to adjust their behavior (e.g. to show a registration dialog if registrations are enabled on the backend.) Currently, only the setRestrictUserCreation flag is exported. Other options may be added in future (in consultation with the security department.) (Add /system/settings endpoint (SQPIT-1141) #2903)

Features

• The coturn Helm chart now has a .tls.ciphers option to allow setting
  the cipher list for TLS connections, when TLS is enabled. By default,
  this option is set to a cipher list which is compliant with BSI
  TR-02102-2. (Add TLS cipher configuration to coturn Helm chart [SQPIT-1512] #2924)

• Nginz helm chart: The list of upstreams is split into nginx_conf.upstreams and
  nginx_conf.extra_upstreams. Extra upstreams are disabled by default. They can
  be enabled by adding their name (entry's key) to
  nginx_conf.enabled_extra_upstreams. nginx_conf.ignored_upstreams is only
  applied to upstreams from nginx_conf.upstreams. In the default configuration
  of nginz extra upstreams are ibis, galeb, calling-test and proxy. If one
  of those is deployed, its name has be be added to
  nginx_conf.enabled_extra_upstreams (otherwise, it won't be reachable). Unless
  nginx_conf.upstreams hasn't been changed manually (overriding its default),
  this should be the only needed migration step. (nginz deactivate unused upstreams (SQPIT-1174) #2849)

• A team member's role can now be provisioned via SCIM ([SQSERVICE-253] Support provisioning role information with SCIM #2851, [SQSERVICES-253] Support provisioning role information with scim PATCH #2855)

• Team search endpoint now supports pagination ([SQSERVICES-1773] Allow pagination for team search endpoints - has_more #2898, [SQSERVICES-1773] Allow pagination for team search endpoint #2895)

• Introduce optional disabledAPIVersions configuration setting (Introduce disabledAPIVersions #2951)

• Add more logs to SMTP mail sending. Ensure that logs are written before the application fails due to SMTP misconfiguration. (bullet proof smtp #2818)

• Added typing indicator status propagation to federated environments ([FS-1188] Enable the typing indicator in federated environments. #2892)

• Allow vhost style addressing for S3 as path style is not supported for newer buckets.

  More info: https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/ (Use amazonka from upstream head intead of fork #2955)

Bug fixes and other updates

• Fix typo for Servicemonitor enable var in default values for helm charts. (update enabled key for servicemonitor in helm chart default values #2896)

• The parser for the AWS/SNS error message to explain that an endpoint is already in use was incorrect. This lead to an "invalid token" error when registering push tokens for multiple user accounts (user ids) instead of updating the SNS endpoint with an additional user id. (fix gundeck aws endpoint parsing #2921)

• Avoid client deletion edge case condition which can lead to inconsistent data between brig and galley's clients tables. (Avoid client deletion edge case condition which can lead to inconsist… #2830)

• Conversations inside events are now serialised using the format of API V2 (Force V2 when serialising conversations in events #2971)

• Do not throw 500 when listing conversations and MLS is not configured ([FS-1271] Do not throw 500 when listing conversations #2893)

• Do not list MLS self-conversation in client API v1 and v2 if it exists ([FS-1249] Do Not List MLS Self-conversation in client API v1 and v2 #2872)

• Limit 2FA code retries to 3 attempts ([SQSERVICES-1824] [fix] Entering 2FA code multiple times does not invalidate it #2960)

• Fix bug in MLS user removal from conversation: the list of removed clients has to be compared with those in the conversation, not the list of all clients of that user (Fix client check when removing clients from MLS conversations #2817)

• Due to sftd changing how configuration is handled for "multi-SFT" calling (starting with version 3.1.10), new options have been added to the sftd Helm chart for compatibility with these newer versions. (Fix SFT Helm chart for new sftd versions #2886)

• For sftd/coturn/restund, fixed a bug in external ip address lookup, in case Kubernetes Node Name doesn't equal hostname. ([charts:sftd,restund,coturn] Fix case when hostname is not node name #2837)

• Requesting a new token with the client_id now works correctly when the old token is part of the request (Add client_id to token in all cases #2860)

Documentation

• Add extra section to the deeplink docs to explain the socks proxy support while login. (Update deeplink doc with proxy support #2885)

• Describe the auth cookie throttling mechanism. And overhaul the description of auth cookies in general. (Overhaul the auth cookie documentation #2941)

• PR guidelines docs are updated with correct helm configuration syntax (Update PR guideline docs #2889)

Internal changes

• Log AWS / SNS invalid token responses. This is helpful for native push notification debugging purposes. (Log AWS SNS invalid token responses (SQCORE-1267) #2908)

• Add tests for invitation urls in team invitation responses. These depend on the settings of galley. (Add test for urls in team invitations (SQPIT-1368) #2797)

• brig: Allow multiple threads to run simultaneously (brig: Allow multiple threads to run simulaneously #2972)

• Remove support for compiling local docker images with buildah. Nix is used to build docker images these days (remove buildah references #2822)

• Nix-created docker images: add some debugging tools in the containers, and add 'make build-image-' for convenience (docker images: Add container utilities #2829)

• Added typeclasses to track uses of federated calls across the codebase. (Track federated calls #2940)

• Split galley API routes and handler definitions into several modules (Split galley servant definition #2820)

• Default intraListing to true. This means that the list of clients, so far saved in both brig's and galley's databases, will still be written to both, but only read from brig's database. This avoids cases where these two tables go out of sync. Brig becomes the source of truth for clients. In the future, if this holds, code and data for galley's clients table can be removed. (allow configuring intra-listing in galley, make default=true #2847)

• Introduce the MakesFederatedCall Servant combinator (MakesFederatedCall servant combinator #2950)

• Bump nixpkgs to latest unstable. Stop using forked nixpkgs. (Bump nixpkgs to latest unstable, unpin openssl #2828)

• Optimize memory usage while creating large conversations (Optimize memory usage while creating large conversations #2970)

• Reduce Polysemy-induced high memory requirements (Upgrade Polysemy to latest to fix OOM in Galley.Run #2947)

• Brig calling API is now migrated to servant ([SQSERVICES-1644] Servantify brig calling API #2815)

• Fixed flaky feature TTL integration test ([SQSERVICES-1618] Flaky test for TTL feature flag #2823)

• Brig teams API is now migrated to servant ([SQSERVICES-1010] Servantify brig team API #2824)

• Add 'inconsistencies' tool to check for, and repair certain kinds of data inconsistencies across different cassandra tables. (Add tool to detect/fix DB inconsistencies #2840)

• Backoffice Swagger 2.x docs is exposed on / and the old Swagger has been removed. Backoffice helm chart only runs stern without an extra nginx. ([SQSERVICES-1787] fix backoffice swagger-ui #2846)

• Give proxy service a servant routing table for swagger (not for replacing wai-route; see comments in source code) ([SQSERVICES-1647] Servantify proxy (the swagger part) #2848)

• Stern API endpoint GET ejpd-info has now the correct HTTP method (Fix stern API endpoint ejpd-info with correct HTTP method #2850)

• External commits: add additional checks (External commits: add additional checks #2852)

• Golden tests for conversation and feature config event schemas ([FS-1008] Golden tests for event schemas #2861)

• Add startup probe to brig helm chart. (add initial delay seconds to brig liveness and readiness probes #2878)

• Track federated calls in types across the codebase. (Track federated calls #2940)

• Update nix pins to point at polysemy-1.8.0.0 (Pin polysemy to 1.8.0.0 #2949)

• Add MakesFederatedCall combinators to Galley (Add MakesFederatedCall combinators to Galley #2957)

• Fix make clean; allow new data constructors in ToSchema Version instance (nit-picks. #2965)

• Refactor and simplify MLS message handling logic (Simplify and cleanup MLS message handling #2844)

• Remove cassandra queries to the user_keys_hash table, as they are never read anymore since 'onboarding' / auto-connect was removed in Remove autoconnect functionality; deprecate end-point. #1005 (Cleanup: Remove cassandra queries to the user_keys_hash table #2902)

• Replay external backend proposals after forwarding external commits.
  One column added to Galley's mls_proposal_refs. (Replay external backend proposals after external commits #2842)

• Remove an unused effect for remote conversation listing (Remove an unused effect for remote conversation listing #2954)

• Introduce types for subconversations (Introduce the subconversation types and change federation message sending request types #2925)

• Use treefmt to ensure consistent formatting of .nix files, use for shellcheck too (Use treefmt to nixpkgs-fmt all .nix files #2831)

Federation changes

• Honour MLS flag in brig's federation API (Honour MLS flag in brig's federation API #2946)

• Split the Proteus and MLS message sending requests into separate types. The MLS request now supports MLS subconversations. This is a federation API breaking change. (Introduce the subconversation types and change federation message sending request types #2925)

• Injects federated calls into the x-wire-makes-federated-calls-to extension of the swagger Operations (MakesFederatedCall servant combinator #2950)