v0.23.0-rc4

Release Notes for v0.23.0-rc4

Mandatory migration tasks

N/A

Optional migration tasks

• enable_opentelemetry has been split into two flags: enable_metrics and enable_tracing
• A new pair of migrations for CockroachDB has been added for RID (4.1.0) and SCD (3.4.0) to remove useless indexes.
  • The DSS will work regardless of the migration status.
  • This migration will improve performance. If that is not the case for a specific scenario that was missed, please downgrade to the previous version and open an issue.
  • These indexes are still useful for Yugabyte, so no such migration exists for it.
  • Details about migrations are available there
• The 3.4.1 (or 1.1.1 for Yugabyte) migration for SCD is mandatory if you plan to use the new SCD hash lock option.
  • This migration can be applied without any impact if you do not plan to use it.
• The default storage class for AWS changed from gp2 to gp3.
  • For existing clusters, ensure the value of aws_kubernetes_storage_class in your terraform config is set to gp2 if it was not defined. Switching storage class is not supported.
  • As gp3 doesn't exist by default in EKS, terraform will now create the new storage class. If gp3 already exists in your cluster, set aws_create_storage_class to false.
  • There is a behaviour change between the previous default storage class and the new one: reclaimPolicy is set to Retain to limit accidental data deletion.

Important information

• Fixed a bug where the evict command ignored entries without a locality. If your DSS instance does not have a locality set, the next evict run may be slow while it processes the backlog of old entries.

• Fixed a bug where Helm charts and Tanka files didn't actually perform any actions via the evict command when run via cron jobs (because no locality was set and no delete flag was specified). If you have a large number of entries, the next run may be slow while it processes the backlog of old entries.

• AWS load balancer names are no longer enforced by Helm charts or Tanka files. Existing clusters will retain their current names, while new ones will use names automatically generated by AWS.

• Server timeout flag has been renamed from server timeout to server_timeout.

• The duration field, previously recorded as a string containing the unit, has been changed to duration_ms, a float representing the duration in milliseconds, rounded to 0.01.

• Grafana version deployed by Helm charts or Tanka files have been upgraded to 13.0. Ensure to read grafana changelog based on your current version.

• The Google Kubernetes Engine (GKE) node disk size has been increased from 15GB to 25GB to prevent issues with saturated system disks.

  • You will need to apply the Terraform state once to recreate the node pool.
  • This process will take some time because it is performed in a non-disruptive rolling update, draining and moving pods across nodes during the upgrade.
  • Please plan accordingly and test the rollout on a non-production cluster first.

Minimal database schema version

Schema	CockroachDB	Yugabyte
AUX	1.1.0	1.1.0
RID	4.0.0	1.0.1
SCD	3.3.0	1.1.0

Validation

USS qualifier versions: v0.31.0
Test configuration and report: See attachment.

Performances

Performance validation of the test scenarios described here: #1541 (comment)

• 3x3 nodes deployed, using CRDB, using time-based notification index
• On a single AWS EC2 instance c8a.24xlarge (96 vCPU, 192 GB RAM)
• With simulated latencies
• Using monitoring repo on 1a75ba496a7cad64a08dc3c5b711ab54eec00ff0

Configuration (environment):

export DSS_IMAGE=interuss/dss:v0.23.0-rc4
export INTRA_USS_NETEM_CONF=delay 600us 40us 25% distribution normal loss 0.0005%
export INTER_USS_NETEM_CONF=delay 16.5ms 7.5ms 50% distribution paretonormal loss 0.025% 25%
export NUM_USS=3
export NUM_NODES=3

Additional DSS start flags:

-datastore_max_open_conns=30
-enable_time_based_notification_index=true
-enable_scd_global_lock=false

Changed CRDB start flags:

--cache=1G
--max-sql-memory=1G

Scenario 1 "Busy S2cell Test"

AUTH_SPEC="DummyOAuth(http://localhost:8085/token,locust)" uv run --index https://pypi.org/simple locust -f ./monitoring/loadtest/locust_files/FlightsInSub.py -H http://localhost:8555 --uss-base-url http://testdummy.interuss.org/uss --cluster-count 1 --base-lat -34.928683 --base-lng 138.593795 --area-radius 550 --max-flight-distance 100 --oi-duration 60

Scenario 2 "Global Throughput (Multi Site) Test Part 1"

AUTH_SPEC="DummyOAuth(http://localhost:8085/token,locust)" uv run --index https://pypi.org/simple locust -f ./monitoring/loadtest/locust_files/FlightsInSub.py -H http://localhost:8555 --uss-base-url http://testdummy.interuss.org/uss --cluster-count 116 --base-lat -34.928683 --base-lng 138.593795 --area-radius 4800 --max-flight-distance 10000 --oi-duration 240

Scenario 3 "Global Throughput (Multi Site) Test Part 2"

AUTH_SPEC="DummyOAuth(http://localhost:8085/token,locust)" uv run --index https://pypi.org/simple locust -f ./monitoring/loadtest/locust_files/FlightsInSub.py -H http://localhost:8555 --uss-base-url http://testdummy.interuss.org/uss --cluster-count 11 --base-lat -34.928683 --base-lng 138.593795 --area-radius 15182 --max-flight-distance 10000 --oi-duration 240

What's Changed

• Reset NEXT_RELEASE_NOTES.md by @mickmis in #1427
• RELEASE.md: add 'NEXT_RELEASE_NOTES.md' workflow and validation section in release notes by @mickmis in #1428
• [aux] Fill nil on empty body on PutDSSInstancesHeartbeat by @the-glu in #1429
• [tooling] Add authentification tests by @the-glu in #1430
• Add .gitkeep file to docker ignore to prevent -dirty version by @the-glu in #1432
• [deploy] Improve recommandations for storage type & instance types by @the-glu in #1433
• Auth: Move authentification validation in generated code by @the-glu in #1434
• [certificates] Add CI hygiene checks, minor fixes on python hygiene checks by @the-glu in #1435
• Bump github.com/jackc/pgx/v5 from 5.7.4 to 5.9.0 by @dependabot[bot] in #1441
• [docs] Improve data storage documentation by @BenjaminPelletier in #1437
• [rid/datastore/ListExpired(Subscriptions|ISAs)] Properly escape writer; Fix removal of expired entities with NULL writer (empty locality) by @the-glu in #1436
• [tooling] Improve shell-lint to ignore git-ignored files by @the-glu in #1443
• Bump github.com/jackc/pgx/v5 from 5.9.0 to 5.9.2 by @dependabot[bot] in #1450
• [store] Squash datastore.Store and datastore.Datastore by @the-glu in #1444
• [store] Revork init structure; use Store interfaces instead of datastore.Store interfaces by @the-glu in #1445
• [store] Rename datastore to sqlstore by @the-glu in #1446
• [store] Move CodeRetryable to generic store by @the-glu in #1447
• [store] Add store type flag by @the-glu in #1448
• Upgrade to go 1.26.2 by @the-glu in #1458
• [Raft] Add Raft store stubs by @MariemBaccari in #1451
• [monitoring] Add and expose opentelemetry metrics by @the-glu in #1454
• [tooling]: Generate all openapi-to-go files; make setAuthError generated only when needed by @the-glu in #1455
• [Raft] Add etcd raft dependencies by @MariemBaccari in #1460
• Bump google.golang.org/protobuf from 1.23.0 to 1.33.0 in /interfaces/openapi-to-go-server/example by @dependabot[bot] in #1465
• Bump golang.org/x/net from 0.30.0 to 0.38.0 in /interfaces/openapi-to-go-server/example by @dependabot[bot] in #1466
• Bump google.golang.org/grpc from 1.29.1 to 1.79.3 in /interfaces/openapi-to-go-server/example by @dependabot[bot] in #1467
• [Doc] Improve monitoring page & cleanup outdated monitoring config/examples by @the-glu in #1457
• [Raft] Add Raft transport by @MariemBaccari in #1464
• [tooling]: Enforce in CI generation of gen.go files by @the-glu in #1456
• [Raft] Add log warning about Raft implementation by @MariemBaccari in #1468
• [tanka/helm] Set localitly on cleanup jobs by @the-glu in #1477
• [doc] Add tracing documentation by @the-glu in #1471
• [terraform] Upgrade to kubernetes version 1.35 by @the-glu in #1478
• [helm/tanka] Normalize wait container for schema, jobs and core-service by @the-glu in #1481
• [terraform] Add option to make GKE cluster deletable by @the-glu in #1484
• [tanka/helm] Remove 'name' anotation in AWS load balancer by @the-glu in #1489
• [doc] Add a deployment checklist by @the-glu in #1491
• [tanka/helm] Actually delete entries in cronjob evict tool by @the-glu in #1487
• [doc] Add command to connect to YugabyteDB in troubleshooting by @the-glu in #1501
• [scd] Remove redundant fetch cells call by @MariemBaccari in #1475
• [tanka] Remove useless yugabyte hooks by @the-glu in #1479
• [geo] normalize s2 loop by @RustedBones in #1485
• [geo] Leverage s2.CrossSign for loop validation by @RustedBones in #1490
• [tooling] Automatic release artefacts: Base script helper & config by @the-glu in #1493
• [tooling] Automatic release artefacts: Cluster spawning by @the-glu in #1494
• [tooling] Automatic release artefacts: Cluster configuration by @the-glu in #1495
• [tooling] Automatic release artefacts: Deploy services by @the-glu in #1496
• [tooling] Automatic release artefacts: Run tests by @the-glu in #1497
• [tooling] Automatic release artefacts: Compile results by @the-glu in #1498
• [tooling] Automatic release artefacts: Destroy clusters by @the-glu in #1499
• [doc] Move cleanup part from README to documentation (partially merged in #1491) by @the-glu in #1469
• [auth] Don't crash on tokens with missing 'exp' field by @the-glu in #1508
• [metrics] Add RID metrics: subscriptions and isa count by @the-glu in #1502
• [interfaces] Remove invalid default value in aux API by @BenjaminPelletier in #1511
• [Raft] Add Raft persistence by @MariemBaccari in #1470
• [metrics] Add SCD metrics: subscriptions, constraints and operational intents count by @the-glu in #1503
• [core] Fix timeout argument by @the-glu in #1514
• [core] Use better timeout handler and log timeouts by @the-glu in #1515
• [release scripts] Ignore yugabyte certificate init errors by @the-glu in #1516
• [opentelemetry] Split enable_opentelemetry into enable_metrics and enable_tracing by @the-glu in #1504
• [logging] Normalize duration to milliseconds, rounded to 0.01 by @the-glu in #1519
• [monitoring] Normalize grafana version to 13.0 by @the-glu in #1505
• [deploy] Add enable_metric flag for DSS + fetch pod metrics in prometheus by @the-glu in #1506
• [metrics] Add DSS grafana dashboard by @the-glu in #1507
• [Raft] Initialization and configuration of consensus by @MariemBaccari in #1473
• [tools] Cleanup dockers container by @the-glu in #1524
• [scd] Update and fetch notification_index in one query by @the-glu in #1518
• [doc] Add documentation about migrations, moved from a too specifc place by @the-glu in #1547
• [db-manager] Add explicit error on unknown migrations by @the-glu in #1533
• [crdb] Remove useless indexes by @the-glu in #1517
• [Raft] Use separate consensus instances per store by @MariemBaccari in #1550
• [doc] Small rework of architecture section by @the-glu in #1560
• [dss] Add flag to enable time-based notification index by @the-glu in #1558
• [deploy] Add enable_time_based_notification_index setting to terrafrom, helm and tanka by @the-glu in #1562
• [deploy] Add datastore_max_open_conns flags to available options by @the-glu in #1563
• [store] Move behaviour flags to a dedicated config object by @the-glu in #1565
• [aux] Add a dedicated endpoint for global options by @the-glu in #1566
• [http] Limit body size to 4Mb by @the-glu in #1567
• [terraform] Increase GKE disk size by @the-glu in #1569
• [yugabyte] Implement Transact retries by @the-glu in #1570
• [rid] Improve UpdateNotificationIdxsInCells by locking row in order by @the-glu in #1571
• [tooling] Upgrade monitoring image to 0.31.0 by @the-glu in #1568
• [Raft] Consensus entries publishing and proposal by @MariemBaccari in #1474
• [raft/store] Add timestamp middleware by @MariemBaccari in #1575
• [http] Fix middleware order by @the-glu in #1576
• [scd] Use database join to fetch uss availability in fetchOperationalIntents by @the-glu in #1522
• [raft] Add base memstore by @the-glu in #1527
• [rid] Remove rid volume conversion by @RustedBones in #1579
• [scd] Remove scd volume conversion by @RustedBones in #1551
• Bump golang.org/x/net from 0.48.0 to 0.55.0 in /interfaces/openapi-to-go-server/example by @dependabot[bot] in #1589
• Bump golang.org/x/net from 0.52.0 to 0.55.0 by @dependabot[bot] in #1590
• [raft/doc] Add Raftstore documentation summary by @MariemBaccari in #1591
• [raft/doc] Add Raftstore terminology by @MariemBaccari in #1593
• [raft/store] Update Transact signature by @MariemBaccari in #1577
• [dss] Only search for operational intents if requested in PutSubscription by @wing-utm-sharing-airspace in #1600
• Bump golang.org/x/crypto from 0.51.0 to 0.52.0 by @dependabot[bot] in #1596
• Bump google.golang.org/grpc from 1.79.3 to 1.82.1 in /interfaces/openapi-to-go-server/example by @dependabot[bot] in #1603
• Bump google.golang.org/grpc from 1.79.3 to 1.82.1 by @dependabot[bot] in #1602
• [dev] Enable Local Metrics and Tracing by @makinje16 in #1605
• [raft/memstore] Add aux memstore by @the-glu in #1529
• [raft/memstore] Add snapshop capability to aux memstore by @the-glu in #1530
• [raft/memstore] Add rid memstore by @the-glu in #1534
• [raft/store] Add IsReadOnly to handler by @MariemBaccari in #1595
• [tooling] Enable more go linters by @the-glu in #1607
• [raft/memstore] Use now from context by @the-glu in #1535
• [raft/memstore] Add Checkpoint/Restore by @the-glu in #1539
• Replace gp2 with gp3 by @wing-utm-sharing-airspace in #1606
• [raft/store] Generate OperationID() on API request types by @MariemBaccari in #1580
• [Raft] Add generic raftstore by @MariemBaccari in #1525
• [infrasctructure/utils] Upgrade variable generator to latest versions by @the-glu in #1609
• [raft/memstore] Add scd memstore by @the-glu in #1542
• [scd] Add hash lock option by @the-glu in #1520
• [rid] Fix query when using enableTimeBasedNotificationIndex by @the-glu in #1616
• [tooling] Enable additional DSS configuration for local deployments by @the-glu in #1617
• [raft/memstore] Add Get and RestoreFrom Snapshot to memstore by @MariemBaccari in #1622
• [raft/memstore] Embed Memrepo in Store by @MariemBaccari in #1625
• [deploy] Add timeout option for evict tool to terraform/helm/tanka by @the-glu in #1630
• [test] Add coverage for isYugabyteRetryable; fix TestNewRSAAuthClient on Windows by @ashishjsharda in #1620
• [rid] Extract polygon conversion from geo module by @RustedBones in #1552
• [scd] Move scd conversions to scd package by @RustedBones in #1584
• [terraform/aws] Create gp3 storage class, add release notes by @the-glu in #1629

New Contributors

• @wing-utm-sharing-airspace made their first contribution in #1600
• @makinje16 made their first contribution in #1605

Full Changelog: interuss/dss/v0.22.0...interuss/dss/v0.23.0-rc4