Upgrade software dependencies 2026-04-13 (#7925)

[dsotirho-ucsc]

Linked issue: #7925

Checklist

Author

• PR is assigned to the author
• Status of PR is In progress
• Target branch is develop
• Name of PR branch matches upgrades/yyyy-mm-dd
• PR is linked to the upgrade issue it resolves
• Status of linked issue is In progress
• PR title matches Upgrade software dependencies yyyy-mm-dd
• PR title references the linked issue

Author (upgrading deployments)

• Ran make docker_images.json and committed the resulting changes _{or this PR does not modify azul_docker_images, or any other variables referenced in the definition of that variable}
• Documented upgrading of deployments in UPGRADING.rst _{or this PR does not require upgrading deployments}
• Added u tag to commit title _{or this PR does not require upgrading deployments}
• This PR is labeled upgrade _{or does not require upgrading deployments}
• This PR is labeled deploy:shared _{or does not modify docker_images.json, and does not require deploying the shared component for any other reason}
• This PR is labeled deploy:gitlab _{or does not require deploying the gitlab component}
• This PR is labeled backup:gitlab
• This PR is labeled deploy:runner _{or does not require deploying the runner image}

Author (before every review)

• Rebased PR branch on develop, squashed fixups from prior reviews
• Ran make requirements_update _{or this PR does not modify Dockerfile, environment, requirements*.txt, common.mk, Makefile or environment.boot}
• Added R tag to commit title _{or this PR does not modify requirements*.txt}
• This PR is labeled reqs _{or does not modify requirements*.txt}
• Updated the AL2023_release variable in gitlab.tf.json.template.py to the most recent AL2023 release _{or no update is available}
• make integration_test passes in personal deployment _{or this PR does not modify functionality that could affect the IT outcome}
• PR is not a draft
• PR is awaiting requested review from system administrator
• Status of PR is Review requested
• PR is assigned to only the system administrator and the author

System administrator (after approval)

• Actually approved the PR
• Labeled linked issue as no demo
• A comment to this PR details the completed security design review
• PR title is appropriate as title of merge commit
• N reviews label is accurate
• Status of PR is Approved
• PR is assigned to only the operator and the author

Operator

• Squashed PR branch and rebased onto develop
• Sanity-checked history
• Pushed PR branch to GitHub

Operator (deploy .shared and .gitlab components)

• Ran _select dev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused _{or this PR is not labeled deploy:shared}
• Ran _select dev.gitlab && python scripts/create_gitlab_snapshot.py --no-restart (see operator manual for details) _{or this PR is not labeled backup:gitlab}
• Ran _select dev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply _{or this PR is not labeled deploy:gitlab}
• Ran _select anvildev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused _{or this PR is not labeled deploy:shared}
• Ran _select anvildev.gitlab && python scripts/create_gitlab_snapshot.py --no-restart (see operator manual for details) _{or this PR is not labeled backup:gitlab}
• Ran _select anvildev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply _{or this PR is not labeled deploy:gitlab}
• Checked the items in the next section _{or this PR is labeled deploy:gitlab}
• PR is assigned to only the system administrator and the author _{or this PR is not labeled deploy:gitlab}

System administrator (post-deploy of .gitlab component)

• Background migrations for dev.gitlab are complete _{or this PR is not labeled deploy:gitlab}
• Background migrations for anvildev.gitlab are complete _{or this PR is not labeled deploy:gitlab}
• PR is assigned to only the operator and the author

Operator (deploy runner image)

• Ran _select dev.gitlab && make -C terraform/gitlab/runner _{or this PR is not labeled deploy:runner}
• Ran _select anvildev.gitlab && make -C terraform/gitlab/runner _{or this PR is not labeled deploy:runner}

Operator (sandbox build)

• Added sandbox label
• Pushed PR branch to GitLab dev
• Pushed PR branch to GitLab anvildev
• Build passes in sandbox deployment
• Build passes in anvilbox deployment
• Reviewed build logs for anomalies in sandbox deployment
• Reviewed build logs for anomalies in anvilbox deployment

Operator (merge the branch)

• All status checks passed and the PR is mergeable
• The title of the merge commit starts with the title of this PR
• Added PR # reference to merge commit title
• Collected commit title tags in merge commit title _{but excluded any p tags}
• Closed related Dependabot PRs with a comment referencing the corresponding commit in this PR _{or this PR does not include any such commits}
• Pushed merge commit to GitHub
• Status of PR is Merged lower
• Status of blocked issues is Triage _{or no issues are blocked on the linked issue}

Operator (main build)

• Pushed merge commit to GitLab dev
• Pushed merge commit to GitLab anvildev
• Build passes on GitLab dev
• Reviewed build logs for anomalies on GitLab dev
• Build passes on GitLab anvildev
• Reviewed build logs for anomalies on GitLab anvildev
• Ran _select dev.shared && make -C terraform/shared apply _{or this PR is not labeled deploy:shared}
• Ran _select anvildev.shared && make -C terraform/shared apply _{or this PR is not labeled deploy:shared}
• Deleted PR branch from GitHub
• PR is assigned to only the operator
• Deleted PR branch from GitLab dev
• Deleted PR branch from GitLab anvildev
• Status of linked issue is Lower

Operator

• At least 24 hours have passed since anvildev.shared was last deployed
• Ran scripts/export_inspector_findings.py against anvildev, imported results to Google Sheet and posted screenshot of relevant¹ findings as a comment on the linked issue.
• Propagated the deploy:shared, deploy:gitlab, deploy:runner and backup:gitlab labels to the next promotion PRs _{or this PR carries none of these labels}
• Propagated any specific instructions related to the deploy:shared, deploy:gitlab, deploy:runner and backup:gitlab labels, from the description of this PR to that of the next promotion PRs _{or this PR carries none of these labels}
• PR is assigned to only the system administrator

¹A relevant finding is a high or critical vulnerability in an image
that is used within the security boundary. Images not used within the boundary
are tracked in azul.docker_images under a key starting with _.

System administrator

• No currently reported vulnerability requires immediate attention
• PR is assigned to no one

Shorthand for review comments

• L line is too long
• W line wrapping is wrong
• Q bad quotes
• F other formatting problem

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.04%. Comparing base (663f462) to head (609ee4b).

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #7934   +/-   ##
========================================
  Coverage    85.04%   85.04%           
========================================
  Files          162      162           
  Lines        23303    23303           
========================================
  Hits         19819    19819           
  Misses        3484     3484           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[coveralls]

coverage: 85.13%. remained the same — upgrades/2026-04-13 into develop

[dsotirho-ucsc]

7934_IT_2026-04-14.txt

[hannes-ucsc]

I pushed a new emulator image with upgrades to the Go SDK. It has fewer open vulnerabilities:

Please test that image and, if successful, include it in this PR.

[dsotirho-ucsc]

Please test that image and, if successful, include it in this PR.

Tests pass with new (0.4.4-61) bigquery_emulator image.

/Users/daniel/repo/azul1/.venv/bin/python /Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pycharm/_jb_unittest_runner.py --path /Users/daniel/repo/azul1/test/indexer/test_tdr.py 
Testing started at 11:23 AM ...
envhook.py: Setting azul_python_version to '3.14.4'
envhook.py: Setting azul_python_image to 'docker.io/library/python@sha256:3b6a769c2263568136038656b39d599df2e15445541d5f4ea02b8f8418654166'
envhook.py: Setting azul_docker_version to '29.4.0'
envhook.py: Setting azul_terraform_version to '1.14.8'
envhook.py: Setting AZUL_DEBUG to '1'
envhook.py: Setting AZUL_ENABLE_REPLICAS to '1'
envhook.py: Setting AZUL_REPLICA_CONFLICT_LIMIT to '10'
envhook.py: Setting azul_docker_registry to ''
envhook.py: Setting azul_docker_images to '{"docker": {"ref": "docker.io/library/docker:29.4.0", "url": "https://hub.docker.com/_/docker"}, "python": {"ref": "docker.io/library/python:3.14.4-slim-trixie", "url": "https://hub.docker.com/_/python"}, "pycharm": {"ref": "docker.io/ucscgi/azul-pycharm:2025.2.6-78", "url": "https://hub.docker.com/repository/docker/ucscgi/azul-pycharm", "is_custom": true}, "opensearch": {"ref": "docker.io/opensearchproject/opensearch:2.19.5", "url": "https://hub.docker.com/r/opensearchproject/opensearch", "is_custom": false}, "bigquery_emulator": {"ref": "docker.io/ucscgi/azul-bigquery-emulator:0.4.4-61", "url": "https://hub.docker.com/repository/docker/ucscgi/azul-bigquery-emulator", "is_custom": true}, "clamav": {"ref": "docker.io/clamav/clamav:1.5.2-35", "url": "https://hub.docker.com/r/clamav/clamav"}, "gitlab": {"ref": "docker.io/gitlab/gitlab-ce:18.10.3-ce.0", "url": "https://hub.docker.com/r/gitlab/gitlab-ce"}, "gitlab_runner": {"ref": "docker.io/gitlab/gitlab-runner:ubuntu-v18.10.1", "url": "https://hub.docker.com/r/gitlab/gitlab-runner"}, "dind": {"ref": "docker.io/library/docker:29.4.0-dind", "url": "https://hub.docker.com/_/docker"}, "_signing_proxy": {"ref": "docker.io/cllunsford/aws-signing-proxy:0.2.2", "url": "https://hub.docker.com/r/cllunsford/aws-signing-proxy"}, "_cerebro": {"ref": "docker.io/lmenezes/cerebro:0.9.4", "url": "https://hub.docker.com/r/lmenezes/cerebro"}, "_opensearch_dashboards": {"ref": "docker.io/opensearchproject/opensearch-dashboards:2.19.3", "url": "https://hub.docker.com/r/opensearchproject/opensearch-dashboards"}}'
envhook.py: Setting AZUL_DSS_DIRECT_ACCESS to '0'
envhook.py: Setting AZUL_DRS_DOMAIN_NAME to 'drs.daniel.dev.singlecell.gi.ucsc.edu'
envhook.py: Setting AZUL_SUBDOMAIN_TEMPLATE to '*.daniel'
...
2026-04-14 11:23:08,276    INFO MainThread azul.docker: Resolving 'bigquery_emulator' image TagImageRef(registry='docker.io', username='ucscgi', repository=('azul-bigquery-emulator',), tag='0.4.4-61') …
2026-04-14 11:23:08,277    INFO MainThread azul.docker: Resolved 'bigquery_emulator' image to DigestImageRef(registry='docker.io', username='ucscgi', repository=('azul-bigquery-emulator',), digest='sha256:29727901fb7d3507f0cba2da0d3cb9e91a3ff6f6987b5199177b3a8ce65a2090')
2026-04-14 11:23:08,277    INFO MainThread azul.docker: Pulling image DigestImageRef(registry='docker.io', username='ucscgi', repository=('azul-bigquery-emulator',), digest='sha256:29727901fb7d3507f0cba2da0d3cb9e91a3ff6f6987b5199177b3a8ce65a2090') …
2026-04-14 11:23:11,481   DEBUG MainThread azul.docker: docker.io/ucscgi/azul-bigquery-emulator@sha256:29727901fb7d3507f0cba2da0d3cb9e91a3ff6f6987b5199177b3a8ce65a2090: docker pull {"status":"Pulling from ucscgi/azul-bigquery-emulator","id":"docker.io/ucscgi/azul-bigquery-emulator@sha256:29727901fb7d3507f0cba2da0d3cb9e91a3ff6f6987b5199177b3a8ce65a2090"}
2026-04-14 11:23:11,636   DEBUG MainThread azul.docker: docker.io/ucscgi/azul-bigquery-emulator@sha256:29727901fb7d3507f0cba2da0d3cb9e91a3ff6f6987b5199177b3a8ce65a2090: docker pull {"status":"Already exists","progressDetail":{},"id":"53196b1f47bd"}
2026-04-14 11:23:11,642   DEBUG MainThread azul.docker: docker.io/ucscgi/azul-bigquery-emulator@sha256:29727901fb7d3507f0cba2da0d3cb9e91a3ff6f6987b5199177b3a8ce65a2090: docker pull {"status":"Pulling fs layer","progressDetail":{},"id":"62c1d01a39ea"}
{"status":"Pulling fs layer","progressDetail":{},"id":"96239002c406"}
{"status":"Pulling fs layer","progressDetail":{},"id":"f42aa0cdbd1e"}
2026-04-14 11:23:12,059   DEBUG MainThread azul.docker: docker.io/ucscgi/azul-bigquery-emulator@sha256:29727901fb7d3507f0cba2da0d3cb9e91a3ff6f6987b5199177b3a8ce65a2090: docker pull {"status":"Downloading","progressDetail":{"current":199454,"total":19765070},"id":"62c1d01a39ea"}
2026-04-14 11:23:12,063   DEBUG MainThread azul.docker: docker.io/ucscgi/azul-bigquery-emulator@sha256:29727901fb7d3507f0cba2da0d3cb9e91a3ff6f6987b5199177b3a8ce65a2090: docker pull {"status":"Downloading","progressDetail":{"current":331909,"total":31789205},"id":"96239002c406"}
...
2026-04-14 11:23:13,467    INFO MainThread azul.docker: Pulled image DigestImageRef(registry='docker.io', username='ucscgi', repository=('azul-bigquery-emulator',), digest='sha256:29727901fb7d3507f0cba2da0d3cb9e91a3ff6f6987b5199177b3a8ce65a2090')
2026-04-14 11:23:13,470    INFO MainThread test.docker_container_test_case: Launching container from image sha256:c75bb8a68773336b57485ab100b6bc3c63a995af60442c1c4cca0c9befa007d8
2026-04-14 11:23:13,703    INFO MainThread test.docker_container_test_case: Launched (or reused) container wonderful_brahmagupta from image sha256:c75bb8a68773336b57485ab100b6bc3c63a995af60442c1c4cca0c9befa007d8 after 0.233s, with container port 9050 mapped to 127.0.0.1:54496 on the host
2026-04-14 11:23:14,135   DEBUG MainThread azul.plugins.repository.tdr_hca: Retrieving 1 entities of type 'links' ...
...


Ran 6 tests in 33.604s

OK

Process finished with exit code 0

[dsotirho-ucsc]

7934_IT_2026-04-14.txt

[hannes-ucsc]

Security design review

• Security design review completed; this PR does not …
  • … affect authentication; for example:
    • OAuth 2.0 with the application (API or Swagger UI)
    • Authentication of developers with Google Cloud APIs
    • Authentication of developers with AWS APIs
    • Authentication with a GitLab instance in the system
    • Password and 2FA authentication with GitHub
    • API access token authentication with GitHub
    • Authentication with Terra
  • … affect the permissions of internal users like access to
    • Cloud resources on AWS and GCP
    • GitLab repositories, projects and groups, administration
    • an EC2 instance via SSH
    • GitHub issues, pull requests, commits, commit statuses, wikis, repositories, organizations
  • … affect the permissions of external users like access to
    • TDR snapshots
  • … affect permissions of service or bot accounts
    • Cloud resources on AWS and GCP
  • … affect audit logging in the system, like
    • adding, removing or changing a log message that represents an auditable event
    • changing the routing of log messages through the system
  • … affect monitoring of the system
  • … introduce a new software dependency like
    • Python packages on PYPI
    • Command-line utilities
    • Docker images
    • Terraform providers
  • … add an interface that exposes sensitive or confidential data at the security boundary
  • … affect the encryption of data at rest
  • … require persistence of sensitive or confidential data that might require encryption at rest
  • … require unencrypted transmission of data within the security boundary
  • … affect the network security layer; for example by
    • modifying, adding or removing firewall rules
    • modifying, adding or removing security groups
    • changing or adding a port a service, proxy or load balancer listens on
• Documentation on any unchecked boxes is provided in comments below