Skip to content

Docker credential helper (docker-credential-octopus) fails on podman workers, ~2.5 min delay per package before fallback #10128

Description

@RobMcCarther

Severity

low

Version

2026.3.113

Latest Version

None

What happened?

When the calamari-use-docker-credential-helper toggle is enabled, Docker/OCI feed package acquisition on a Linux worker running podman 3.4.4 via the podman-docker shim (not Docker Engine) fails to authenticate through docker-credential-octopus. Calamari retries login five times with linear backoff, then falls back to a plain docker login, which succeeds. The feature self-heals, but each affected package incurs ~2.5 minuts of added acquisition time from the retry sequence.

Impact

~2.5 min added latency per affected package (retry backoff: 150s).

The plaintext fallback defeats the security purpose of the helper (the encrypted-credential-store feature that #1981 was built to deliver).

podman-via-podman-docker is a common worker configuration, so blast radius likely extends beyond this one customer as the toggle rolls out.

Regression from the customer's perspective: an unpined weekly worker rebuild pulled Calamari 2026.3.113 with the toggle newly enabled, the same unchanged bootstrap script produced wrking plain docker login behaviour the prior week.

Reproduction

Configure a Linux polling worker using Podman (via the podman-docker shim) instead of Docker Engine.
Ensure the worker is running a Tentacle that downloads Calamari 2026.3.113 (or later with the docker-credential-octopus helper enabled).
Configure an Azure Container Registry (ACR) Docker feed in Octopus.
Create a deployment that acquires a container image from the ACR feed.
Run the deployment.

Error and Stacktrace

Calamari configures the docker-credential-octopus credential helper.
Docker login repeatedly fails with errors similar to:
Configured Docker credential helper for https://<registry>.azurecr.io
credentials not found in native keychain
Error: get credentials: 1 error occurred:
* error getting credentials - err: exit status 1
Login Failed

More Information

No response

Workaround

calamari-use-docker-credential-helper feature being turned off on customers instance as a workaround.

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugThis issue represents a verified problem we are committed to solving

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions