Using azblob backend, pulumi stack ls runs quite slowly

[SirensOfTitan]

Hello!

• Vote on this issue by adding a 👍 reaction
• To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already)

Issue details

When using an azblob:// pulumi backend, pulumi stack ls command is quite slow, from five successive trials:

# first attempt takes 20s
pulumi stack ls  1.97s user 0.67s system 13% cpu 20.084 total
# second attempt 1m13s
pulumi stack ls  2.04s user 1.39s system 4% cpu 1:13.20 total
# third attempt 20s 
pulumi stack ls  1.87s user 0.66s system 12% cpu 19.735 total
# fourth attempt 20s 
pulumi stack ls  1.84s user 0.61s system 13% cpu 18.325 total
# fifth attempt 1m 36s
pulumi stack ls  2.20s user 1.66s system 4% cpu 1:36.01 total

About 20s is the floor, but I've seen this command take over 3 minutes before. Others on my team have seen similar behavior.

The storage account on Azure is a StorageV2 account with Standard/Hot access in the same location that my team and I live and work.

We're seeing a large disparity between E2E latency on Azure and server latency:

Red is E2E latency, and blue is server latency. We're all on fast, stable connections with low CPU and memory usage, so my feeling is that this is on the pulumi side, but I can't say for sure.

We have three mostly identical stacks stored in our backend, for one particular stack:

pulumi stack export | wc -c
11229149

Currently running pulumi version v3.22.1.

A particular pulumi stack ls network request downloads almost 40MB and it occurs very very slowly (very qualitative as I watched the network tab in Activity Monitor on macos). It seems like commands like pulumi stack ls could be sped up avoiding a full download of all of the state files.

Versions:

nodejs 17.3.1
typescript 4.4.4

# packages
    "@pulumi/azuread": "^5.13.0",
    "@pulumi/azure-native": "^1.53.0",
    "@pulumi/kubernetes": "^3.14.0",
    "@pulumi/kubernetesx": "^0.1.6",
    "@pulumi/pulumi": "^3.22.1",
    "@pulumi/random": "^4.3.1",
    "@pulumi/tls": "^4.1.0",

Steps to reproduce

Most of the details here are in the summary, if it would be helpful I can try to put together a simple test case.

Expected: pulumi stack ls command runs quickly
Actual: pulumi stack ls takes a long time to run.

[jitbasemartin]

I confirm, Azure backend is very slow. I also saw a lot of requests. I don't know if it's normal ?

[Frassle]

I don't know if it's normal ?

Yes, this is just how the blob storage backend works. I would expect a lot of traffic for ls currently.

[simonkarman]

I am running into the same issue. I simply want to know what the s3 backend is and the stack that is currently used by the Pulumi cli.

Currently I'm using these commands for that:

current_backend=$(pulumi whoami --verbose | grep 'Backend URL: ' | sed 's/Backend URL: //')
current_stack=$(pulumi stack --show-name)

However, these operation take approximately 7 seconds to complete, which is very very slow. Is this an issue that will get some attention?

---

Update:
I'm now using the following which are a few ms instead of multiple seconds long.

current_backend=$(jq --raw-output ".current" < "$HOME/.pulumi/credentials.json")
current_stack=$(jq --raw-output ".stack" < "$(ls "$HOME/.pulumi/workspaces/your-workspace-prefix-here"*)")

[SirensOfTitan]

@simonkarman: I wrote some little scripts a while back to quickly access stack in a generalized way:

get-pulumi-stack-file.sh:

#!/usr/bin/env bash

# This is relative to our git root for a project, you can adjust as necessary.
CFG_PATH="$(git rev-parse --show-toplevel)/pulumi/Pulumi.yaml"
WORKSPACE_HASH=$(echo -n "$CFG_PATH" | sha1sum | sed 's/ .*//')
WORKSPACE_NAME=$(grep name: "$CFG_PATH" | sed 's/name: //g')

echo -n "$HOME/.pulumi/workspaces/$WORKSPACE_NAME-$WORKSPACE_HASH-workspace.json"

current-stack-fast.sh:

#!/usr/bin/env bash

# adjust to refer to the script above:
WORKSPACE_FILE=$(./get-pulumi-stack-file)

if [ -f "$WORKSPACE_FILE" ]; then
   jq -r '.stack' "$WORKSPACE_FILE"
fi

Timing for my script: current-stack-fast 0.02s user 0.01s system 95% cpu 0.040 total
Timing for pulumi: pulumi stack --show-name 0.92s user 0.29s system 16% cpu 7.378 total

We use this primarily to switch the kubernetes context depending on the currently active pulumi stack, sharing here as it might also be useful:

update-k8s-creds.sh:

#!/usr/bin/env bash

CURRENT_STACK=$("$INFRA_SCRIPT_ROOT"/current-stack-fast)
KUBE_CONTEXT="ourcompany-${CURRENT_STACK}"

# Updates k8s creds to currently active stack
if ! kubectl config use-context "$KUBE_CONTEXT"; then
  "$INFRA_SCRIPT_ROOT"/get-k8s-creds
fi

get-k8s-creds.sh:

#!/usr/bin/env bash
#
if [ ! -x "$(command -v jq)" ]; then
  echo "Error: jq is not installed"
  exit 1
fi

if [ ! -x "$(command -v az)" ]; then
  echo "Error: az is not installed"
  exit 1
fi

if [ ! -x "$(command -v pulumi)" ]; then
  echo "Error: pulumi is not installed"
  exit 1
fi

STACK_OUTPUT=$(pulumi stack output --json)
CLUSTER_NAME=$(echo "$STACK_OUTPUT" | jq -r '.clusterName')
CURRENT_STACK=$(
  "$INFRA_SCRIPT_ROOT"/current-stack-fast
)
RG=$(echo "$STACK_OUTPUT" | jq -r '.resourceGroupName')

set -x
env -u KUBECONFIG az aks get-credentials \
  --name "$CLUSTER_NAME" \
  --resource-group "$RG" \
  --context "ourcompany-$CURRENT_STACK" \
  --overwrite-existing
set +x

We use this in conjunction with direnv.