IAP Cloud Build: SSH Permission Denied

[oinkbark]

Overview of the Issue

I am trying to use Google Cloud Build to create Packer images, but with the machines being configured to have no external IP. The IAP options seem perfect for this, but I have yet to get a standard build to finish with those options being set. It always fails when trying to establish the SSH connection.

I have verified that it is not a permissions issue with the firewall rules or the service account IAM permissions as I can impersonate it on the CLI and there is no problem connecting through SSH to the private machines.

Reproduction Steps

1. Prepare a Google Cloud Build run, with the relevant steps to execute Packer with this plugin (example below).
2. Supply Packer with the config file described below (I do it through a cloud source repo).
3. Assign the proper permissions to the cloud build service account (I use a custom one). Most notable is IAP-secured Tunnel User but others will probably be required.
4. Configure your VPC firewall rules to allow the IAP CIDR (35.235.240.0/20) into all the instances in the VPC.
5. Trigger the build step and see it hang and timeout on this message: "Waiting for SSH to become available...".

Plugin and Packer version

Plugin: 1.0.13
Packer: Docker image, light-1.8.2

Simplified Packer Buildfile

Note: I have tried with a dynamic and static IAP localhost port, with no setting making a difference.

source "googlecompute" "base-deb11" {
  source_image = "a custom Debian 11 image"
  enable_secure_boot = true
  enable_vtpm = true
  enable_integrity_monitoring = true
  
  network = "network"
  subnetwork = "subnet"
  omit_external_ip = true
  use_internal_ip = true
  use_iap = true
  # iap_localhost_port = 22

  service_account_email = "cloud-build@${var.project_id}.iam.gserviceaccount.com"
  impersonate_service_account = "cloud-build@${var.project_id}.iam.gserviceaccount.com"

  temporary_key_pair_type = "ed25519"
  ssh_username = "auto-replaced"
  use_os_login = false
}

Operating system and Environment details

• Running in Google Cloud Build. I have tried both the default worker space and private worker pools.
• VM image is a lightly modified version of Debian 11.3. The firewall allows SSH (port 22) and works fine when an external IP is attached, so this is also not the issue.

source:
  repoSource:
    projectId: projects/redacted
    repoName: redacted
    branchName: base-deb11
    dir: base-deb11
steps:
  - name: 'hashicorp/packer:light'
    args:
      - '-c'
      - packer init base-deb11.gcloud.pkr.hcl
    dir: /workspace/base-deb11
    entrypoint: bash
  - name: 'hashicorp/packer:light'
    env:
      - PACKER_LOG=1
      - PACKER_LOG_PATH=/workspace/packer.log
    args:
      - '-c'
      - packer build base-deb11.gcloud.pkr.hcl
    dir: /workspace/base-deb11
    entrypoint: bash

Log Fragments and crash.log files

These are the relevant lines from packer.log, with the entire gist still being linked:

2022/06/30 00:04:08 packer-plugin-googlecompute_v1.0.13_x5.0_linux_amd64 plugin: 2022/06/30 00:04:08 Finding an available TCP port for IAP proxy
2022/06/30 00:04:08 packer-plugin-googlecompute_v1.0.13_x5.0_linux_amd64 plugin: 2022/06/30 00:04:08 Found available port: 8290 on IP: 0.0.0.0
2022/06/30 00:04:08 packer-plugin-googlecompute_v1.0.13_x5.0_linux_amd64 plugin: 2022/06/30 00:04:08 Setting up proxy to listen on localhost at 8290
2022/06/30 00:04:08 packer-plugin-googlecompute_v1.0.13_x5.0_linux_amd64 plugin: 2022/06/30 00:04:08 Creating tunnel launch script with args []string{"compute", "start-iap-tunnel", "base-deb11", "22", "--local-host-port=localhost:8290", "--zone", "redacted", "--project", "redacted", "--impersonate-service-account='cloud-build@redacted.iam.gserviceaccount.com'"}
2022/06/30 00:04:08 packer-plugin-googlecompute_v1.0.13_x5.0_linux_amd64 plugin: 2022/06/30 00:04:08 [INFO] (google): Prepending inline gcloud setup script with #!/bin/sh
2022/06/30 00:04:09 packer-plugin-googlecompute_v1.0.13_x5.0_linux_amd64 plugin: 2022/06/30 00:04:09 stderr: /tmp/gcloud-setup3357350656: line 2: gcloud: not found
2022/06/30 00:04:38 packer-plugin-googlecompute_v1.0.13_x5.0_linux_amd64 plugin: 2022/06/30 00:04:38 No error detected after tunnel launch; continuing...
2022/06/30 00:04:38 packer-plugin-googlecompute_v1.0.13_x5.0_linux_amd64 plugin: 2022/06/30 00:04:38 Using host value: localhost
2022/06/30 00:04:38 ui: �[1;32m==> googlecompute.base-deb11: Using SSH communicator to connect: localhost�[0m
2022/06/30 00:04:38 packer-plugin-googlecompute_v1.0.13_x5.0_linux_amd64 plugin: 2022/06/30 00:04:38 [INFO] Waiting for SSH, up to timeout: 5m0s
2022/06/30 00:04:38 ui: �[1;32m==> googlecompute.base-deb11: Waiting for SSH to become available...�[0m
2022/06/30 00:04:38 packer-plugin-googlecompute_v1.0.13_x5.0_linux_amd64 plugin: 2022/06/30 00:04:38 Using host value: localhost
2022/06/30 00:04:38 packer-plugin-googlecompute_v1.0.13_x5.0_linux_amd64 plugin: 2022/06/30 00:04:38 [DEBUG] TCP connection to SSH ip/port failed: dial tcp 127.0.0.1:8290: connect: connection refused

Potential Workarounds

Internally, it appears that this plugin is using this command, which is causing the issue:

gcloud compute start-iap-tunnel <instance> 22 --local-host-port=localhost:<locahost port> --zone <zone> --project <project> --impersonate-service-account='<service account>'

I have been able to get the build to work using these two alternatives:

1. Using tunnel-though

gcloud compute ssh --tunnel-through-iap <instance> --project <project> --zone <zone>

2. Using private worker pools. This replaces IAP (ie you disable it) and makes the cloud build node apart of a private worker pool that directly has access to the local IPs of the machines that it creates. This requires some extra firewall rules, IAM permissions, peering, and private service connections.

[nywilken]

Hi @oinkbark apologies for the delayed response, and thank you for the provided logs. Looking at the logs I see the error "gcloud not found". Which means that Packer is unable to access the Google Cloud CLI gcloud.

Do you know if the host machine running the Packer build has the gcloud CLI installed?

Actually, it looks like you are running Packer as a Docker container. The Packer container does not contain the Google SDK, which would explain the gcloud not found error. I don't believe you will be able to access the installed gcloud CLI from the runner as it is outside of the Packer container. The path forward here would be to install and run Packer directly on the runner after installing the Google Cloud CLI.

I have to look into Cloud Build to understand the options for installing and running Packer. But to get this working with IAP Packer needs access to the gcloud executable.