Envbuilder, Gitea, External Git Authentication - Git auth works in container, but envbuilder cannot clone repo

[danhenrydev]

I'm struggling to make sense of external auth when using envbuilder.

The Issue:
I can get git authentication to work in envbuilder to pull a private repo from Gitea, and I can get git authentication to work properly in the workspace. I cannot however get git authentication to work properly for both.

When I use the environment variables to set the GIT_USERNAME and GIT_PASSWORD, envbuilder can successfully clone the private repository, however the token of course eventually expires so git eventually stops working properly in the workspace.

  env = [
...
    "GIT_USERNAME=oauth2",
    "GIT_PASSWORD=${data.coder_external_auth.gitea.access_token}",
    "GIT_URL=${data.coder_parameter.repo.value == "custom" ? data.coder_parameter.custom_repo_url.value : data.coder_parameter.repo.value}",
...
  ]

When I remove the GIT_USERNAME and GIT_PASSWORD environment variables, envbuilder is unable to authenticate so it reverts to the fallback image.

With the environment variables still commented out in the template, if I pull a public repository using the same template I am then able to clone the private repository from within the workspace as expected:

For clarity, every redacted git repository is the exact same private repository hosted on a self hosted Gitea server.

I am quite new to Coder, Envbuilder, and open source collaboration in general, and I have tried to be as verbose as possible in my issue. I assume this is an oversight on my part, or a fundamental misunderstanding of how I should be doing this and not an issue with Coder/Envbuilder.

Here is the complete Terraform file for the template.

terraform {
  required_providers {
    coder = {
      source = "coder/coder"
    }
    docker = {
      source = "kreuzwerker/docker"
    }
  }
}

data "coder_external_auth" "gitea" {
  # Matches the ID of the git auth provider in Coder.
  id = "gitea"
}

data "coder_provisioner" "me" {
}

provider "docker" {
}

data "coder_workspace" "me" {
}

resource "coder_agent" "main" {
  arch           = data.coder_provisioner.me.arch
  os             = "linux"
  startup_script = <<-EOT
    set -e
    git config --global credential.useHttpPath true

    # install and start code-server
    curl -fsSL https://code-server.dev/install.sh | sh -s -- --method=standalone --prefix=/tmp/code-server --version 4.11.0
    /tmp/code-server/bin/code-server --auth none --port 13337 >/tmp/code-server.log 2>&1 &
  EOT
  dir            = "/workspaces"
  
  # These environment variables allow you to make Git commits right away after creating a
  # workspace. Note that they take precedence over configuration defined in ~/.gitconfig!
  # You can remove this block if you'd prefer to configure Git manually or using
  # dotfiles. (see docs/dotfiles.md)
  env = {
    GIT_AUTHOR_NAME     = coalesce(data.coder_workspace.me.owner_name, data.coder_workspace.me.owner)
    GIT_AUTHOR_EMAIL    = "${data.coder_workspace.me.owner_email}"
    GIT_COMMITTER_NAME  = coalesce(data.coder_workspace.me.owner_name, data.coder_workspace.me.owner)
    GIT_COMMITTER_EMAIL = "${data.coder_workspace.me.owner_email}"


  }



  # The following metadata blocks are optional. They are used to display
  # information about your workspace in the dashboard. You can remove them
  # if you don't want to display any information.
  # For basic resources, you can use the `coder stat` command.
  # If you need more control, you can write your own script.
  metadata {
    display_name = "CPU Usage"
    key          = "0_cpu_usage"
    script       = "coder stat cpu"
    interval     = 10
    timeout      = 1
  }

  metadata {
    display_name = "RAM Usage"
    key          = "1_ram_usage"
    script       = "coder stat mem"
    interval     = 10
    timeout      = 1
  }

  metadata {
    display_name = "Home Disk"
    key          = "3_home_disk"
    script       = "coder stat disk --path $HOME"
    interval     = 60
    timeout      = 1
  }

  metadata {
    display_name = "CPU Usage (Host)"
    key          = "4_cpu_usage_host"
    script       = "coder stat cpu --host"
    interval     = 10
    timeout      = 1
  }

  metadata {
    display_name = "Memory Usage (Host)"
    key          = "5_mem_usage_host"
    script       = "coder stat mem --host"
    interval     = 10
    timeout      = 1
  }

  metadata {
    display_name = "Load Average (Host)"
    key          = "6_load_host"
    # get load avg scaled by number of cores
    script   = <<EOT
      echo "`cat /proc/loadavg | awk '{ print $1 }'` `nproc`" | awk '{ printf "%0.2f", $1/$2 }'
    EOT
    interval = 60
    timeout  = 1
  }

  metadata {
    display_name = "Swap Usage (Host)"
    key          = "7_swap_host"
    script       = <<EOT
      free -b | awk '/^Swap/ { printf("%.1f/%.1f", $3/1024.0/1024.0/1024.0, $2/1024.0/1024.0/1024.0) }'
    EOT
    interval     = 10
    timeout      = 1
  }
}

resource "coder_app" "code-server" {
  agent_id     = coder_agent.main.id
  slug         = "code-server"
  display_name = "code-server"
  url          = "http://localhost:13337/?folder=/workspaces"
  icon         = "/icon/code.svg"
  subdomain    = false
  share        = "owner"

  healthcheck {
    url       = "http://localhost:13337/healthz"
    interval  = 5
    threshold = 6
  }
}


resource "docker_volume" "workspaces" {
  name = "coder-${data.coder_workspace.me.id}"
  # Protect the volume from being deleted due to changes in attributes.
  lifecycle {
    ignore_changes = all
  }
  # Add labels in Docker to keep track of orphan resources.
  labels {
    label = "coder.owner"
    value = data.coder_workspace.me.owner
  }
  labels {
    label = "coder.owner_id"
    value = data.coder_workspace.me.owner_id
  }
  labels {
    label = "coder.workspace_id"
    value = data.coder_workspace.me.id
  }
  # This field becomes outdated if the workspace is renamed but can
  # be useful for debugging or cleaning out dangling volumes.
  labels {
    label = "coder.workspace_name_at_creation"
    value = data.coder_workspace.me.name
  }
}

data "coder_parameter" "repo" {
  name         = "repo"
  display_name = "Repository (auto)"
  order        = 1
  description  = "Select a repository to automatically clone and start working with a devcontainer."
  mutable      = true
  option {
    name        = "vercel/next.js"
    description = "The React Framework"
    value       = "https://github.com/vercel/next.js"
  }
  option {
    name        = "home-assistant/core"
    description = "🏡 Open source home automation that puts local control and privacy first."
    value       = "https://github.com/home-assistant/core"
  }
  option {
    name        = "discourse/discourse"
    description = "A platform for community discussion. Free, open, simple."
    value       = "https://github.com/discourse/discourse"
  }
  option {
    name        = "denoland/deno"
    description = "A modern runtime for JavaScript and TypeScript."
    value       = "https://github.com/denoland/deno"
  }
  option {
    name        = "microsoft/vscode"
    icon        = "/icon/code.svg"
    description = "Code editing. Redefined."
    value       = "https://github.com/microsoft/vscode"
  }
  option {
    name        = "Custom"
    icon        = "/emojis/1f5c3.png"
    description = "Specify a custom repo URL below"
    value       = "custom"
  }
}

data "coder_parameter" "custom_repo_url" {
  name         = "custom_repo"
  display_name = "Repository URL (custom)"
  order        = 2
  default      = ""
  description  = "Optionally enter a custom repository URL, see [awesome-devcontainers](https://github.com/manekinekko/awesome-devcontainers)."
  mutable      = true
}

resource "docker_container" "workspace" {
  count = data.coder_workspace.me.start_count
  # Find the latest version here:
  # https://github.com/coder/envbuilder/tags
  image = "ghcr.io/coder/envbuilder:0.2.9"
  # Uses lower() to avoid Docker restriction on container names.
  name = "coder-${data.coder_workspace.me.owner}-${lower(data.coder_workspace.me.name)}"
  # Hostname makes the shell more user friendly: coder@my-workspace:~$
  hostname = data.coder_workspace.me.name
  # Use the docker gateway if the access URL is 127.0.0.1
  
  env = [
    "CODER_AGENT_TOKEN=${coder_agent.main.token}",
    "CODER_AGENT_URL=${replace(data.coder_workspace.me.access_url, "/localhost|127\\.0\\.0\\.1/", "host.docker.internal")}",
    # "GIT_USERNAME=oauth2",
    # "GIT_PASSWORD=${data.coder_external_auth.gitea.access_token}",
    "GIT_URL=${data.coder_parameter.repo.value == "custom" ? data.coder_parameter.custom_repo_url.value : data.coder_parameter.repo.value}",
    "INIT_SCRIPT=${replace(coder_agent.main.init_script, "/localhost|127\\.0\\.0\\.1/", "host.docker.internal")}",
    "FALLBACK_IMAGE=codercom/enterprise-base:ubuntu" # This image runs if builds fail

  ]

  host {
    host = "host.docker.internal"
    ip   = "host-gateway"
  }
  volumes {
    container_path = "/workspaces"
    volume_name    = docker_volume.workspaces.name
    read_only      = false
  }
  # Add labels in Docker to keep track of orphan resources.
  labels {
    label = "coder.owner"
    value = data.coder_workspace.me.owner
  }
  labels {
    label = "coder.owner_id"
    value = data.coder_workspace.me.owner_id
  }
  labels {
    label = "coder.workspace_id"
    value = data.coder_workspace.me.id
  }
  labels {
    label = "coder.workspace_name"
    value = data.coder_workspace.me.name
  }
}

[kylecarbs]

Will look thoroughly in a bit. Thanks for being so descriptive in your issue!

[stauder]

I can confirm the same behavior but with github. To clone the private repo with the .envbuilder directory in does not work. But inside of a final container (I tested with the fallback image) it works to clone very same repo.

[stauder]

I could solve it in the meanwhile with this documentation (that was not obvious for me to find).

You already have configured your external auth:

data "coder_external_auth" "gitea" {
  # Matches the ID of the git auth provider in Coder.
  id = "gitea"
}

In my case for github (not sure for gitea) I had to add this:

resource "docker_container" "workspace" {
  env = [
    "GIT_USERNAME=${data.coder_external_auth.gitea.access_token}"
  ]
}

I also tried to set ENVBUILDER_GIT_USERNAME but this had no effect.

[matifali]

I also tried to set ENVBUILDER_GIT_USERNAME but this had no effect.

This change in environment variables names hasn't been released yet. This is the reason ENVBUILDER_GIT_USERNAME didn't work but GIT_USERNAME did.

[danhenrydev]

I could solve it in the meanwhile with this documentation (that was not obvious for me to find).

You already have configured your external auth:

data "coder_external_auth" "gitea" {
  # Matches the ID of the git auth provider in Coder.
  id = "gitea"
}

In my case for github (not sure for gitea) I had to add this:

resource "docker_container" "workspace" {
  env = [
    "GIT_USERNAME=${data.coder_external_auth.gitea.access_token}"
  ]
}

I also tried to set ENVBUILDER_GIT_USERNAME but this had no effect.

The problem with this solution is that once the access token expires git will no longer work within the workspace. I'm not sure how long Github's token is good for, but with my Gitea instance git stops working after an hour.

[matifali]

The problem with this solution is that once the access token expires git will no longer work within the workspace. I'm not sure how long Github's token is good for, but with my Gitea instance git stops working after an hour.

To solve this you should not inject the token into the workspace (inner container) and rely on Coder's external auth to mange git authentication.

https://coder.com/docs/admin/external-auth#native-git-authentication-will-auto-refresh-tokens

[danhenrydev]

The problem with this solution is that once the access token expires git will no longer work within the workspace. I'm not sure how long Github's token is good for, but with my Gitea instance git stops working after an hour.

To solve this you should not inject the token into the workspace (inner container) and rely on Coder's external auth to mange git authentication.

https://coder.com/docs/admin/external-auth#native-git-authentication-will-auto-refresh-tokens

Right, which is the problem. When I do it that way, envbuilder is not able to clone the repository.

[matifali]

Right, which is the problem. When I do it that way, envbuilder is not able to clone the repository.

For envbuilder you need to inject the environment variable. Envbuilder is running as the outer container which needs the token to clone the repo and build the devcontainer. We just should not add it into the built workspace.

---

Edit:
There is no nesting involved and hence no outer or inner container. envbuilder container builds the image as per the devcontainer spec and runs it.

[matifali]

I will request @Emyrk to explain this better.

[danhenrydev]

Right, which is the problem. When I do it that way, envbuilder is not able to clone the repository.

For envbuilder you need to inject the environment variable. Envbuilder is running as the outer container which needs the token to clone the repo and build the devcontainer. We just should not add it into the built workspace.

That makes sense to me, and I appreciate your help. What's still lost on me is how I would do that.

Thank you again for your time and help.

[Emyrk]

Right, which is the problem. When I do it that way, envbuilder is not able to clone the repository

Yea, this makes sense. When you do not inject the token, we set GIT_ASKPASS to use the coder executable to fetch the token on demand. This allows coderd to do any refreshing.

I am not familiar with env builder, is the coder executable present in the outer container? I feel it should be possible to make the outer container do the dynamic fetch.

[danhenrydev]

Right, which is the problem. When I do it that way, envbuilder is not able to clone the repository

Yea, this makes sense. When you do not inject the token, we set GIT_ASKPASS to use the coder executable to fetch the token on demand. This allows coderd to do any refreshing.

I am not familiar with env builder, is the coder executable present in the outer container? I feel it should be possible to make the outer container do the dynamic fetch.

I'm not sure how to answer this question properly, as I'm not sure I fully understand the concept of inner and outer containers.. The coder executable is for sure available within the workspace after it has launched, as I can use GIT_ASKPASS properly and it uses the coder executable. I suspect that this is the 'inner container' that the envbuilder container actually builds.

I think that would mean that the container from the envbuilder image is the 'outer container'

...
resource "docker_container" "workspace" {
  count = data.coder_workspace.me.start_count
  # Find the latest version here:
  # https://github.com/coder/envbuilder/tags
  image = "ghcr.io/coder/envbuilder:0.2.9"
...

Looking at the README for the envbuilder project
https://github.com/coder/envbuilder
It doesn't mention the coder executable.

Am I moving in the right direction here?

Thank you for your time, help, and patience.

[Emyrk]

@danhenrydev 100% correct in everything you said. I just talked with someone more familiar with enbuilder and he confirmed the coder executable is not present.

So you cannot use the auto refresh tokens in the envbuilder outer container. Only inner container (the workspace) dynamic tokens can be used.

---

EDIT: If this repo is the dev containers repo, then my suggestion below is moot 😢. I was under the impression this was some random git repo

One workaround suggestion I have is to not use GIT_URL. That instructs envbuilder to do a clone, which is done in golang, not git, to even further complicate this matter.

I have not tried this, so maybe @matifali can provide some more detailed instruction, but you could instead instruct the workspace agent (which is the workspace/inner container) to do the clone. There exists a module to make this even more simple: https://registry.coder.com/modules/git-clone

(I hope modules work with envbuilder 🤞)

[Emyrk]

@danhenrydev to clarify. Is the repo you are trying to clone defining a dev container? Or is it just a git repo you want in your working directory inside the workspace?