GitHub Codespace entered recovery mode and persistent workspace files disappeared

[cesarsan]

🏷️ Discussion Type

Bug

Body

Hello GitHub Community,

I'm experiencing a serious GitHub Codespaces recovery issue and would appreciate guidance from anyone who has encountered something similar.

Codespace details

• Repository: cesarsan/sistemas-dev
• Codespace: redesigned-couscous-7r9569vr9xcpj7w
• Branch: develop

What happened

My Codespace failed to start its development container and automatically entered recovery mode.

The creation logs included:

• unable to find user codespace: no matching entries in passwd file
• Failed to create container
• Creating recovery container

The recovery environment starts, but the original working directories are missing.

Previously, the Codespace contained several repositories and substantial uncommitted work, including:

• /workspaces/sistemas-dev
• /workspaces/awp
• /workspaces/compliance

Currently, /workspaces/sistemas-dev is empty, while /workspaces/awp and /workspaces/compliance are missing.

The mounted /workspaces filesystem appears almost empty, despite the Codespace previously containing extensive development work.

What I have checked

• The Codespace still exists in GitHub and is shown as active.
• The recovery container starts successfully.
• The expected Git repositories are not present in /workspaces.
• Closing and reopening the Codespace did not restore the directories.
• I have not intentionally deleted or reset the Codespace.

My main concern

Could recovery mode be mounting a newly initialized filesystem instead of the original persistent volume?

Is there a safe way to recover the original Codespace filesystem, including uncommitted files, without rebuilding or recreating the Codespace?

I have already opened GitHub Support ticket #4847245 and am preserving the existing Codespace without rebuilding or deleting it.

Any guidance on diagnosing this safely would be greatly appreciated.

Thank you.

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐

[cesarsan]

Thank you very much for your help. I followed up on your suggestions and wanted to share the results.

1. Checking /workspaces

I inspected the recovery container:

• /workspaces/sistemas-dev exists but is empty.
• /workspaces/awp is missing.
• /workspaces/compliance is missing.
• A previous search for Git repositories under /workspaces found none.

These repositories were previously located under /workspaces, not in the container's home directory.

2. Checking the Dev Container configuration

On the develop branch, devcontainer.json does not specify remoteUser or containerUser.

The Dockerfile uses:

FROM mcr.microsoft.com/devcontainers/python:3.12-bullseye

This image normally uses the vscode user.

However, the creation log shows that GitHub attempted to restart an existing container and failed with:

unable to find user codespace: no matching entries in passwd file

The recovery container subsequently started successfully as vscode.

3. Checking the persistent storage

I inspected the current mount:

/codespacemount/workspace /workspaces rw,nodev,relatime master:256 - ext4 /dev/loop4 rw

The backing file for /dev/loop4 is:

/mnt/cloudenvdata/dockerlib

I still cannot determine whether this is the original persistent storage or a different filesystem state.

4. Rebuilding and backing up

I have not performed a rebuild because /workspaces is missing the original files, and I want to avoid any operation that could compromise recovery.

I also cannot run git stash or export the missing files using gh codespace cp, since the repositories and uncommitted files are not accessible in the recovery container.

The Codespace stopped unexpectedly while an AI coding agent was working. I have not yet found evidence that the agent modified the container configuration.

I have an open GitHub Support ticket (#4847245), but no response yet.

Given these results, do you think the codespace user error could be caused by GitHub's internal container configuration? More importantly, is there a safe way to verify whether the original workspace storage is still attached, without rebuilding the container?

I would really appreciate any additional ideas. Thank you again!

[mohittguptaa]

Don't delete the codespace, and don't create a new one. Your files are most likely still on the codespace's disk; the recovery container just isn't the same environment.

Why it happened: unable to find user codespace: no matching entries in passwd file means the dev container config sets remoteUser/containerUser to codespace, but the image it builds from doesn't have that user. That's typical after switching from the default universal image to a custom or mcr.microsoft.com/devcontainers/* image, where the user is vscode or node.

Find your files from the recovery container:

ls -la /workspaces
sudo find / -maxdepth 4 -type d -name ".git" 2>/dev/null | grep -v "^/proc"
df -h | grep -i workspaces

Everything under /workspaces lives on the persistent disk and survives rebuilds. If you cloned extra repos outside /workspaces (for example into ~), those were inside the old container and are the parts at risk.

Fix the config and rebuild:

1. In the recovery container, edit .devcontainer/devcontainer.json and change "remoteUser": "codespace" to the user your image actually has (often "vscode"), or remove the line.
2. Run Codespaces: Rebuild Container (not "Full Rebuild"). A normal rebuild keeps /workspaces.

Before rebuilding, back up anything uncommitted you can reach:

cd /workspaces/<repo> && git stash -u && git stash show -p > /tmp/backup.patch

From your machine, gh codespace cp -e 'remote:/workspaces/<repo>/<file>' . copies files out.

If /workspaces really is empty, open a support ticket right away with the codespace name. GitHub keeps the disk for a while, but deleting the codespace removes it for good.