Basically I was trying to drop --privileged with more fine grained isolation, but I couldn't easily figure out how to get the right fine grained enough args that it'd work in both codespaces/vscode etc.
See cgwalters/devaipod@9bc6978 - basically here we cut over to using --privileged by default in our .devcontainer.json but also add the special devaipod flag.