Skip to content

Docker instances created by Cloud Orchestrator enables vhost_user_vsock as true. #1721

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
0405ysj wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Docker instances created by Cloud Orchestrator enables vhost_user_vsock as true. #1721

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d2ae636
Canonical config supports vhost_user_vsock option
0405ysj Oct 20, 2025
236f47f
Host Orchestrator can specify default value of vhost user vsock option.
0405ysj Oct 21, 2025
d248bf8
Docker instances created by CO enables vhost user vsock as default
0405ysj Oct 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions base/cvd/cuttlefish/host/commands/cvd/cli/parser/instance/cf_vm_configs.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,12 @@ static std::string V4l2Proxy(const Instance& instance) {
return crosvm.has_v4l2_proxy() ? crosvm.v4l2_proxy() : default_val;
}

static std::string VhostUserVsock(const Instance& instance) {
const auto& crosvm = instance.vm().crosvm();
const auto& default_val = CF_DEFAULTS_VHOST_USER_VSOCK;
return crosvm.has_vhost_user_vsock() ? crosvm.vhost_user_vsock() : default_val;
}

static Result<std::optional<std::string>> CustomConfigsFlagValue(
const Instance& instance) {
if (instance.vm().custom_actions().empty()) {
Expand Down Expand Up @@ -155,6 +161,7 @@ Result<std::vector<std::string>> GenerateVmFlags(
GenerateInstanceFlag("enable_sandbox", cfg, EnableSandbox),
GenerateInstanceFlag("crosvm_simple_media_device", cfg, SimpleMediaDevice),
GenerateInstanceFlag("crosvm_v4l2_proxy", cfg, V4l2Proxy),
GenerateInstanceFlag("vhost_user_vsock", cfg, VhostUserVsock),
};
return MergeResults(std::move(flags), CF_EXPECT(CustomConfigsFlags(cfg)));
}
Expand Down
1 change: 1 addition & 0 deletions base/cvd/cuttlefish/host/commands/cvd/cli/parser/load_config.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,7 @@ message Crosvm {
optional bool enable_sandbox = 1;
optional bool simple_media_device = 2;
optional string v4l2_proxy = 3;
optional string vhost_user_vsock = 4;
}

message Gem5 {}
Expand Down
4 changes: 4 additions & 0 deletions docker/guest/run_services.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
#!/usr/bin/env bash

if [[ -n "$HANDLED_BY_CLOUD_ORCHESTRATOR" ]]; then
echo -n 'vhost_user_vsock="true"' >> /etc/default/cuttlefish-host_orchestrator
fi

service nginx start
service cuttlefish-host-resources start
service cuttlefish-operator start
Expand Down
4 changes: 4 additions & 0 deletions frontend/debian/cuttlefish-orchestration.cuttlefish-host_orchestrator.default
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,3 +28,7 @@ orchestrator_cvd_artifacts_dir=/var/lib/cuttlefish-common
# Indicates whether to use the GCE metadata to get the Build API credentials.
# Defaults to false.
# build_api_credentials_use_gce_metadata=
#
# Select a default option of vhost_user_vsock for launching CVD.
Copy link
Member

@ser-io ser-io Oct 21, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If vhost_user_vsock is part of the canonical configuration sent in the create cvd request why do we need this as part of the HO configuration as well?

Copy link
Collaborator Author

@0405ysj 0405ysj Oct 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I want to achieve is enabling vhost_user_vsock option for any HO in docker instance launched by CO. If vhost_user_vsock is disabled, we cannot launch CF instances across multiple docker instances. It's enabled on arm64, but it's disabled on x86_64.

When HO receives POST /cvds API request, it may or may not contain canonical configuration as input. If so, it calls cvd load to launch CF with given canonical configuration. Or, it calls cvd create to launch CF based on option flags. As cvdr create deals with both kinds via POST /cvds, HO in docker instance should be able to handle both approaches to achieve above one.

Copy link
Member

@ser-io ser-io Oct 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

POST /cvds without canonical config is deprecated, please do not add anything new to this path.

Having said that, the source of truth for cuttlefish creation in HO is the canonical config received in the request, this ensure reproducibility, I can copy/paste the config received by the HO and create the same device in my desktop using cvd create directly, this something I do on a regular basis when device create fails in remote hosts. By adding this config parameter as a HO service configuration you are disrupting this behavior adding a new source of configuration towards device creation breaking the HO device creation reproducibility capability.

If you need to alter the canonical config in Docker Deployment, please do that at the Cloud Orchestrator layer, where you can intercept the request and modify it, there's no need to add another source of configurations towards creating cuttlefish devices, canonical configurations should be the only one.

Copy link
Collaborator Author

@0405ysj 0405ysj Oct 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would take a look whether intercepting and modifying HTTP request from https://github.com/google/cloud-android-orchestration/blob/main/pkg/app/instances/hostclient.go by CO is applicable or not. If it's applicable, I think CO with DockerIM can adjust canonical config by itself to add vhost_user_vsock as true.

However, I still think I need to support enabling vhost_user_vsock via HO for POST /cvds without canonical config too, though it's deprecated. I'm understanding one of main flows at cvdr is cvdr create with --build_id or build_target options, but it still doesn't use canonical config at HO. Also, I'm not even sure whether canonical config supports build ID/target, which may require help from cvd fetch. I agree with deprecating all flows without canonical config from HO or cvdr, but I think enabling vhost_user_vsock should be orthogonal from the conversion task, not a blocker. HDYT? @ser-io

Copy link
Member

@ser-io ser-io Oct 23, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to make relevant changes to cvdr having commands like cvdr create --build_id --build_target using canonical config behind scenes, this is something I've been trying to do but don't have the time.

google/cloud-android-orchestration#374 is a reference of migrating to canonical config when using local artifacts.

# Defailts to ""
# vhost_user_vsock=
3 changes: 3 additions & 0 deletions frontend/debian/cuttlefish-orchestration.cuttlefish-host_orchestrator.init
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,9 @@ start_orchestrator() {
if [[ -n "${build_api_credentials_use_gce_metadata}" ]]; then
args+=("--build_api_credentials_use_gce_metadata=${build_api_credentials_use_gce_metadata}")
fi
if [[ -n "${vhost_user_vsock}" ]]; then
args+=("--vhost_user_vsock=${vhost_user_vsock}")
fi

start-stop-daemon --start \
--pidfile "${ORCHESTRATOR_PIDFILE}" \
Expand Down
2 changes: 2 additions & 0 deletions frontend/src/host_orchestrator/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,7 @@ func main() {
address := flag.String("listen_addr", DefaultListenAddress, "IP address to listen for requests.")
logFile := flag.String("log_file", "", "Path to file to write logs to.")
buildAPICredsUseGCEMetadata := flag.Bool("build_api_credentials_use_gce_metadata", false, "Indicates whether to use the GCE metadata to get the Build API credentials")
vhostUserVsock := flag.String("vhost_user_vsock", "", "Selected default option of vhost_user_vsock for launching CVD.")

flag.Parse()

Expand Down Expand Up @@ -179,6 +180,7 @@ func main() {
BuildAPICredentials: orchestrator.BuildAPICredentialsConfig{
UseGCEMetadata: *buildAPICredsUseGCEMetadata,
},
VhostUserVsock: *vhostUserVsock,
},
OperationManager: om,
WaitOperationDuration: 2 * time.Minute,
Expand Down
2 changes: 2 additions & 0 deletions frontend/src/host_orchestrator/orchestrator/controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ type Config struct {
Paths IMPaths
AndroidBuildServiceURL string
BuildAPICredentials BuildAPICredentialsConfig
VhostUserVsock string
}

type BuildAPICredentialsConfig struct {
Expand Down Expand Up @@ -255,6 +256,7 @@ func (h *createCVDHandler) Handle(r *http.Request) (interface{}, error) {
UserArtifactsDirResolver: h.UADirResolver,
FetchCredentials: creds,
BuildAPIBaseURL: h.Config.AndroidBuildServiceURL,
VhostUserVsock: h.Config.VhostUserVsock,
}
return NewCreateCVDAction(opts).Run()
}
Expand Down
53 changes: 53 additions & 0 deletions frontend/src/host_orchestrator/orchestrator/createcvdaction.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ package orchestrator

import (
"encoding/json"
"errors"
"io/ioutil"
"log"
"os"
Expand All @@ -38,6 +39,7 @@ type CreateCVDActionOpts struct {
UserArtifactsDirResolver UserArtifactsDirResolver
FetchCredentials cvd.FetchCredentials
BuildAPIBaseURL string
VhostUserVsock string
}

type CreateCVDAction struct {
Expand All @@ -51,6 +53,7 @@ type CreateCVDAction struct {
userArtifactsDirResolver UserArtifactsDirResolver
fetchCredentials cvd.FetchCredentials
buildAPIBaseURL string
VhostUserVsock string
}

func NewCreateCVDAction(opts CreateCVDActionOpts) *CreateCVDAction {
Expand All @@ -64,6 +67,7 @@ func NewCreateCVDAction(opts CreateCVDActionOpts) *CreateCVDAction {
userArtifactsDirResolver: opts.UserArtifactsDirResolver,
fetchCredentials: opts.FetchCredentials,
buildAPIBaseURL: opts.BuildAPIBaseURL,
VhostUserVsock: opts.VhostUserVsock,
execContext: execCtx,
cvdCLI: cvd.NewCLI(execCtx),
}
Expand Down Expand Up @@ -97,6 +101,9 @@ func (a *CreateCVDAction) launchCVD(op apiv1.Operation) {
}

func (a *CreateCVDAction) launchWithCanonicalConfig(op apiv1.Operation) (*apiv1.CreateCVDResponse, error) {
if err := a.applyVhostUserVsockOptionIfNeeded(); err != nil {
return nil, err
}
data, err := json.MarshalIndent(a.req.EnvConfig, "", " ")
if err != nil {
return nil, err
Expand Down Expand Up @@ -178,6 +185,7 @@ func (a *CreateCVDAction) launchFromAndroidCI(
startParams := startCVDParams{
InstanceCount: instancesCount,
MainArtifactsDir: targetDir,
VhostUserVsock: a.VhostUserVsock,
}
if buildSource.KernelBuild != nil {
startParams.KernelDir = targetDir
Expand Down Expand Up @@ -230,3 +238,48 @@ func createTempFile(pattern string, data string, mode os.FileMode) (*os.File, er
}
return file, nil
}

func (a *CreateCVDAction) applyVhostUserVsockOptionIfNeeded() error {
if a.VhostUserVsock == "" {
// Skip this function if VhostUserVsock option is not specified.
return nil
}
instances, exists := a.req.EnvConfig["instances"]
if !exists {
return nil
}
instancesArr, ok := instances.([]interface{})
if !ok {
return errors.New("unexpected type different from JSON array on 'instances'")
}
for _, ins := range instancesArr {
insMap, ok := ins.(map[string]interface{})
if !ok {
return errors.New("unexpected type different from JSON object on the member of 'instances'")
}
vm, exists := insMap["vm"]
if !exists {
vm = make(map[string]interface{})
insMap["vm"] = vm
}
vmMap, ok := vm.(map[string]interface{})
if !ok {
return errors.New("unexpected type different from JSON object on 'vm'")
}
crosvm, exists := vmMap["crosvm"]
if !exists {
crosvm = make(map[string]interface{})
vmMap["crosvm"] = crosvm
}
crosvmMap, ok := crosvm.(map[string]interface{})
if !ok {
return errors.New("unexpected type different from JSON object on 'crosvm'")
}
if _, exists := crosvmMap["vhost_user_vsock"]; !exists {
// Specify VhostUserVsock option only when canonical config doesn't
// specify any options.
crosvmMap["vhost_user_vsock"] = a.VhostUserVsock
}
}
return nil
}
4 changes: 4 additions & 0 deletions frontend/src/host_orchestrator/orchestrator/cvd/cvd.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -537,6 +537,7 @@ type StartOptions struct {
InitramfsImage string
BootloaderRom string
ReportUsageStats bool
VhostUserVsock string
}

func (o *StartOptions) toArgs() []string {
Expand All @@ -558,6 +559,9 @@ func (o *StartOptions) toArgs() []string {
} else {
args = append(args, "--report_anonymous_usage_stats=n")
}
if o.VhostUserVsock != "" {
args = append(args, "--vhost_user_vsock", o.VhostUserVsock)
}
return args
}

Expand Down
3 changes: 3 additions & 0 deletions frontend/src/host_orchestrator/orchestrator/instancemanager.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,8 @@ type startCVDParams struct {
KernelDir string
// OPTIONAL. If set, bootloader relevant artifacts will be pulled from this dir.
BootloaderDir string
// OPTIONAL.
VhostUserVsock string
}

func CreateCVD(ctx hoexec.ExecContext, p startCVDParams) (*cvd.Group, error) {
Expand All @@ -169,6 +171,7 @@ func CreateCVD(ctx hoexec.ExecContext, p startCVDParams) (*cvd.Group, error) {

startOpts := cvd.StartOptions{
ReportUsageStats: reportAnonymousUsageStats,
VhostUserVsock: p.VhostUserVsock,
}
if p.KernelDir != "" {
startOpts.KernelImage = fmt.Sprintf("%s/bzImage", p.KernelDir)
Expand Down
Loading