Skip to content

flowey: split openvmm-deps into separate archives and use Linux 6.18 test kernel #3530

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
moor-coding wants to merge 4 commits into
base: main
Choose a base branch
from
+57 −58
Open

flowey: split openvmm-deps into separate archives and use Linux 6.18 test kernel #3530

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ef10d04
flowey: split openvmm-deps into separate archives and add Linux 6.18 …
moor-coding May 19, 2026
2e275c4
flowey: add Linux 6.18 test kernel and set as default
moor-coding May 20, 2026
3aea3ee
fix(flowey): use tar.gz extraction for openvmm-deps archives
moor-coding May 20, 2026
690899a
refactor(flowey): remove single-variant SourceArchive enum
moor-coding May 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion flowey/flowey_lib_hvlite/src/_jobs/cfg_versions.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ pub const NODEJS: &str = "24.x";
// increases with each release from the respective branch.
pub const OPENHCL_KERNEL_DEV_VERSION: &str = "6.18.0.2";
pub const OPENHCL_KERNEL_STABLE_VERSION: &str = "6.18.0.2";
pub const OPENVMM_DEPS: &str = "0.3.0-29";
pub const OPENVMM_DEPS: &str = "0.3.0-33";
pub const PROTOC: &str = "27.1";
Comment thread
moor-coding marked this conversation as resolved.

flowey_request! {
Expand Down
105 changes: 53 additions & 52 deletions flowey/flowey_lib_hvlite/src/resolve_openvmm_deps.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,16 @@
// Licensed under the MIT License.

//! Download various pre-built `openvmm-deps` dependencies, or use a local path if specified.
//!
//! The openvmm-deps release publishes separate archives:
//! - `openvmm-deps.{arch}.{ver}.tar.gz` — SDK tools (dbgrd, shell, sysroot, petritools)
//! - `openvmm-test-initrd.{arch}.{ver}.tar.gz` — shared test initrd
//! - `openvmm-test-linux-{kernel_ver}.{arch}.{ver}.tar.gz` — test kernel
Comment on lines 4 to +9

use crate::common::CommonArch;
use flowey::node::prelude::*;
use std::collections::BTreeMap;
use std::collections::BTreeSet;

/// Which file to extract from the openvmm-deps archive.
#[derive(Serialize, Deserialize, Copy, Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
Expand Down Expand Up @@ -87,9 +93,6 @@ impl FlowNodeWithConfig for Node {
return Ok(());
}

// Which architectures have at least one dep requested?
let needs_arch = |arch: CommonArch| deps.keys().any(|(_, a)| *a == arch);

if !local_paths.is_empty() {
ctx.emit_rust_step("use local openvmm-deps", |ctx| {
let deps = deps.claim(ctx);
Expand Down Expand Up @@ -118,67 +121,65 @@ impl FlowNodeWithConfig for Node {
return Ok(());
}

let extract_tar_gz_persistent_dir = ctx.persistent_dir();

let download_archive = |arch: CommonArch, ctx: &mut NodeCtx<'_>| {
let version = version.clone().expect("local requests handled above");
let arch_str = match arch {
CommonArch::X86_64 => "x86_64",
CommonArch::Aarch64 => "aarch64",
};
ctx.reqv(|v| flowey_lib_common::download_gh_release::Request {
repo_owner: "microsoft".into(),
repo_name: "openvmm-deps".into(),
needs_auth: false,
tag: version.clone(),
file_name: format!("openvmm-deps.{arch_str}.{version}.tar.gz"),
path: v,
})
};
let version = version.expect("local requests handled above");

// Determine which architectures we need to download.
let needed_archs: BTreeSet<CommonArch> = deps.keys().map(|(_, arch)| *arch).collect();

let persistent_dir = ctx.persistent_dir();

let openvmm_deps_tar_gz_x64 =
needs_arch(CommonArch::X86_64).then(|| download_archive(CommonArch::X86_64, ctx));
let openvmm_deps_tar_gz_aarch64 =
needs_arch(CommonArch::Aarch64).then(|| download_archive(CommonArch::Aarch64, ctx));
// Download each unique architecture.
let downloads: BTreeMap<CommonArch, ReadVar<PathBuf>> = needed_archs
.into_iter()
.map(|arch| {
let arch_str = match arch {
CommonArch::X86_64 => "x86_64",
CommonArch::Aarch64 => "aarch64",
};
let file_name = format!("openvmm-deps.{arch_str}.{version}.tar.gz");
let path = ctx.reqv(|v| flowey_lib_common::download_gh_release::Request {
repo_owner: "microsoft".into(),
repo_name: "openvmm-deps".into(),
needs_auth: false,
tag: version.clone(),
file_name,
path: v,
});
(arch, path)
})
.collect();

ctx.emit_rust_step("unpack openvmm-deps archive", |ctx| {
Comment thread
moor-coding marked this conversation as resolved.
Comment thread
moor-coding marked this conversation as resolved.
let extract_tar_gz_persistent_dir = extract_tar_gz_persistent_dir.claim(ctx);
let openvmm_deps_tar_gz_x64 = openvmm_deps_tar_gz_x64.claim(ctx);
let openvmm_deps_tar_gz_aarch64 = openvmm_deps_tar_gz_aarch64.claim(ctx);
let persistent_dir = persistent_dir.claim(ctx);
let downloads: BTreeMap<_, _> = downloads
.into_iter()
.map(|(key, var)| (key, var.claim(ctx)))
.collect();
let deps = deps.claim(ctx);
let version = version.clone().expect("local requests handled above");
let version = version.clone();
move |rt| {
let persistent_dir = extract_tar_gz_persistent_dir.map(|d| rt.read(d));
let extract_dir_x64 = openvmm_deps_tar_gz_x64
.map(|file| {
let file = rt.read(file);
flowey_lib_common::_util::extract::extract_tar_gz_if_new(
rt,
persistent_dir.as_deref(),
&file,
&version,
)
})
.transpose()?;
let extract_dir_aarch64 = openvmm_deps_tar_gz_aarch64
.map(|file| {
let file = rt.read(file);
flowey_lib_common::_util::extract::extract_tar_gz_if_new(
let persistent_dir = persistent_dir.map(|d| rt.read(d));

// Extract each downloaded archive, keyed by architecture.
let extract_dirs: BTreeMap<CommonArch, PathBuf> = downloads
.into_iter()
.map(|(arch, var)| {
let file = rt.read(var);
let dir = flowey_lib_common::_util::extract::extract_tar_gz_if_new(
rt,
persistent_dir.as_deref(),
&file,
&version,
)
)?;
Comment thread
moor-coding marked this conversation as resolved.
Ok((arch, dir))
})
.transpose()?;

let base_dir = |arch| match arch {
CommonArch::X86_64 => extract_dir_x64.clone().unwrap(),
CommonArch::Aarch64 => extract_dir_aarch64.clone().unwrap(),
};
.collect::<anyhow::Result<_>>()?;

for ((dep, arch), vars) in deps {
let path = base_dir(arch).join(dep.filename());
let extract_dir = extract_dirs
.get(&arch)
.expect("archive was downloaded for this arch");
let path = extract_dir.join(dep.filename());
rt.write_all(vars, &path)
}

Expand Down
8 changes: 3 additions & 5 deletions flowey/flowey_lib_hvlite/src/resolve_openvmm_test_linux_kernel.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,10 @@ use std::collections::BTreeSet;

/// Which Linux test kernel version to fetch from the openvmm-deps GitHub
/// release.
///
/// The `openvmm-deps` release currently only ships the 6.1 kernel; additional
/// kernel lines (e.g. 6.6, 6.12) are intended to be added as purely additive
/// follow-ups, both upstream and as new variants of this enum.
#[derive(Serialize, Deserialize, Copy, Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub enum LinuxTestKernelVersion {
Linux6_1,
Copy link
Copy Markdown
Contributor

@smalis-msft smalis-msft May 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this is more a maintainer question than a question for this PR, but do we still want to use 6.1 for anything or can we just delete it?

Copy link
Copy Markdown
Contributor Author

@moor-coding moor-coding May 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like we have one test that is not working with the 6.18 kernel. Looking at the lift to have that one test using the prior kernel version

Root cause: The Linux 6.18 MANA/GDMA driver crashes with a NULL pointer dereference in mana_gd_probe() when probing the
second GDMA device. The 6.18 driver sends an SMC request (0x0) that the OpenVMM GDMA emulator doesn't support, and the
driver doesn't handle the failure gracefully — it dereferences a NULL pointer at offset 0x28.

Copy link
Copy Markdown
Contributor

@smalis-msft smalis-msft May 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personally would be ok with deleting 6.1 entirely and marking that one test as unstable to get this in, but let's check with some mana folks first to see if they think it would be an easy fix. Either way I think 6.1 should go away.

Linux6_18,
}

impl LinuxTestKernelVersion {
Expand All @@ -36,6 +33,7 @@ impl LinuxTestKernelVersion {
pub fn artifact_tag(self) -> &'static str {
match self {
Self::Linux6_1 => "6.1",
Self::Linux6_18 => "6.18",
}
}
}
Expand Down Expand Up @@ -73,7 +71,7 @@ impl OpenvmmTestKernelFile {
/// The default Linux test kernel version. Call sites that don't otherwise care
/// which kernel they're using should pass this.
pub const DEFAULT_LINUX_TEST_KERNEL_VERSION: LinuxTestKernelVersion =
LinuxTestKernelVersion::Linux6_1;
LinuxTestKernelVersion::Linux6_18;

flowey_config! {
/// Config for the resolve_openvmm_test_linux_kernel node.
Expand Down
Loading