Skip to content

Merge master to feature/trusted-certs#7010

Merged
minglumlu merged 87 commits intoxapi-project:feature/trusted-certsfrom
minglumlu:private/mingl/feature/trusted-certs
Apr 15, 2026
Merged

Merge master to feature/trusted-certs#7010
minglumlu merged 87 commits intoxapi-project:feature/trusted-certsfrom
minglumlu:private/mingl/feature/trusted-certs

Conversation

@minglumlu
Copy link
Copy Markdown
Member

@minglumlu minglumlu commented Apr 14, 2026

commit 4da9ef6bddeed813a9687d51116062ae9914c97a
Merge: 1c4938316 043e6bde3
Author: Ming Lu <ming.lu@cloud.com>
Date:   Wed Apr 15 09:56:22 2026 +0800

    Merge branch 'master' into private/mingl/feature/trusted-certs

diff --cc ocaml/idl/datamodel_common.ml
index 93a3fbc6f,0fe08b11c..25e689f4f
--- a/ocaml/idl/datamodel_common.ml
+++ b/ocaml/idl/datamodel_common.ml
@@@ -3,21 -3,21 +3,21 @@@
  open Datamodel_types
  open Lifecycle
  open Datamodel_roles
  
  (* IMPORTANT: Please bump schema vsn if you change/add/remove a _field_.
                You do not have to bump vsn if you change/add/remove a message
                When introducing a new release, bump the schema minor version to the next hundred
                to leave a gap for potential hotfixes needing to increment the schema version.*)
  let schema_major_vsn = 5
  
- let schema_minor_vsn = 794
 -let schema_minor_vsn = 901
++let schema_minor_vsn = 902
  
  (* Historical schema versions just in case this is useful later *)
  let rio_schema_major_vsn = 5
  
  let rio_schema_minor_vsn = 19
  
  let miami_release_schema_major_vsn = 5
  
  let miami_release_schema_minor_vsn = 35
  
diff --cc ocaml/idl/schematest.ml
index e92f9e944,c963c8f11..a7fc9bd13
--- a/ocaml/idl/schematest.ml
+++ b/ocaml/idl/schematest.ml
@@@ -1,16 -1,16 +1,16 @@@
  let hash x = Digest.string x |> Digest.to_hex
  
  (* BEWARE: if this changes, check that schema has been bumped accordingly in
     ocaml/idl/datamodel_common.ml, usually schema_minor_vsn *)
  
- let last_known_schema_hash = "ce90c659723cbcd5265e4dd856802b74"
 -let last_known_schema_hash = "32bbba07579ca8844fa6162164530268"
++let last_known_schema_hash = "a64e7e1133a10add1b39d22cd6bbc352"
  
  let current_schema_hash : string =
    let open Datamodel_types in
    let hash_of_obj x =
      List.map rpc_of_content x.contents
      |> List.map Jsonrpc.to_string
      |> String.concat ""
      |> hash
    in
    Datamodel.all_system |> List.map hash_of_obj |> String.concat ":" |> hash

edwintorok and others added 30 commits March 13, 2026 16:05
Also introduce a batching delay on all Event.{from,next} calls, except
Event.from calls for tasks. This should help batch up multiple events from
field updates.

There are no extra delays for tasks, because these typically aren't immediately ready,
and when they are we want to immediately find out about it.

This is part of the "feature flag" that turns on the feature
(previously you could've manually edited xapi.conf to make these changes.)

Signed-off-by: Edwin Török <edwin.torok@cloud.com>
Signed-off-by: Christian Lindig <christian.lindig@citrix.com>
Signed-off-by: Christian Lindig <christian.lindig@citrix.com>

Signed-off-by: Christian Lindig <christian.lindig@citrix.com>
This will enable short-circuiting internal API calls.

When the FD is missing mark the call as Internal.

Signed-off-by: Edwin Török <edwin.torok@cloud.com>
This picks up an earlier pull request by @edwintorok that was abandoned.
We need to be sure that the code we are calling it thread safe.
…#6881)

Signed-off-by: Christian Lindig <christian.lindig@citrix.com>
Removing this device class from the USB config would let admins
configure VMs for digital signing purposes.

These devices are also not very useful in Dom0.

Signed-off-by: Tu Dinh <ngoc-tu.dinh@vates.tech>
Quicktest was listing potentially thousands of VDIs and querying its
properties, just to find out whether the resulting list was empty or not.

We don't need to produce the full list to check whether it is empty or not: we
can stop once we have the 1st element. Replace the List with a Seq.

Before:
```
time ./quicktest.exe  -run-only Quicktest_vm_lifecycle
[...]
real    0m13.957
```

After:
```
real    0m12.584s
```

Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Some (read-only) SRs can contain a large number of VDIs,
but these cannot "leak" VDIs if we cannot create or destroy them.

Before:
```
real    0m12.584s
```

After:
```
real    0m7.853s
```

Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Before:
```
real 0m7.853s
```

After:
```
real    0m7.746s
```

Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Before:
```
real    0m7.746s
```

After:
```
real    0m6.095s
```

Signed-off-by: Edwin Török <edwin.torok@citrix.com>
The difference is not really measurable.
But I found out there is another pre-existing flag that can speed it up,
`-default-sr`. Then the leak checks are only done on that SR, and not all the
SRs.

So the best way is to run this (at least during development):
```
time ./quicktest.exe -run-only Quicktest_vm_lifecycle -skip-stress -default-sr
[...]
real 0m1.709s
```

Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Removing this device class from the USB config would let admins
configure VMs for digital signing purposes.

These devices are also not very useful in Dom0.
Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Instead of xmlrpc

Signed-off-by: Edwin Török <edwin.torok@citrix.com>
…forward the API call directly like we do with the CLI for calls to the coordinator when we are the coordinator

Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Instead of parsing the string backtrace, use the record that is available since OCaml 4.02.
This is more future-proof, and performance is also improved ~10x:

Before:
```
Backtrace.is_important (ns):
 { monotonic-clock per run = 2674.017426 (confidence: 2685.374519 to 2661.996895);
   r² = Some 0.999347 }
```

After:
```
Backtrace.is_important (ns):
 { monotonic-clock per run = 263.881150 (confidence: 264.748572 to 263.176838);
   r² = Some 0.999726 }
```

Signed-off-by: Edwin Török <edwin.torok@citrix.com>
With a long recursive call there could be thousands of these.
Show just one in the logs.

There is a small <5% performance impact:

Before:
```
Backtrace.is_important (ns):
    { monotonic-clock per run = 263.881150 (confidence: 264.748572 to 263.176838);
    r² = Some 0.999726 }
```

After:
```
Backtrace.is_important (ns):
 { monotonic-clock per run = 275.430030 (confidence: 275.818027 to 275.087604);
   r² = Some 0.999938 }
```

Signed-off-by: Edwin Török <edwin.torok@citrix.com>
forkexecd, resources and xapi-stdext-pervasives wasn't using it.

Signed-off-by: Edwin Török <edwin.torok@citrix.com>
…se_with_backtrace

Signed-off-by: Edwin Török <edwin.torok@citrix.com>
…ndency

The backtrace wasn't retrieved before calling the logging function, so
it was potentially lost.
Use Printexc module instead.

Signed-off-by: Edwin Török <edwin.torok@citrix.com>
…ject#6968)

This is long overdue, but we finally got some QA resource to complete
the test plan, and no major bugs were found.

There is one improvement that could be made: when deleting a user/group
(or otherwise changing permissions) on the XAPI side we could
immediately expire any caches related to that user.
OCaml 4.02 added column location, and 4.11 added function names.
Drop any wrapper modules by dropping anything before the last `__` in
the string, if any.

For backward compatibility with python interop exceptions these new fields are optional.

Backtrace.t_of_sexp is only called in `xapi_xenopsd.ml`, and we require updating XAPI and xenopsd together already.
For everything else (the Task API) we'll accept backtraces without the additional info, but will start producing backtraces
with the additional info.

<10% performance impact:

Before:
```
Backtrace.is_important (ns):
    { monotonic-clock per run = 275.430030 (confidence: 275.818027 to 275.087604);
    r² = Some 0.999938 }
```

After:
```
Backtrace.is_important (ns):
 { monotonic-clock per run = 292.372339 (confidence: 293.039727 to 291.671323);
   r² = Some 0.999771 }
```

Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Ensures that when make_rpc switches protocols between XMLRPC and
JSONRPC, then timeboxed_rpc follows.

Have to be careful to pass the correct values for subtask_of (outer
task), and task_id (inner task).

Signed-off-by: Edwin Török <edwin.torok@citrix.com>
When using `opam` to install all packages from XAPI some libraries
failed to build because the internal `backtrace` library was only
available in the `xapi-log` package.
The build worked when using `dune` directly (or `koji`), because they
build the entire repository instead of individual packages.

To avoid conflicts with the `xapi-backtrace` installed from `xs-opam`
make the backtrace library properly part of `xapi-log`, by exposing it
as `xapi-log.backtrace`.

Put the cram tests into the xapi-log.backtrace package too.

Add rpclib as a dependency for xapi-log, it was a dependency of
xapi-backtrace.

Fixes: e4da294 ("[maintenance]: use internal backtrace library")

Signed-off-by: Edwin Török <edwin.torok@citrix.com>
kc284 and others added 26 commits April 7, 2026 10:27
Signed-off-by: Konstantina Chremmou <konstantina.chremmou@citrix.com>
…mporting the module.

Signed-off-by: Konstantina Chremmou <konstantina.chremmou@citrix.com>
There is a minor change in the datamodel, the rest are SDK files. These
changes will need porting to `26.1-lcm` afterwards.
…ons (xapi-project#6989)

A new live patch update may drop live patch support for a component
running with an old version. This means the old version is EOL in sense
of live patching support.

Previously, the logic collected all live patches that shared the same
base build ID for the running component and treated them as cumulatively
applicable.

With this change, if the base build ID is marked EOL in the latest
relevant update (for the same live patching component), the component is
considered not live‑patchable
A few functions in message_forwarding need to do some cleanup when
encountering an exception, but in doing so, the original backtrace is
lost. This makes it difficult to track down the original location that
raised the exception.

I've considered other options instead of calling Backtrace.is_important:
- Call `Backtrace.reraise e e`: this is more expensive than it needs to
  be.
- Create a new function that can be easily used: This is awkward to
  backport, and in my attempts the resulting code was not great to read.

So I ended up with the compromise of using Backtrace.is_important, which
does exactly what's needed here, even though it's difficult to enforce
correctly across the codebase.

Signed-off-by: Pau Ruiz Safont <pau.safont@vates.tech>
QEMU 10.x has changed some of the arguments' format, de-couple the
xenopsd and qemu builds by making the qemu-wrapper script compatible
with both the old and new versions of qemu.

Signed-off-by: Gerald Elder-Vass <gerald.elder-vass@citrix.com>
Signed-off-by: Gerald Elder-Vass <gerald.elder-vass@citrix.com>
In QEMU upstream commit 7851b21 the piix3-ide-xen was removed as it
duplicated the piix3-ide device (after 0f84458 made them identical).
Therefore the global option(s) for piix3-ide-xen will be ignored unless
they are updated to apply to piix3-ide instead.

Provide both piix3-ide-xen and piix3-ide options to maintain behaviour
across versions during the upgrade period.
Options which are not relevant or used will simply generate a warning in
the daemon.log but are otherwise ignored.

Signed-off-by: Gerald Elder-Vass <gerald.elder-vass@citrix.com>
Newer versions of QEMU accept slightly difference arguments than older
versions.

- chroot and runas were replaced with run-with
- 'on'/'off' boolean values are accepted in most versions
- global variables are machine/device dependent

Adding global variables for both piix3-ide and piix3-ide-xen ensures the
variables are present the relevant device, variables which do not apply
to any devices in use will log a warning (in daemon.log) but are
otherwise ignored.

Example of the warning(s) produced:
```
<time> <host> qemu-dm-<domid>[<pid>]: qemu-dm-<domid>: warning: global piix3-ide-xen.subvendor_id has invalid class name
<time> <host> qemu-dm-<domid>[<pid>]: qemu-dm-<domid>: warning: global piix3-ide-xen.subsystem_id has invalid class name
```

These changes were tested alongside QEMU version 4.2.1 and QEMU version
10.1.0, in both cases guests booted as expected and the respective
warning appeared in the daemon log.
This was updated in xs-opam, but missing in this repo

Signed-off-by: Pau Ruiz Safont <pau.safont@vates.tech>
…project#6995)

A few functions in message_forwarding need to do some cleanup when
encountering an exception, but in doing so, the original backtrace is
lost. This makes it difficult to track down the original location that
raised
 the exception.

I've considered other options instead of calling Backtrace.is_important:
- Call `Backtrace.reraise e e`: this is more expensive than it needs to
  be.
- Create a new function that can be easily used: This is awkward to
  backport, and in my attempts the resulting code was not great to read.

So I ended up with the compromise of using Backtrace.is_important, which
does exactly what's needed here, even though it's difficult to enforce
correctly across the codebase.
This was updated in xs-opam, but missing in this repo
Previously these dependences were added to the opam file, but those are
autogenerated. Add the metadata to the source of truth

Signed-off-by: Pau Ruiz Safont <pau.safont@vates.tech>
Previously these dependences were added to the opam file, but those are
autogenerated. Add the metadata to the source of truth
When we are using a known memory configuration it includes the overhead.
So we only have to add it when we create a new configuration.

Localhost migration in a loop before and after the patch, memory
reported by list_domains -memory -bytes:

  Before:

  id |         uuid |  state |  tot bytes |  max bytes | shadow bytes
  17 | 8fc1ada1-... |    B H | 4296003584 | 4336898048 |     36700160
  18 | 8fc1ada1-... |    B H | 4296003584 | 4376743936 |     36700160
  19 | 8fc1ada1-... |    B H | 4296003584 | 4416589824 |     36700160
  20 | 8fc1ada1-... |    B H | 4296003584 | 4456435712 |     36700160
  21 | 8fc1ada1-... |    B H | 4296003584 | 4496281600 |     36700160
  22 | 8fc1ada1-... |    B H | 4296003584 | 4536127488 |     36700160
  23 | 8fc1ada1-... |    B H | 4296003584 | 4575973376 |     36700160
  24 | 8fc1ada1-... |    B H | 4296003584 | 4615819264 |     36700160
  25 | 8fc1ada1-... |    B H | 4296003584 | 4655665152 |     36700160
  26 | 8fc1ada1-... |    B H | 4296003584 | 4695511040 |     36700160

  After:

  id |         uuid |  state |  tot bytes |  max bytes | shadow bytes
  27 | 8fc1ada1-... |    B H | 4256157696 | 4297052160 |     36700160
  28 | 8fc1ada1-... |    B H | 4256157696 | 4297052160 |     36700160
  29 | 8fc1ada1-... |    B H | 4256157696 | 4297052160 |     36700160
  30 | 8fc1ada1-... |    B H | 4256157696 | 4297052160 |     36700160
  31 | 8fc1ada1-... |    B H | 4256157696 | 4297052160 |     36700160
  32 | 8fc1ada1-... |    B H | 4256157696 | 4297052160 |     36700160
  33 | 8fc1ada1-... |    B H | 4256157696 | 4297052160 |     36700160
  34 | 8fc1ada1-... |    B H | 4256157696 | 4297052160 |     36700160
  35 | 8fc1ada1-... |    B H | 4256157696 | 4297052160 |     36700160
  36 | 8fc1ada1-... |    B H | 4256157696 | 4297052160 |     36700160

Based on work by Edwin Torok.

Signed-off-by: Christian Lindig <christian.lindig@citrix.com>
Signed-off-by: Konstantina Chremmou <konstantina.chremmou@citrix.com>
This fix in the previous patch should obsolete the workaround in the
quickcheck.

Signed-off-by: Christian Lindig <christian.lindig@citrix.com>
When we are using a known memory configuration it includes the overhead.
So we only have to add it when we create a new configuration.

Localhost migration in a loop before and after the patch, memory
reported by list_domains -memory -bytes:

  Before:
```
  id |         uuid |  state |  tot bytes |  max bytes | shadow bytes
  17 | 8fc1ada1-... |    B H | 4296003584 | 4336898048 |     36700160
  18 | 8fc1ada1-... |    B H | 4296003584 | 4376743936 |     36700160
  19 | 8fc1ada1-... |    B H | 4296003584 | 4416589824 |     36700160
  20 | 8fc1ada1-... |    B H | 4296003584 | 4456435712 |     36700160
  21 | 8fc1ada1-... |    B H | 4296003584 | 4496281600 |     36700160
  22 | 8fc1ada1-... |    B H | 4296003584 | 4536127488 |     36700160
  23 | 8fc1ada1-... |    B H | 4296003584 | 4575973376 |     36700160
  24 | 8fc1ada1-... |    B H | 4296003584 | 4615819264 |     36700160
  25 | 8fc1ada1-... |    B H | 4296003584 | 4655665152 |     36700160
  26 | 8fc1ada1-... |    B H | 4296003584 | 4695511040 |     36700160
```
  After:
```
  id |         uuid |  state |  tot bytes |  max bytes | shadow bytes
  27 | 8fc1ada1-... |    B H | 4256157696 | 4297052160 |     36700160
  28 | 8fc1ada1-... |    B H | 4256157696 | 4297052160 |     36700160
  29 | 8fc1ada1-... |    B H | 4256157696 | 4297052160 |     36700160
  30 | 8fc1ada1-... |    B H | 4256157696 | 4297052160 |     36700160
  31 | 8fc1ada1-... |    B H | 4256157696 | 4297052160 |     36700160
  32 | 8fc1ada1-... |    B H | 4256157696 | 4297052160 |     36700160
  33 | 8fc1ada1-... |    B H | 4256157696 | 4297052160 |     36700160
  34 | 8fc1ada1-... |    B H | 4256157696 | 4297052160 |     36700160
  35 | 8fc1ada1-... |    B H | 4256157696 | 4297052160 |     36700160
  36 | 8fc1ada1-... |    B H | 4256157696 | 4297052160 |     36700160
```
Based on work by Edwin Torok.
…api-project#6971)

Instead of xmlrpc.

Targeting feature branch. Will also likely need a new `ocaml-rpc`
release (the current one doesn't handle Base64 and Datetime in JSONRPC
correctly).
Resolves conflicts in helpers: the feature branch added an rpcfun, while
master has moved out some code to a function and applied a functional
argument.
Resolves conflicts in helpers: the feature branch added an rpcfun, while
master has moved out some code to a function and applied a functional
    argument.

Prerequisite of xapi-project#6987
Draft, waiting for testing  and PR reviews to complete.

Depends on:
xapi-project#6974
xapi-project#6971

Also would be good to have an updated ocaml-rpc for the JSONRPC fixes in
xs-opam.

PRs for ocaml-rpc optimizations:
mirage/ocaml-rpc#194
mirage/ocaml-rpc#193
mirage/ocaml-rpc#192
mirage/ocaml-rpc#184

The optimizations are not strictly required, but the first PR about
fixing 4.14 build is, otherwise we can't take the new version into
xs-opam.
…out"

It changes behaviour of the audit_log endpoint

This reverts commit ab802e6.

Signed-off-by: Pau Ruiz Safont <pau.safont@vates.tech>
…out (xapi-project#7014)

It changes behaviour of the audit_log endpoint

This reverts commit ab802e6.
@minglumlu minglumlu force-pushed the private/mingl/feature/trusted-certs branch from 987092d to 4da9ef6 Compare April 15, 2026 02:18
@minglumlu minglumlu merged commit 31944c9 into xapi-project:feature/trusted-certs Apr 15, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.