Skip to content

Add Azure Linux support to MongoDB/YCSB workload (declarative SupportedLinuxDistributions filtering) - #784

Merged
Bryan DeYoung (brdeyo) merged 5 commits into
microsoft:mainfrom
moprashant:users/moprashant/mongodb-azurelinux-distro-filter
Aug 13, 2026
Merged

Add Azure Linux support to MongoDB/YCSB workload (declarative SupportedLinuxDistributions filtering)#784
Bryan DeYoung (brdeyo) merged 5 commits into
microsoft:mainfrom
moprashant:users/moprashant/mongodb-azurelinux-distro-filter

Conversation

@moprashant

Copy link
Copy Markdown
Contributor

Summary

Adds Azure Linux 3 (Mariner) support to the MongoDB/YCSB workload, which previously ran on Ubuntu only. Also fixes two product bugs in MongoDBServerExecutor that prevented the server from starting on any RPM-based distribution, and adds a reusable SupportedLinuxDistributions parameter to VirtualClientComponent.

Note for reviewers: this is one of two alternative implementations. The companion PR (mongodb-azurelinux-runtime-detection) achieves the same outcome by detecting the package manager at runtime inside the shell steps. Only one should be merged — they conflict in PERF-MONGODB-YCSB.json. Both share the identical C# fixes. This PR is the more declarative option: each distro's install steps are visible in the profile, at the cost of a longer dependency list and a new framework-level parameter.

Product fixes (MongoDBServerExecutor.cs)

These are genuine bugs, not Azure-Linux-specific workarounds:

  1. Hardcoded mongodb service account. The data directory was chowned to a hardcoded mongodb user. RPM packages create the account as mongod, so mongod could not write to its own dbPath and failed to start. Now resolved at runtime per package family (mongodb on Debian/Ubuntu, mongod on RPM).

  2. MongoDB port never opened on the local firewall. Ubuntu's default iptables INPUT policy is ACCEPT, so this was silently unnecessary there. Azure Linux defaults to DROP, so the client could never reach the server. InitializeAsync now opens the port via the existing IFirewallManager.

Framework change (VirtualClientComponent.cs)

Adds a SupportedLinuxDistributions parameter allowing any component to declare which Linux distributions it applies to. Components whose distribution doesn't match are skipped. Matching is case-insensitive, whitespace-tolerant, comma-delimited, and honours upstream/downstream distro relationships.

This is generally reusable beyond MongoDB, but note it expands the framework surface area — reviewers may prefer the companion PR if that isn't desirable.

Profile changes (PERF-MONGODB-YCSB.json)

  • Dependency steps carry explicit SupportedLinuxDistributions values (Debian,Ubuntu vs AzureLinux), so the install path for each distro is declared rather than inferred.
  • Adds lshw and gnupg2, which Ubuntu images preinstall but Azure Linux does not.
  • Adds -p mongodb.upsert=true to the Read_Latest and Short_Range_Scan scenarios (see below).

Fixes pre-existing duplicate-key insert failures

Read_Latest (workloadd) and Short_Range_Scan (workloade) both perform 5% run-phase inserts. YCSB seeds both insert sequences at recordcount, so whichever runs second collides on every insert:

E11000 duplicate key error collection: ycsb.usertable index: _id_

The error count matched INSERT-FAILED-Operations exactly (3445/3445). This is pre-existing on main and reproduces identically on Ubuntu — it is not introduced or caused by Azure Linux. mongodb.upsert=true is the documented remedy for a partially loaded data set.

Behavioural note: colliding inserts now become updates, so INSERT latency is measured where it previously reported 0 successful ops. RecordCount, metric names, the operation mix, and scenario names are unchanged.

Testing

Unit tests (new):

  • MongoDBServerExecutorTests: ..._ResolvesTheMongoDBServiceUserForThePlatform, ..._OpensTheMongoDBPortOnTheLocalFirewall
  • VirtualClientComponentTests: 12 cases covering empty/undefined parameters, delimited parsing, case and whitespace tolerance, downstream-distro matching, non-Linux platforms, and interaction with SupportedPlatforms

92/92 MongoDB tests and 12/12 SupportedLinuxDistributions tests pass against current main.

End-to-end, two-VM client/server (Standard_D8s_v5):

Distro Result
Ubuntu 24.04 Exit Code 0 — 7/7 scenarios, 0 failed operations
Azure Linux 3 Exit Code 0 — 7/7 scenarios (validated prior to the upsert change; see below)

Distro filtering was verified directly on Ubuntu: all three AzureLinux-only steps executed 0 times and dnf was invoked 0 times.

Ubuntu regression risk was explicitly exercised, not just reasoned about: the firewall rule was flushed before the Ubuntu run so the new code path genuinely executed, and it completed without throwing. Measured data-directory ownership confirmed mongodb on Ubuntu and mongod on Azure Linux.

Known gap

The Azure Linux end-to-end run for this branch predates the mongodb.upsert change. The upsert change is byte-identical to the companion PR, which was validated end-to-end on Azure Linux with it applied — but for full transparency, this exact combination was not re-run here.

moprashant and others added 2 commits August 11, 2026 15:53
Extends PERF-MONGODB-YCSB.json to run on Azure Linux in addition to Ubuntu,
and introduces a general-purpose mechanism for scoping any profile component
to specific Linux distributions.

SupportedLinuxDistributions component parameter
  Components may now declare "SupportedLinuxDistributions": "AzureLinux,Ubuntu".
  IsSupported() evaluates it after the existing SupportedPlatforms check, and a
  component that declares it never runs on non-Linux systems.

Profile
  The MongoDB installation steps are split into an apt path scoped to
  Debian/Ubuntu and a dnf path scoped to AzureLinux. lshw is added to the
  package prerequisites; VirtualClient uses it for disk discovery and it is not
  present by default on Azure Linux. gnupg is corrected to gnupg2 for the
  dnf/yum package names.

MongoDBServerExecutor
  The mongod service account is resolved at runtime rather than hardcoded to
  mongodb. RPM packages create mongod while Debian packages create mongodb, so
  the data directory was left owned by root on Azure Linux and mongod exited
  with status 100.

  The MongoDB port is opened via IFirewallManager during initialization. Azure
  Linux applies a default-deny policy to inbound traffic, so the client could
  not reach the server and runs completed with zero recorded operations.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Read_Latest (workloadd) and Short_Range_Scan (workloade) both perform 5%
run-phase inserts. YCSB seeds the insert key sequence for both scenarios at
recordcount, so the scenario that runs second collides with keys already
written by the first and every insert fails with:

  E11000 duplicate key error collection: ycsb.usertable index: _id_

Measured on a 2500000 record run, Short_Range_Scan reported
INSERT-Operations=0 with INSERT-FAILED-Operations=3445, matching the 3445
duplicate key errors in the logs exactly.

Setting mongodb.upsert=true makes colliding inserts update the existing
document instead of failing. This is the documented behaviour of the YCSB
MongoDB binding for partially loaded data sets and requires no change to
RecordCount, operation mix, metric names or scenario names.

This issue is not platform specific and reproduces on both Ubuntu and
Azure Linux.

Verified on Azure Linux 3 and Ubuntu 24.04 (2 VM client/server, exit code 0,
all 7 scenarios): Short_Range_Scan now reports non-zero INSERT-Operations
with zero failed operations and no E11000 errors.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@moprashant

Copy link
Copy Markdown
Contributor Author

Alternative implementation of the same feature: #783 (runtime package-manager detection, keeps the profile scenario list identical to \main). These two PRs conflict in \PERF-MONGODB-YCSB.json\ — only one should be merged. Happy to close whichever the team doesn't prefer.

moprashant and others added 2 commits August 12, 2026 16:53
The profile metadata declares SupportedOperatingSystems as 'AzureLinux,Ubuntu',
but the workload documentation still listed Ubuntu only. This aligns the
documentation with the profile.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous commit was uploaded from a CRLF working copy, which rewrote
every line. This restores LF endings so the change is the intended single line.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@moprashant

Copy link
Copy Markdown
Contributor Author

Added one commit here: the AzureLinux line in website/docs/workloads/mongodb/mongodb-profiles.md.

That was the only change in #783 not already covered by this PR. The profile metadata already declared "SupportedOperatingSystems": "AzureLinux,Ubuntu", but the docs page still listed Ubuntu only, so the two had drifted. #783 is now closed in favour of this PR.

The diff is a single added line:

 * **Supported Operating Systems**
+  * AzureLinux
   * Ubuntu

Bryan DeYoung (@brdeyo) apologies -- pushing this dismissed your approval. Could I get a re-approve when you have a moment? All 5 checks are green, and no code changed since you reviewed.

@moprashant

Copy link
Copy Markdown
Contributor Author

This PR is now fully ready to merge — all blockers are cleared:

  • reviewDecision: APPROVED (Bryan DeYoung (@brdeyo) — thank you for re-approving after the docs commit)
  • mergeStateStatus: CLEAN
  • mergeable: MERGEABLE
  • All 5 checks green: CodeQL, Document Website Build Check, PR build + test (Linux), Pull request build + test, license/cla

I refreshed the branch against main (it had fallen behind) and CI re-ran clean on the merged result.

Could a maintainer with write access please squash-merge this? I only have pull permission, so both mergePullRequest and enablePullRequestAutoMerge are denied for my account.

@brdeyo
Bryan DeYoung (brdeyo) merged commit f87ae6b into microsoft:main Aug 13, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants