Add Azure Linux support to MongoDB/YCSB workload (declarative SupportedLinuxDistributions filtering) - #784
Conversation
Extends PERF-MONGODB-YCSB.json to run on Azure Linux in addition to Ubuntu, and introduces a general-purpose mechanism for scoping any profile component to specific Linux distributions. SupportedLinuxDistributions component parameter Components may now declare "SupportedLinuxDistributions": "AzureLinux,Ubuntu". IsSupported() evaluates it after the existing SupportedPlatforms check, and a component that declares it never runs on non-Linux systems. Profile The MongoDB installation steps are split into an apt path scoped to Debian/Ubuntu and a dnf path scoped to AzureLinux. lshw is added to the package prerequisites; VirtualClient uses it for disk discovery and it is not present by default on Azure Linux. gnupg is corrected to gnupg2 for the dnf/yum package names. MongoDBServerExecutor The mongod service account is resolved at runtime rather than hardcoded to mongodb. RPM packages create mongod while Debian packages create mongodb, so the data directory was left owned by root on Azure Linux and mongod exited with status 100. The MongoDB port is opened via IFirewallManager during initialization. Azure Linux applies a default-deny policy to inbound traffic, so the client could not reach the server and runs completed with zero recorded operations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Read_Latest (workloadd) and Short_Range_Scan (workloade) both perform 5% run-phase inserts. YCSB seeds the insert key sequence for both scenarios at recordcount, so the scenario that runs second collides with keys already written by the first and every insert fails with: E11000 duplicate key error collection: ycsb.usertable index: _id_ Measured on a 2500000 record run, Short_Range_Scan reported INSERT-Operations=0 with INSERT-FAILED-Operations=3445, matching the 3445 duplicate key errors in the logs exactly. Setting mongodb.upsert=true makes colliding inserts update the existing document instead of failing. This is the documented behaviour of the YCSB MongoDB binding for partially loaded data sets and requires no change to RecordCount, operation mix, metric names or scenario names. This issue is not platform specific and reproduces on both Ubuntu and Azure Linux. Verified on Azure Linux 3 and Ubuntu 24.04 (2 VM client/server, exit code 0, all 7 scenarios): Short_Range_Scan now reports non-zero INSERT-Operations with zero failed operations and no E11000 errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
Alternative implementation of the same feature: #783 (runtime package-manager detection, keeps the profile scenario list identical to \main). These two PRs conflict in \PERF-MONGODB-YCSB.json\ — only one should be merged. Happy to close whichever the team doesn't prefer. |
The profile metadata declares SupportedOperatingSystems as 'AzureLinux,Ubuntu', but the workload documentation still listed Ubuntu only. This aligns the documentation with the profile. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous commit was uploaded from a CRLF working copy, which rewrote every line. This restores LF endings so the change is the intended single line. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Added one commit here: the That was the only change in #783 not already covered by this PR. The profile metadata already declared The diff is a single added line: * **Supported Operating Systems**
+ * AzureLinux
* UbuntuBryan DeYoung (@brdeyo) apologies -- pushing this dismissed your approval. Could I get a re-approve when you have a moment? All 5 checks are green, and no code changed since you reviewed. |
|
This PR is now fully ready to merge — all blockers are cleared:
I refreshed the branch against Could a maintainer with write access please squash-merge this? I only have |
Summary
Adds Azure Linux 3 (Mariner) support to the MongoDB/YCSB workload, which previously ran on Ubuntu only. Also fixes two product bugs in
MongoDBServerExecutorthat prevented the server from starting on any RPM-based distribution, and adds a reusableSupportedLinuxDistributionsparameter toVirtualClientComponent.Product fixes (
MongoDBServerExecutor.cs)These are genuine bugs, not Azure-Linux-specific workarounds:
Hardcoded
mongodbservice account. The data directory waschowned to a hardcodedmongodbuser. RPM packages create the account asmongod, somongodcould not write to its owndbPathand failed to start. Now resolved at runtime per package family (mongodbon Debian/Ubuntu,mongodon RPM).MongoDB port never opened on the local firewall. Ubuntu's default
iptablesINPUTpolicy isACCEPT, so this was silently unnecessary there. Azure Linux defaults toDROP, so the client could never reach the server.InitializeAsyncnow opens the port via the existingIFirewallManager.Framework change (
VirtualClientComponent.cs)Adds a
SupportedLinuxDistributionsparameter allowing any component to declare which Linux distributions it applies to. Components whose distribution doesn't match are skipped. Matching is case-insensitive, whitespace-tolerant, comma-delimited, and honours upstream/downstream distro relationships.This is generally reusable beyond MongoDB, but note it expands the framework surface area — reviewers may prefer the companion PR if that isn't desirable.
Profile changes (
PERF-MONGODB-YCSB.json)SupportedLinuxDistributionsvalues (Debian,UbuntuvsAzureLinux), so the install path for each distro is declared rather than inferred.lshwandgnupg2, which Ubuntu images preinstall but Azure Linux does not.-p mongodb.upsert=trueto theRead_LatestandShort_Range_Scanscenarios (see below).Fixes pre-existing duplicate-key insert failures
Read_Latest(workloadd) andShort_Range_Scan(workloade) both perform 5% run-phase inserts. YCSB seeds both insert sequences atrecordcount, so whichever runs second collides on every insert:The error count matched
INSERT-FAILED-Operationsexactly (3445/3445). This is pre-existing onmainand reproduces identically on Ubuntu — it is not introduced or caused by Azure Linux.mongodb.upsert=trueis the documented remedy for a partially loaded data set.Behavioural note: colliding inserts now become updates, so INSERT latency is measured where it previously reported 0 successful ops.
RecordCount, metric names, the operation mix, and scenario names are unchanged.Testing
Unit tests (new):
MongoDBServerExecutorTests:..._ResolvesTheMongoDBServiceUserForThePlatform,..._OpensTheMongoDBPortOnTheLocalFirewallVirtualClientComponentTests: 12 cases covering empty/undefined parameters, delimited parsing, case and whitespace tolerance, downstream-distro matching, non-Linux platforms, and interaction withSupportedPlatforms92/92 MongoDB tests and 12/12
SupportedLinuxDistributionstests pass against currentmain.End-to-end, two-VM client/server (
Standard_D8s_v5):Distro filtering was verified directly on Ubuntu: all three
AzureLinux-only steps executed 0 times anddnfwas invoked 0 times.Ubuntu regression risk was explicitly exercised, not just reasoned about: the firewall rule was flushed before the Ubuntu run so the new code path genuinely executed, and it completed without throwing. Measured data-directory ownership confirmed
mongodbon Ubuntu andmongodon Azure Linux.Known gap
The Azure Linux end-to-end run for this branch predates the
mongodb.upsertchange. The upsert change is byte-identical to the companion PR, which was validated end-to-end on Azure Linux with it applied — but for full transparency, this exact combination was not re-run here.