Benchmarks Are Snapshots

Benchmarks Are Snapshots

In short

A YABS result describes one machine, on one shared host, during roughly ten minutes of one particular day. It is a photograph, not a biography. Everything in the report — disk, network and CPU alike — can and does change with the time of day, with who else is on the host, with how long the server has been running, and with the state of the far end of a network test. The date printed at the top of every report is there for a reason.

This page explains where the variance comes from, how big it can be, and how to benchmark in a way that produces conclusions you can defend.

Why one run proves little

You are sharing

Almost everyone running YABS is on a virtual server. The physical CPU cores, memory bandwidth, storage device and network uplink under your VM are shared with other tenants whose behaviour you cannot see. A neighbour starting a backup at 02:00, a crypto-miner on the next vCPU, or a popular website having a good day all change your numbers without anything changing about your server.

The hypervisor also makes policy decisions: CPU scheduling fairness, I/O throttling, burst credits, memory ballooning, and (on containers) cgroup quotas. Those policies can differ between busy and quiet hours.

The far end is shared too

The network test depends on seven public iperf3 servers donated by hosting companies. They are used by thousands of YABS runs a day plus everyone else who knows about them. A busy result or a low figure for one location says, first of all, something about that server at that moment. Even the route between you and it changes: transit links congest in the evening, peering changes, and a 10-second TCP test is very sensitive to packet loss.

The machine itself is not in a steady state

• Fresh VMs are often still applying updates or running cloud-init; unattended-upgrades in the middle of Geekbench costs real points.
• CPU clocks move: an idle core sits at 800 MHz, boosts to its maximum for a single-threaded test, and settles lower when every core on the host is busy or the host hits a power/thermal limit.
• Storage has caches that fill and flush, SSDs that garbage-collect, and QoS schemes with burst allowances that refill over minutes or hours. A 30-second test measures the burst; a 10-minute test measures the floor.
• First-run effects: the first fio pass lays out a fresh 2 GiB file; the first Geekbench download warms nothing useful but does consume time.

The tools are a compromise

YABS uses short tests (30 s per block size, ~10 s per iperf direction, a single Geekbench pass) so that a full run finishes in minutes and so that the shared iperf servers survive. Short tests are noisier than long ones by design. As Anthony Smith of InceptionHosting put it when the fio test was being designed in January 2020: he was "not sure how a 1GB or 4GB or 60 second test captures anything you are ever likely to use a VPS for in the real world." The community's answer was not that it does, but that a consistent, repeatable, quick test is still far more useful than none — provided everyone remembers what it is.

How much variance is normal?

Experience from years of posted results (not a controlled study):

Component	Typical run-to-run spread on a healthy, moderately loaded shared host	What a large spread suggests
Geekbench single-core	±5–10 %	Steal/oversubscription, thermal or power capping, CPU quota
Geekbench multi-core	±10–20 %	Same, amplified; also GB6/7 scaling choices
fio 4k IOPS	±10–30 %	Noisy neighbours on storage, cache state, QoS/burst credits
fio 1m MB/s	±10–20 %	Host-level storage contention
iperf3 to nearest server	±5–15 %	Uplink contention on the node or at the provider's edge
iperf3 to far servers	±30–100 %	Internet weather; mostly not about your server

If your results move more than this between runs at different hours, that variance is the finding: the server's performance depends on when you use it.

Benchmarking properly with YABS

1. Run more than once, at different times

The minimum that supports a conclusion is three runs spread over different hours of at least two days (for example 03:00, 14:00 and 21:00 local to the datacentre). Keep the median and note the spread. A LowEndTalk regular described the community's thorough version in 2023: "72 hours long random interval yabs testing. Just cron and hammer vps to the ground. This would shed some light on node overselling, limitations and even technical support."

A cron-friendly recipe that keeps each run's JSON:

# every 6 hours, keep JSON per run; -r limits iperf to 3 locations to be kind to bandwidth
0 */6 * * * cd /root/bench && curl -sL yabs.sh | bash -s -- -r -w "yabs-$(date +\%Y\%m\%d-\%H\%M).json" > "yabs-$(date +\%Y\%m\%d-\%H\%M).txt" 2>&1

Mind the bandwidth: even with -r, each run moves several gigabytes on a 1 Gbit/s port. Use -i to skip the network test entirely if your plan is metered and you only want CPU/disk trends.

2. Record steal time alongside each run

In a second SSH session:

vmstat 1

Watch the st column while Geekbench runs. 0–1 is normal; sustained values above 5 mean the host was oversubscribed at that moment and the CPU numbers are depressed through no fault of the VM. Write it down next to the result. (top shows the same figure as %st.)

3. Make sure the box is idle

Before a run: ps -e -o comm= | grep -E 'apt|dpkg|yum|dnf|unattended'. Let first-boot updates finish. Stop your own services if you want a hardware baseline, or leave them running if you want to know what headroom remains under your real load — but be consistent.

4. Change one thing at a time

Compare results from the same YABS version era and the same Geekbench version. Compare providers at the same hour on the same day; otherwise you have measured the time of day. Do not compare a -r run's network table with a full run's.

5. Look at distributions, not single numbers

The results databases that accept -s uploads (YABSdb, VPSBenchmarks, ServerVerify, serverlist.dev, VPS Host Review) and the Geekbench Browser show you many results for the same CPU model or plan. Where your run falls in that distribution matters more than the absolute value.

6. Test the right thing

• cd to the disk you actually use before running; YABS tests the partition under the current directory.
• If a plan says "NVMe" for the OS disk and "HDD" for the data volume, test both (yabs.sh -ig in each).
• If you care about sustained storage performance, follow up with a long fio run by hand (see Tests: Disk (fio) for the command YABS uses as a starting point; add --time_based --runtime=600).

7. Finish with your own workload

Synthetic tools rank machines; only your workload says whether a machine is enough. Time a real job — a build, a database import, a page render under load — on each candidate. A build that takes 4 minutes on one host and 7 on another has settled the question whatever Geekbench said.

Reading other people's snapshots

When someone posts a YABS result in a forum or a database:

• Check the date and the YABS version in the banner. Results from 2020 ran Geekbench 5 against different iperf servers and under-reported disk speed by ~2.4 % (KiB/KB); a 2026 result runs Geekbench 6 by default.
• Assume the moment was chosen. People post flattering runs. Provider-posted results doubly so.
• Look for uptime. A result taken at 0 days, 0 hours, 9 minutes is a brand-new VM on a possibly empty host.
• Prefer threads with multiple runs over time (LowEndSpirit's "Post some YABS bench here" often has follow-ups weeks later).

What YABS does not measure at all

To keep expectations straight, a YABS run says nothing about:

• memory bandwidth or latency (Geekbench's memory-heavy sub-tests touch it indirectly; GB4 weighted it more);
• storage latency percentiles or sustained write endurance;
• single-stream TCP throughput, packet loss, jitter, or routing quality (one ping -c1 is not a latency test);
• GPU, web-server, database or application-level performance;
• reliability, uptime, support quality, or how the provider behaves when a neighbour misbehaves;
• anything about tomorrow.

For those, use longer-running or purpose-built tools — several are listed in Comparison with Other Bench Scripts.

See also

• Interpreting Results — what the numbers mean once you have enough of them
• Design Principles — why YABS reports rather than judges
• Understanding the Output
• Tests: Network (iperf3) — why busy is about the far end