Speed Test G is a new performance testing system that takes the best parts of traditional speed tests and combines them with the benefits of benchmarks. For full details of how it works please watch this video: Speed Test G - A new way to test real-life smartphone performance.
If you want to see why Speed Test G exists, please watch this video: 5 Reasons Why Smartphone Speed Tests Are Fundamentally Flawed
Speed Test G was launched in November 2018. In September/October 2019, Speed Test G 2.0 and Speed Test GX 2.0 were launched. For more information about V2.0 Please watch these two videos: Speed Test G 2.0 is Coming SOON! and Speed Test G for iOS.
Speed Test G videos appear on the Gary Explains and also on the dedicated Speed Test G channel. The difference? Major and important Speed Test G videos will be published on the GE channel, whereas videos with various combinations and permutations of devices will appear on the Speed Test G channel. Speed Test G videos are published more frequently on the dedicated channel, often 4 or 5 times per week.
You can't. Sorry. The sole reason is that I don't want smartphone makers to have access to this code and then try to "cheat" the system. The idea is that this test remains independent and trustworthy.
Yes. As of Speed Test G 2.0, it is available for Android and iOS. There are two versions of Speed Test G 2.0. The vanilla Speed Test G 2.0 (for Android only) and the cross-platform Speed Test GX 2.0 (for Android and iOS). The suites are almost identical except that Speed Test G 2.0 contains two 3D tests, one written in Unity and one written in Unreal. The GX version only has the Unity test. When comparing cross-platform, GX times on one platform can be compared to GX times on the other. In other words, you can compare GX times on Android with GX times on iOS. See also #26.
3. Can the results of Speed Test G 1.0 be compared with the results from Speed Test G 2.0 or Speed Test GX 2.0?
No. Speed Test G 2.0 is a complete re-write and you can't compare the times.
Absolutely. First, there was Speed Test G 1.0, then Speed Test G 2.0 and Speed Test GX 2.0. Speed Test 3.0 will be developed at some point.
Before recording, we do multiple test runs to ensure that the results we are seeing are correct. This includes running the tests from a known state (i.e. after a reboot, etc). To ensure that CPU throttling (due to heat) isn't an issue the devices are left to cool before each test run and before the recording of the final test. All tests are done in airplane mode. See question #6 for more.
Yes. For example, three different test runs on three different days on the Pixel 3 XL yielded the times of 1:49.26, 1:49.27, and 1:49.33. If a device is showing a difference of more than 1 (maximum 2) seconds between tests then showing isn't right then the device/environment/conditions need to be examined.
7. Is it fair to compare a device running Android X.x with a device running a different version of Android, e.g. Android Y.y?
Yes, for two reasons. First, this is a device test. This means that when you buy the device, this is what you get. If the device underperforms because of a software issue then consumers need to know that. Second, there are rarely major advances in Android performance from one version to the next. It does happen, for example when Android moved from Davlik to ART, but that doesn't happen every day! If and when that becomes a factor then it will be duly noted during the test run.
That may well be true, but here we are testing the devices as supplied by the manufacturers.
9. Device XYZ has a Turbo mode/Performance mode/Burst mode/Game mode. Did you enable that before the test?
The devices are used in their defaults configurations. If the relevant performance mode is enabled by default that is how the device is tested. Occasionally, if warranted, we may perform two tests, one with the mode enabled and one with it disabled.
10. Why haven't you tested Device SPQR? I have asked several times on social media or in the comments.
Could be for one of two reasons. 1) The device isn't that interesting. Sorry. 2) But more likely, I don't have access to that device at the moment. I don't have a huge box with every released device which I just open and take the device for the next video. Devices are expensive. Sometimes I can borrow a device. Sometimes my teammates have access to the device and I can use that. Sometimes I need to buy the device and hope to sell it afterwards.
At the moment the test shows the CPU, CPU/GPU (mixed), and GPU times. Those numbers should be sufficient for most people.
Yes. I am planning on making a separate (but similar) test suite for testing throttling and sustained performance. Sustained performance testing will feature in separate videos and not as part of the normal Speed Test G run.
Yes. I am planning on making a separate (but similar) test suite for testing battery life. Battery life testing will feature in separate videos and not as part of the normal Speed Test G run.
That is true, it doesn't. At the moment I have no plans to change that, for philosophical and technical reasons. It certainly isn't planned for the Speed Test G 3.0.
15. Shouldn't the tapping of the Start button be synchronized somehow (via Bluetooth, at a fixed time, by a laser bouncing off the moon)?
No. Each device records the total test time internally from within the Speed Test G app. The time reported at the end comes from the app itself, not from an external source, not from the video editing, not from a stopwatch, not from an hourglass. The devices can be started weeks apart but the time result is the same. The only benefit of synchronization is for the video and the "race" aspect of the video. For that, human synchronization is good enough.
16. Does the screen resolution affect the performance? If device A has a lower resolution screen, won't it be quicker? Is that fair?
Absolutely. And since this is a real-world test and not a theoretical test then that is what we are trying to test. If you buy device A, is the GPU powerful enough to handle the display at the native resolution? Remember these tests are device tests, not processor (SoC) tests. For example, the Pixel 3 completes the test 5 seconds quicker than the Pixel 3 XL. Why? Because both have the same processor but one has a smaller screen resolution.
From time to time, when demand necessitates, I will also make videos with devices set to their non-default screen resolutions.
That is certainly something that can be considered for V2.0. The simplest way would be to use the Android Neural Networks API. However, if the test was to become cross-platform (Android and iOS) then this would present a problem.
Yes! SpeedTestG.com
I currently don't show a leaderboard at the ends of the video because most people don't watch these videos in chronological order, which means that the leaderboard will be a) out of date, b) irrelevant unless you watched all the videos, c) confusing viewers who arrive via a search, d) misleading in that it might make device A look better than it is because at the time of that particular video devices X or Y or Z hadn't yet been tested. Also see #19.
During the development of Speed Test G I found some compatibility problems with the way the custom launcher interacts with the Android framework. Some phones would work one way, some a different way and some work either way! So there are technically two variants of Speed Test G, which are used depending on the actual phone. The changes are very minor, literally some flags which are used differently when the launcher installs itself. The changes don't alter the speed of the test run at all and don't impact the results. I have tested this empirically via devices which work with the L and non-L version, there is no difference in the final results.
The SQLite test adds 500 records to an SQLite database and then removes them. SQLite is the most widely deployed database in the world. It is the default way to store persistent data on Android and is used (probably unknown to most users) by hundreds of thousands of apps. Talking of SQLite on Android, the Google developer docs state that, "Apps that handle non-trivial amounts of structured data can benefit greatly from persisting that data locally. The most common use case is to cache relevant pieces of data. That way, when the device cannot access the network, the user can still browse that content while they are offline." In other words, if a device is able to process SQLite transactions quickly then it improves the overall user experience.
Update: For Speed Test G 2.0 and Speed Test GX 2.0, the number of records was increased to 750. Additionally, the test is run twice using different optimization settings for SQLite. Specifically, the first test run forces the rollback journal to be deleted at the conclusion of each transaction. For the second test, the journaling mode uses a write-ahead log instead of a rollback journal to implement transactions. See SQLite PRAGMA Statements for details.
Because people don't play games off-screen. This is a real-world device test.
24. Why do your results show different relative performance numbers compared to the popular benchmark apps?
It comes down to three things:
-
What are you measuring? If you are measure raw CPU performance, with little access to the memory, lots of L1 and L2 cache hits, little or no access to the IO etc, then you will get a different result than if you are measure overall system performance. Overall system performance will include more memory access, more cache misses, more IO access plus other factors like app start-up time. Same goes with the GPU. Many of these benchmarks use "off-screen" testing to see how the GPU performs when it doesn't need to display the result and regardless of the screen resolution, that is fine, but at a system-level (real-world) level nobody plays games "off-screen".
-
Can smartphone makers optimize their systems for benchmarks? Absolutely. Some have been caught cheating, by artificially increasing the performance of the CPU/GPU during a benchmark, but even if a manufacturer doesn't do that, you can be sure that it has a lab of people making sure that the device works very, very well for the popular benchmarks, so that they can make % claims during keynotes. See also question #1.
-
Improvements to CPU and GPU processing power never translate to the same increase at a system level as "the system" includes more than just the processor, there is the RAM, the internal storage, the motherboard (with interrupts and buses and IO channels etc), the display driver, the video encoder/decoder, other components of the SoC like the neural engine etc. Bottom line: Speed Test G tests the whole device, not just the CPU or GPU. That is what users will see in the real world.
A smartphone is more than just its processor. You have to consider the RAM, the internal storage, the motherboard, etc.
26. Is it fair to compare iOS with Android? An app written in Java can't be comparted one written in Swift or vice versa.
If Speed Test GX 2.0 was written using two different languages and two different codebases, then yes, it would be unfair. However, Speed Test GX is written using a cross-platform toolkit that produces native (that means Arm machine code, not Java or some other type of intermediate language) apps for both Android and iOS. There is one codebase. The cross-platform toolkit is equally optimized for iOS and for Android, one isn't more or less optimized than the other.
27. I have played XYZ on Device A and it is better than on Device B, but your test shows otherwise. Why is that?
While I don't suggest that your experience is incorrect, it is subjective and in no way quantifiable. You have no way to determine even the graphic settings on Device A compared to B, or if the "high" graphics settings are the same on one platform/device compared to the other. You can't guarantee that the textures and models used on one device (or platform) is the same as on the other. Speed Test G is controlled, measured and accurate. For the sake of gameplay speed, app creators will change lots of stuff on one platform compared to another to improve gameplay while reducing other factors and features. Unless you do a pixel by pixel comparison of the same moment shown on two devices you can't make any claims about the power of the processor, only that the app works differently on each device.
28. Speed Test G shows the GPU in Device XYZ is slow, but I can play 3D at the highest settings on that phone, your test must be broken!
Comparing the GPU tests of Speed Test G with games (and gameplay) isn't a good idea. The whole point of games is to offer smooth gameplay, often at the cost of other things like details, textures, lighting, etc. The GPU tests for Speed Test G are designed deliberately to be hard for the GPU, to see what the GPU can handle.
Also, note that the "high settings" on one device will likely be different from the "high settings" on another device. These settings are often relative to the detected hardware. The game developer isn't going to offer settings that will cripple the gameplay.
29. In Speed Test G 2.0, Device A had a higher FPS in the smoke particle test than Device B, but yet it lost in GPU tests, how?
The smoke particle test is a 2D test. The Unity and Unreal tests are 3D tests. There is a significant difference in how the GPU handles 2D work and 3D work. That is why there are both types of test.
LOL
Speed Test G isn't "optimized" for any particular device or processor. The test is written using standard tools and libraries (like Unity) and the same test runs on every device. Any lacks or possible underperformance are due to the device, not the test.
I am adding this section because of the Mate 30 and Mate 30 Pro. These devices use the Kirin 990, but due to the USA's ban on Huawei, these devices don't have Google's services like the Play Store, etc.
I tried installing Speed Test G on the Mate 30 and Mate 30 Pro, but it didn't work. Since this is a fairly unique device, with a unique configuration, I didn't spend any more time trying to find out why.
If I find more devices that don't work with the test then I will update this section.