Consistency issue: % phones that cannot support advertising mode

[micolous]

These numbers are all different:

From https://vmware.github.io/herald/

Herald supports 100% of the phones in the UK that support advertising, as well as the 35% of Android phones that cannot act as ‘advertisers’ and so remain unseen by advertising-only based protocols.

This one looks like an error, as if trying to factor in Android market share into numbers that were already Android-only.

From https://vmware.github.io/herald/protocol/

Supports phones and Operating Systems with known Bluetooth performance issues
Especially those ~14% of UK phones that do not support Advertising, and so cannot be served by other protocols (E.g. GAEN) - via the write characteristic approach

And then later in the same page, there's a different number:

Uses a ‘write characteristic’ allowing phones that cannot advertise (16% approx of UK phones) to tell other phones they are neaby (sic), and still provide regular distance estimation data

All of these numbers are unclear as to whether they refer to:

• percentage of total phone models available in the UK market

  Using this as a metric would inflate the prominence of unpopular phone models over popular ones.

  Some models also have dozens of SKUs (different storage capacity, carrier-specific models, dual SIM, etc.) which could further skew the statistics, depending on how they are counted.

• percentage of total phones sold in the UK market in the last N year(s)

  Using this as a metric would have the effect of inflating recently sold phone models, and over-represent consumers who buy a new phone every 1-2 years.

• percentage of total Android phones in active use in the UK market this year

  Using this as a metric would accurately demonstrate the impact of the issue – ie: N% of UK Android users can't use Advertising-based protocols.

These numbers should be consistent, and be explicit about what they're referring to, so as not to unintentionally mislead or seed distrust in the reader. 😄

[micolous]

Also, it's not clear whether you're including devices that do not support BLE at all, which also could be the cause of the much higher 35% no-advertiser number.

[adamfowleruk]

Ah yes I should be more consistent. Thanks for bringing this up. To clarify, it's ~35% of Android phones in the UK, which equates to ~14% of all UK phones. If you have a read of the paper it clears this up: https://vmware.github.io/herald/efficacy/paper

I'll clear up the website so it's using the same stat everywhere, and points to the section in the paper that has the explanation. Thanks for pointing it out!

[adamfowleruk]

Oh and I forgot to mention - it's based on 'active use' data from the BBC, geofenced to the UK.

[adamfowleruk]

Related to #5

[micolous]

Thanks for clarifying! IMO the best number to use on those tech details is the percent of _just_ Android phones, because then you don't introduce a second variable to debate (Android market share). I found another number, 14.37%, here: https://vmware.github.io/herald/bluetooth/hardware The BBC numbers there are unclear and unsourced - where are they getting this data from (BBC app installs)? Are they counting devices that have multiple BBC apps installed twice? Are they adjusting for biased demographics in who is more likely to have BBC apps installed versus the general population (eg: BBC app users could be an older demographic who are less likely to have new phones)? Do you have a complete list of which devices don't support advertising, but do support BLE? The 35% number seems exceptionally large on its own - does this also include devices that don't support BLE, and wouldn't support Herald's BLE protocol anyway? I recall seeing somewhere in Herald's notes that only ~80% of Android devices support BLE at all.

…

On Mon, 30 Nov 2020, 20:55 Adam Fowler, ***@***.***> wrote: Related to #5 <#5> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#14 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB4IP53PIGOLV4IHKUPDLLSSNT2NANCNFSM4UG5Y6YQ> .

[adamfowleruk]

Hi there! I think I've sorted this all out.

I've created a general Bluetooth stats page, relevant to Herald and other protocols, that talks about the issues, provides a summary, and links to source material: https://vmware.github.io/herald/efficacy/statistics

I've also created a page that summarises Herald specific performance statistics and links to the relevant test results: https://vmware.github.io/herald/efficacy/herald

I've also amended the pages you mention to make them consistent, and link to the above stats pages.

I've also added more detail to the latest report, showing contact events, detections, payload exchanges from real tests. For example: https://vmware.github.io/herald/results/herald-2020-11-28

I think that makes all the info hang together nicer. Please let me know what you think. If there are any other suggestions in future, please open another issue! Thanks for taking the time to give feedback, it's very much appreciated!

[adamfowleruk]

Although the Mermaid pie charts aren't showing up nicely... I'll try and fix that.

[adamfowleruk]

Image URLs fixed. Enjoy!

[micolous]

Thanks, that's significantly better!

One thing to note is that using BBC App install data is biased and not representative of the population, and you should talk to them about compensating for their demographics so that it looks more like "person living in the UK" than "person who watches BBC" 😄

The data source you're using from AltBeacon is poor quality data:

• Several models are listed multiple times; eg: Pixel C is listed four times with two different Android versions, where each both say yes and no for BLE advertisement support. Nexus 9, Nexus 5, SM-G900F, Sony F5121 have similar issues.

• Several other models appear with differences in formatting / spacing / captialisation, or things like Samsung AOSP on Galaxy Nexus, Samsung Galaxy Nexus.

• The list itself is old, where the newest Google phone listed is the Pixel 2.

• AltBeacon itself was still discovering vendors with incorrect isMultipleAdvertisementSupported values, which also skews data their users provided before commits like: AltBeacon/android-beacon-library@f3d4337

Figure W2 is based on number of Android models (this should be called out as "models" rather than devices actually in use), and those duplicated entries will skew the numbers. Because the AltBeacon data is old, it will significantly skew the Android numbers towards old devices.

What's needed there is to clean up the AltBeacon data, and then figure out what to do about the models that aren't in the table, and publish that. It may also be "reasonable" to presume most devices on Android 6 or later that support BLE support advertisements properly.

Figure W3 is suggests 14% of devices do not support advertising -- but then Figure W1 shows that 9% of devices do not support BLE. As far as I can tell, Herald still requires BLE to work, which means it isn't able to target the entire 14%, only the subset that do support BLE (5%, based on your numbers). That should be amended to show devices that don't support BLE as a separate slice of that pie.

But as I mention above, the data from AltBeacon isn't good, mainly as a result of it not being an Android hardware feature code and OEMs breaking things (then again, even when it is a feature code, they can still break things).

The OFCOM data for "phones with BLE support" (72.53%) in the latest report shows an even higher amount of devices that don't support BLE -- but I don't know how to reconcile that part.

Also, please don't use seasons in dates, because they mean different things depending on which hemisphere the reader is in, and marginalises people in the other hemisphere to the author. Specify months or quarters instead. 🙃

[micolous]

Oh also, consider excluding non-phone devices (like tablets and set-top boxes). This again is going to change around the BBC data, which is likely to bias towards iPlayer users (and thus, video playback, on devices with larger screens).

[adamfowleruk]

Yes indeed. I've pored over the OFCOM data too, especially for the efficacy paper. It's hard finding accurate data on this. Herald itself does log whether a phone supports advertising or not. So app developers could, if they chose, track this in their own install base to get an accurate number.

Great feedback.

I believe the 14% is not inclusive of the 9%, but I shall double check and make it clear on the stats page.

[adamfowleruk]

I have confirmed that the 14% is not inclusive of the 9%. So it's "~14% of phone users in the UK have a phone that does not support advertising"

[micolous]

Thanks!

There's still an issue with Figure W2, which is based on number of Android models, that isn't called out as "models" rather than devices actually in use.

I'm also unclear as to whether has excluded non-phone Android devices, such as tablets and set-top boxes, which are unlikely to be used for any sort of Bluetooth contact tracing app, but are secondary devices that may be used for a BBC app. Some of these non-phone devices also appear in the AltBeacon list.

Calling the advertising support data used in this calculation merely "a little old" is an exceptionally charitable characterisation. Given the egregious data problems, the documentation still doesn't make clear how this was reconciled. Even if one cannot get access to the same (BBC) data you used, it should be possible to use another data source to plug those numbers through the same process, to determine what fraction of devices do or don't support advertising.

There is also a statement about new low end devices "possibly" not supporting full Bluetooth functionality, there is no data given to support this claim. As a counter-point, these low-end devices are likely to have other problems with their Bluetooth stack, such as being unable to support multiple connections, to which GATT-based apps like Herald is are not immune: we've seen interference on low-end devices impacting continuous blood-glucose monitors and other Bluetooth devices.

At present, Herald's website makes an exceptionally bold claim about the reach of other contact tracing solutions. While the changes made thus far are a significant improvement, there's still a gap in these numbers, and these options come with significant trade-offs that are not fully elided in the technical documentation.