Skip to content
This repository has been archived by the owner. It is now read-only.

Discovery fails when both devices are locked #2

Closed
lukeredpath opened this issue May 7, 2020 · 133 comments
Closed

Discovery fails when both devices are locked #2

lukeredpath opened this issue May 7, 2020 · 133 comments

Comments

@lukeredpath
Copy link

lukeredpath commented May 7, 2020

Describe the bug
I've been trying to see how the app works around the issue where an iOS app central cannot discover an iOS app peripheral on another device if they are both in the background (and/or the device is locked).

I did a lot of initial testing of this with an Apple sample project yesterday and could not find a way to make it work and am able to reproduce this problem with the COVID-19 app as well.

A few notes, I've had to make a few small mods to get the app running on my device to the point where it's actively scanning and broadcasting:

  • Disabled remoteNotificationManager.configure() in the app delegate to bypass Firebase
  • Faked registration by creating my own Registration, using the device advertising identifier as the UUID and a fake secret/SecKey.
  • Commented out the peripheral state restoration code in the broadcaster so my two test devices always appear as new devices to each other.

To Reproduce

  1. Install app on two iOS devices. In my test I'm using an iPad and an iPhone.
  2. Install the app on the iPhone, launch the app and make sure its working. Kill the app, turn on Airplane mode.
  3. Install the app on the iPad, launch the app, make sure its working (I'm running this one attached to the debugger so I can watch the log output). Background the app (verify using logs) and lock the device.
  4. Move out of range of the iPad with the iPhone, wake the iPhone up, disable Airplane mode, launch the app, close the app, lock the device.
  5. Move in to range of the iPad and observe no log output - the didDiscover delegate method is not being called on the iPad.
  6. Wake the iPad screen by pressing the home button - the didDiscover delegate is called and it springs to life.

Expected behaviour
This was the behaviour I expected based on my previous testing however I was lead to believe that a way had been found around this limitation.

My understanding of the problem is thus:

  • iOS apps acting as a peripheral do not broadcast their service UUID in the advertising packet while they are broadcasting in the background. The service UUID is stored in an "overflow area" to quote the Apple docs.
  • iOS apps acting as a central can only find other iOS peripherals by scanning for a specific service UUID if when they are running in the background - because the other device is not broadcasting it's service UUID, it doesn't find it.
  • If either app is in the foreground, the connection is made - either the central is in the foreground and is able to find the peripheral without relying on the service UUID being advertised or the peripheral is in the foreground and is broadcasting it's service UUID so the backgrounded central can find it.

Smartphone (please complete the following information):
I'm using an iPhone XS and an iPad mini 4 running the latest version of iOS 13.

EDIT - some further detailed scenarios below.

@lukeredpath
Copy link
Author

lukeredpath commented May 7, 2020

I appreciate my testing of this is limited at the moment and would be grateful if anyone else is able to reproduce. I've pushed the branch I was running with my small modifications to a fork which might make it easier for people to get up and running:

https://github.com/lukeredpath/COVID-19-app-Documentation-BETA/tree/mods-to-run-on-local-device

You might still need to copy the example GoogleService-Info.plist file to .secret/, update the team ID and configure provisioning.

@benabrahamson
Copy link

benabrahamson commented May 7, 2020

Hi - not sure how helpful this might be to your particular question, but this article goes into detail about the technical details and workarounds used in the app: https://reincubate.com/blog/staying-alive-covid-19-background-tracing/

@lukeredpath
Copy link
Author

lukeredpath commented May 7, 2020

Sadly I don't think the above article really addresses the core issue. I've heard some suggestions that it might be relying on the Android app to wake iOS apps up but I'm not convinced that would work - even if an Android app connects to the iOS app (which is running in the background), it won't bring the app to the foreground, iOS will wake the app up in the background briefly (for about 10 seconds) and the device will of course remain locked.

Not ruling it out - I don't have an Android device to test this theory, but if this is the case then it seems like a pretty shonky workaround.

Before we speculate how they are working around this it would be good to establish if what I'm seeing is right, a known issue and that there isn't anything special going on in the app itself to bypass this issue.

@jdmwood
Copy link

jdmwood commented May 7, 2020

I worked on a similar problem a couple of years ago and saw similar behaviour to what you're describing. Apple are very strict on restricting this kind of stuff so I'd be amazed if a trivial workaround could allow iOS devices to detect each other when locked. Having said all that, a practical test beats all else (so kudos to original poster). Someone else needs to repeat this test to add more data points.

@jamiebishop
Copy link

jamiebishop commented May 7, 2020

I believe they have a workaround which is feasible for use in the wild. We're writing it up now!

@stephenheron
Copy link

stephenheron commented May 7, 2020

Some quite interesting stuff around how they might be getting around the background iOS restrictions can be found in the Android source.

Looking at the Apple documentation on background peripheral bluetooth, they state "All service UUIDs contained in the value of the CBAdvertisementDataServiceUUIDsKey advertisement key are placed in a special “overflow” area; they can be discovered only by an iOS device that is explicitly scanning for them"

I wonder if they have managed to reverse engineer this overflow area so it is accessible via Android?

Someone did some looking into this and it seems at least feasible: https://crownstone.rocks/2018/06/27/ios-advertisements-in-the-background

Will be very interesting to see if we can get some more details on how this operates from the developers.

@jdmwood
Copy link

jdmwood commented May 7, 2020

It definitely appears to be the case that they're using Android to bypass this feature, but the key question remains: what happens if two iOS devices, both locked (inactive) encounter each other when there are no android devices in the vicinity

@lukeredpath
Copy link
Author

lukeredpath commented May 7, 2020

Yes, based on that comment in the Android source and the Crownstone stuff which I was reading yesterday, it does seem like it is possible for Androids to detect the iOS app even if its in the background (although we're all aware of the risks in relying on reverse-engineering a private, undocumented implementation detail like this).

Whether or not an Android device connecting to a background iOS peripheral on a locked device is enough to make that device visible to other locked iOS devices is what I'm not convinced by, until somebody can demonstrate it working and explain how it's doing it. And if that is the workaround, as @jdmwood2 says, that isn't much help if there are no Android devices around - how many encounters will be missed in this scenario?

@jdmwood
Copy link

jdmwood commented May 7, 2020

Just a thought having read a bunch of threads on Twitter about this. Anyone trying to reproduce this must absolutely account for a) new devices finding each other (the OPs point about peripheral state restoration and b) finding each other when both devices are locked before coming into range.

I think it would be very easy to get false positives if you're testing with devices which had already discovered each other (even in a previous test).

The gold standard is: can two locked iOS devices (as in, phone is in your pocket) find each other when they come into range for the for first time.

@lukeredpath
Copy link
Author

lukeredpath commented May 7, 2020

I've continued to do some testing and I'll try and capture this on video tomorrow, but these are the scenarios I've tried. Interestingly it looks like there's more to this than the apps simply being in the background but it could also be related to the power state of the device.

In all scenarios I am starting with the following test conditions:

  • The iPad app is running with the peripheral state restoration code in ConcreteBTLEListener commented out - the behaviour I'm seeing only seems to occur if two devices have not seen each other. If they have seen each other the central restores them from cache during state restoration (I don't know when this expires). I'm aware that this source code modification potentially invalidates these tests, but I think it's a reasonable assumption that doing this is equivalent to using two devices that have never seen each other before.
  • The iPad app is connected to Xcode so I can observe the logs.
  • The app on my iPhone is not connected to Xcode and at the start of each test I ensure that a) the app is running and b) my phone is in Airplane mode.

Scenario 1: App on Device 1 is in foreground, Device 2 app is in background

  1. Launch the app on the iPad from Xcode, observe it is I the foreground.
  2. Unlock my iPhone and turn off Airplane mode.

Result: iPad discovers the iPhone immediately.

Scenario 2: App on Device 1 is in background (screen on), Device 2 app is in background (screen on)

  1. Launch the app on the iPad from Xcode, press the home button and wait for log output to confirm app is in background mode.
  2. Unlock iPhone and turn off Airplane mode.

Result: iPad discovers the iPhone immediately.

#### Scenario 3: App on Device 1 is in background (screen off), Device 2 app is in background (screen on)

  1. Launch the app on the iPad from Xcode, press the home button and wait for log output to confirm app is in background mode.
  2. Lock the iPad.
  3. Unlock iPhone and turn off Airplane mode.

Result: iPad does not discover iPhone (waited for up to 2 minutes keeping iPhone screen on throughout).

After waiting to see that the devices do not discover each other, discovery can be triggered immediately in one of two ways:

  1. Launching the app into the foreground on the iPhone (iPad still locked)
  2. Waking the iPad by pressing the home button to turn the screen on. This works with the iPhone both awake and locked.

In my earlier testing the above scenarios behaved the same if instead of using Airplane mode, I went out of range before launching the app on my iPhone and then came back into range. I'm using Airplane mode for convenience here so I don't have to walk up and down my driveway all night.

So it seems clear to me that the issue isn't solely related to the apps being in the background, but also to do with whether or not the screen is on. Some potential confounding variables that could be throwing off my tests:

  • I'm using an iPad to test instead of iPhones (an iPad mini 3). Could there be something different about the Bluetooth stack and power management compared to an iPhone?
  • Could the iPad being tethered in some way affect the behaviour?
  • Would the results be different if I were to swap the iPad and iPhone in the above scenarios?
  • Could the removed state restoration code be affecting this test, as noted above?

@lukeredpath
Copy link
Author

lukeredpath commented May 7, 2020

Would be great if somebody fancies reproducing my tests above with two iPhones.

@stephenheron
Copy link

stephenheron commented May 7, 2020

I have had strange behaviour in the past with background behaviour while using Xcode to read logs.

I did discover that the app has a debug screen that allows you to see logs while not connected to Xcode. If you shake the device you will get a development menu with a number of useful options.

@lukeredpath
Copy link
Author

lukeredpath commented May 7, 2020

Good spot - just tried scenario 3 untethered from Xcode - I put my iPhone back on Airplane mode again before unlocking the iPad, launching the app and using the debug screen to send the logs to my iMac. Log output confirms the same result.

It also sends over the contact events in plist format so I thought I'd take a look at that with a clean slate.

Deleting the app and re-installing on my iPad was also useful as it seemed to clear the cached state restoration data which means I was able to run the test with the state restoration code intact. I observed the same behaviour as before, here's a snippet of the log from the point I backgrounded the app:

23:40:04 info: Application AppDelegate.swift:183:applicationWillResignActive(_:) – Will Resign Active
23:40:05 info: Application AppDelegate.swift:194:applicationDidEnterBackground(_:) – Did Enter Background
2020-05-07 23:40:05.943201+0100 Sonar[784:192358] Can't end BackgroundTask: no background task exists with identifier 2 (0x2), or it may have already been ended. Break in UIApplicationEndBackgroundTaskError() to debug.

**** TURNED OFF AIRPORT MODE ON MY IPHONE HERE ****

23:40:18 error: Registration RegistratonService.swift:94:register() – Registration did not complete within 20.0 seconds
23:40:18 error: Registration RegistratonService.swift:177:fail(withError:reason:) – Registration failed: Sonar.(unknown context at $100eef8ac).RegistrationTimeoutError
23:40:18 info: BTLE BTLEBroadcaster.swift:217:peripheralManager(_:didReceiveRead:) – responding to read request with empty payload
2020-05-07 23:40:18.836045+0100 Sonar[784:192428] [AppCenter] WARNING: -[MSChannelUnitDefault enqueueItem:flags:]_block_invoke/141 Channel Analytics disabled in log discarding mode, discard this log.
23:40:19 info: BTLE BTLEBroadcaster.swift:217:peripheralManager(_:didReceiveRead:) – responding to read request with empty payload

**** TURNED ON AIRPORT MODE ON MY IPHONE HERE ****

23:42:16 info: Application AppDelegate.swift:198:applicationWillEnterForeground(_:) – Will Enter Foreground
2020-05-07 23:42:16.524522+0100 Sonar[784:192438] [AppCenter] WARNING: -[MSChannelUnitDefault enqueueItem:flags:]_block_invoke/141 Channel Analytics disabled in log discarding mode, discard this log.
23:42:16 info: Application AppDelegate.swift:187:applicationDidBecomeActive(_:) – Did Become Active

As you can see there's no discovery messages in the logs and on inspection the contactEvents.plist file was empty.

@jamiebishop
Copy link

jamiebishop commented May 7, 2020

As promised, here is the blog post explaining how the iOS <-> iOS workaround works. https://reincubate.com/blog/nhs-covid-19-background-tracing-details/

Should hopefully explain the behaviour you’ve been seeing.

@lukeredpath
Copy link
Author

lukeredpath commented May 7, 2020

If I'm reading your blog post right, you're saying that, in order for scenario 3 in my tests above to work effectively it requires either:

  • Existing proximity to an Android device or,
  • Existing proximity with an already established connection to a third iOS device?

If this is the case, that still means the scenario where any two iOS users with their phones locked in their pockets, who have not seen each other before and who are not in range of any Android devices, will still fail?

Also, if the issue seems related to the screen being off more than the app being in a background/suspended state, then where does that fit in to this?

@jamiebishop
Copy link

jamiebishop commented May 7, 2020

Providing the device has a managed to establish a connection with any device (whether it be Android or iOS), it appears as though the device is able to effectively poll as though it is in the foreground even when the device is locked.

If this is the case, that still means the scenario where any two iOS users with their phones locked in their pockets, who have not seen each other before and who are not in range of any Android devices, will still fail?

This would depend on the time it last saw an Android or iOS device, because there is a grace period before the app is suspended again (and thus falls back into limited BT-polling mode)

@lukeredpath
Copy link
Author

lukeredpath commented May 7, 2020

OK, I will re-test scenario three with a third device tomorrow. Is this what you'd expect to see?

  1. iPad app in background, screen locked.
  2. Turn off Airplane mode on iPhone 1, leave app in background - discovery fails as per the above test.
  3. Turn off Airplane mode on iPhone 2 - still no discovery of this one yet.
  4. Bring app on iPhone 2 into foreground - iPad should discover iPhone 2 and iPhone 1?

And for completeness:

  1. iPad app in background, screen locked.
  2. Leave iPhone 1 in Airplane mode.
  3. Turn off Airplane mode on iPhone 2 and run the app so it is discovered by the iPad.
  4. Turn on Airplane mode on iPhone 2 to disconnect it from the iPad.
  5. Within some undefined window after this, turn off Airplane mode on iPhone 1 and it should be discovered?

@Jeff-Stelling
Copy link

Jeff-Stelling commented May 7, 2020

So what happens with the app if no android devices are around to wakeup?

@jamiebishop
Copy link

jamiebishop commented May 7, 2020

@lukeredpath I’m not sure what I’d expect to see under those scenarios at the moment if I’m honest as I’m exhausted and have a headache (it’s all that reversing!). I’ll get back to you in the morning, but hey there’s no harm in trying.

@jamiebishop
Copy link

jamiebishop commented May 7, 2020

@Jeff-Stelling and there’s no other iOS devices around for it to talk to? The device would go silent until woken by an Android device

@lukeredpath
Copy link
Author

lukeredpath commented May 7, 2020

For those interested in how the keep-alive works, it's effectively a ping-pong between the central/peripheral running on each device:

  1. After discovery, the central on each device subscribes to changes in the keep-alive characteristic of the other device's peripheral.

  2. Device 1 central (the listener) gets an RSSI ping from Device 2's peripheral and triggers the didReadRSSI delegate callback.

  3. Device 1 uses the above ping as a trigger to update the value of its the keep alive peripheral characteristic.

  4. Device 2 is notified of the change to Device 1's keep-alive value and triggers the didUpdateValueFor delegate callback where it in turn triggers an update of its keep-alive characteristic, notifying Device 1.

  5. The above process also happens in reverse, with Device 2 updating the its keep-alive value when when it gets an RSSI ping from Device 1 and Device 1 updating its keep-alive value when it receives the updated value from Device 2.

  6. There's some throttling in place to ensure each device only updates it's keep-alive value no more than every 8 seconds.

Thanks to Jamie for his investigation into this. If the final conclusion is that a) the behaviour I'm seeing is indeed a genuine issue and b) the workaround is as Jamie describes then I'm not very confident it will be an effective workaround, although I do applaud whoever came up with it because it's probably the best possible workaround you could manage. I hope in wide-scale testing I'm proved wrong (although if you're hoping this approach is abandoned completely in favour of the decentralised Apple/Google solution, then perhaps you're hoping I'm right).

@deathchurch
Copy link

deathchurch commented May 8, 2020

What happens when Apple fix this bug that allows Android devices to wake background processes on iOS, this is clearly a battery drain and should be fixed?

@chriswunsch00
Copy link

chriswunsch00 commented May 8, 2020

What happens when Apple fix this bug that allows Android devices to wake background processes on iOS, this is clearly a battery drain and should be fixed?

Yeah, if only Apple and Google had provided a proper solution to this problem ...

@mrjcgoodwin
Copy link

mrjcgoodwin commented May 8, 2020

What happens when Apple fix this bug that allows Android devices to wake background processes on iOS, this is clearly a battery drain and should be fixed?

Read through this thread, the whole time thinking exactly that. Surely Apple are gonna close the workaround.

@lukeredpath
Copy link
Author

lukeredpath commented May 8, 2020

If Jamie is right then it’s not just Android devices that are the key to this - the idea is that any other device actively scanning for the service UUID should in theory wake the locked device - so Android devices or other unlocked iPhones. I don’t have any Android to test but will test this with an unlocked iPhone later. It’s hard to say if this is a bug in iOS or just expected behaviour.

@thoutbeckers
Copy link

thoutbeckers commented May 8, 2020

It's not a "workaround" or "bug", it's using a legitimate Apple Framework that allows your app to function as a peripheral.

This functionality is not limited to use by Android phones, it's also used by other iOS devices (where the app is running in the foreground) to detect the advertising iPhone . Hence why an iOS app that is open can also act as a bridge between two devices where the app is in the background. "Fixing" this could break many other apps that use this for good reasons.

The difference is in how iOS chooses to advertise services (requiring an extra step to discover them) and how it scans for peripherals in foreground (taking the extra step) vs background (just passively listening without taking the extra step).

@thoutbeckers
Copy link

thoutbeckers commented May 8, 2020

Of course the Exposure Notification framework from Apple and Google fixes this by advertising the relevant service without requiring any extra steps to discover it. The method is uses to do this is not available for regular apps.

Since the privacy preserving features from that framework or not compatible with the backend architecture of the NHS it will not be useable however for this app.

@jdmwood
Copy link

jdmwood commented May 8, 2020

@thoutbeckers it absolutely is a workaround if you're using an API in the manner that the owners didn't intend. I can't believe Apple intended the peripheral API (something intended for communicating with devices) to be constantly scanning in the background. If they did, they wouldn't have put in the restriction in the first place. Apple absolutely don't care about breaking existing apps if they feel the user experience of their customers (battery life, security) is an issue.

@nrbrook
Copy link

nrbrook commented May 8, 2020

Firstly, a good way to understand the behaviour of the App is to use PacketLogger (available in additional tools for Xcode), connect to one device (must have Bluetooth profile installed), and filter on "85bf" (the start of the identifier characteristic, which must be read when one device is discovering another). In this way you can determine if the released App is working in a given scenario.

If there is an Android in range it will wake the iOS App most of the time (unless advertising has been stopped by iOS after 1h50m of inactivity). However, it seems there is a bug in this which will prevent the Android App discovering iOS devices that have multiple background advertisements, although that is easily fixed.

The way the Android App discovers the iOS App is the way backgrounded advertising in iOS works, and preventing it would mean breaking the functionality between iOS devices. But I could see Apple going this far, because they are very careful about this stuff.

My testing showed that the one scenario it won't work is when there are only iOS devices in range and screens are off. Even tapping the screen to wake it is sufficient for the devices to communicate. People use their phones a lot, and there is probably an Android phone around most of the time, but this is one place where it will fail some of the time.

Another scenario is if the screen is on and the App has been in the background for a long time (>1h50m, perhaps overnight) then it will not reconnect to another backgrounded iOS device. There must be no Android phones in range for this to occur. I witnessed this with two iOS devices with the App were in range overnight. Presumably this is because the screens were off and so the Apps were not able to discover and connect to each other, so eventually the iOS background scanning/advertising timeout was triggered. If either device had been woken up before the timeout it may have been sufficient to refresh the App. iOS seems to give it 6540 seconds (1h50m) of background advertising time.

In summary it might work "well enough", but some contacts will definitely be missed. Will they accept that and continue anyway because the data they can obtain is "more useful"? Or will they switch to the Apple-Google model being standardised in several other countries in Europe? It seems like they may be reconsidering their position...

@lukeredpath
Copy link
Author

lukeredpath commented May 11, 2020

@christhomas for the record the app doesn’t actually gather any location data (it doesn’t need it), the iOS app doesn’t even request location access. You can see what data is stored in the contact event file in the examples I posted above.

@christhomas
Copy link

christhomas commented May 11, 2020

Well that's a partial relief. I'd read in previous versions it was uploading location data as well.

Still, if you had a choice between an app which only works with the NHS in the UK, or an app supported by your operating system globally that doesn't violate your privacy in the above described ways. I can't imagine why you'd want to use this app. The battery drain alone is enough to make me avoid it. My phone lasts a little more than a day as it is.

But I am grateful you've done this analysis. It was very enlightening.

@edent
Copy link
Contributor

edent commented May 18, 2020

Participants are reminded of this repo's Code of Conduct.

Specifically:

be respectful of differing viewpoints and experiences

I have removed a couple of comments. We're very happy to have technical discussions on this thread.

@nihp-public nihp-public deleted a comment from christhomas May 18, 2020
@nihp-public nihp-public deleted a comment from philipswift May 18, 2020
@nihp-public nihp-public deleted a comment from christhomas May 18, 2020
@nihp-public nihp-public deleted a comment from philipswift May 18, 2020
@nihp-public nihp-public deleted a comment from j-r-t May 18, 2020
@nihp-public nihp-public deleted a comment from christhomas May 18, 2020
@nrbrook
Copy link

nrbrook commented May 18, 2020

I am increasingly concerned that it seems the government is continuing to push forward with this App, but we have had no comment on the issues raised here other than the mention of the testing at RAF Leeming. This article does not provide any detail on the testing. Useful details would include the hardware used, OS versions, and the time the App had been in the background before the tests were undertaken. A test where the App is installed and then immediately tested is not sufficient to test background functionality.
The Isle of Wight test cannot provide useful feedback on missed contacts because of the nature of the problem, unless the Beta App is verifying tracing via location tracking or other means, but the code on this repository and the permissions it requires suggests it isn’t.
Can anyone from the development team provide additional detail on the testing undertaken and how we could verify the results for ourselves?

@hiddentao
Copy link

hiddentao commented May 20, 2020

This problem isn’t unique to the NHSX app of course. Competing frameworks like DP-3T will have the same Bluetooth issues (until they switch to the Apple exposure framework anyway).

The Apple/Google API is heavily inspired by DP-3T so I doubt they will have the same issues.

@thoutbeckers
Copy link

thoutbeckers commented May 20, 2020

The Apple/Google API is heavily inspired by DP-3T so I doubt they will have the same issues.

Architecture has nothing to do with access to APIs. DP-3T has the same problem as this app with background support, just look in their source.

@lukeredpath
Copy link
Author

lukeredpath commented May 20, 2020

Yes, what @thoutbeckers said. The issue is going to be a problem for any app trying to roll their own Core Bluetooth contact tracing system.

However, DP-3T can be made to work with the Apple/Google APIs as its decentralised whereas this app cannot, as it stands.

@thoutbeckers
Copy link

thoutbeckers commented May 20, 2020

DP-3T version 2 can not be made to work Apple/Google, as the Cuckoo filter cannot be applied on device due to Apple/Google doing all the work matching the exposure keys.

The same more or less applies to version 1. The differences are less big, but you still need to adjust to the differences, and in the end you end up with Google/Apple, not DP-3T. But it will fit reasonably well with the design of an app made for DP-3T.

To keep it on-topic, the rework for NHSX would probably be a lot bigger since the architecture differences are much bigger.

@lukeredpath
Copy link
Author

lukeredpath commented May 20, 2020

Interesting, I was under the impression that DP-3T was going to use the Apple/Google framework. If they aren’t, then it’s likely to be no more successful than this approach for the same technical reasons.

Perhaps people should now be looking to the work Shopify has done on this instead.

https://www.covidshield.app/

@nrbrook
Copy link

nrbrook commented Jun 14, 2020

This in the Telegraph today:

Gus Hosein, a member of the ethics advisory board overseeing the app’s development, described a “frustrating” lack of communication with the government over data resulting from Bluetooth tests.

“I have asked repeatedly for data on how the trials and tests went with Bluetooth data, and they have never shared those results,” he said.

However, signs have emerged that there may be issues with how the app is locating users. A contract released by the Government last week showed the focus of the app’s testing was around “whether the proximity service satisfies the requirements”.

It is not clear that these issues are the issues discussed in this thread, but we have seen no evidence that the issues identified here over a month ago have been mitigated or even thoroughly tested. We could have an App using the Apple Google framework released and in use now...

@hiddentao
Copy link

hiddentao commented Jun 14, 2020

On a side note, it's not clear whether these sorts of apps are going to really catch on / actually work as intended -> https://www.nbcnews.com/tech/tech-news/coronavirus-contact-tracing-apps-were-tech-s-chance-step-they-n1230211

"The factual analytical assessment is it's not a high upside," said Andy Slavitt, an Obama administration health care official who is chair of the nonprofit United States of Care. "The bulk of the investment you need to make is in manpower."

At some point, some of the developers behind the efforts renamed the apps as "exposure notification" projects rather than contact tracing — in effect, removing a connotation about tracking while also possibly lowering expectations of what the apps were going to do.

On this second point the app linked to in #2 (comment) is exactly that - "exposure notification".

@lukeredpath
Copy link
Author

lukeredpath commented Jun 14, 2020

@nrbrook I suspect it could be they are referring to problems getting accurate proximity from the Bluetooth signal strength which is a genuine concern that people ha e raised and it applies to the Apple/Google solution too. Nobody knows how reliably this will work in the long run.

@nrbrook
Copy link

nrbrook commented Jun 14, 2020

@lukeredpath it does seem to be focussed on proximity issues, but it does say

Experts have claimed that the use of Bluetooth technology to track people involves a number of limitations, including a lack of accuracy that could register phones a distance away.

@nrbrook
Copy link

nrbrook commented Jun 17, 2020

This article in The Guardian perhaps gives an indication of how well the UK app might perform, given the COVIDSafe App doesn’t use the Apple-Google framework either. This seems to match our experiences:

The federal government’s Covidsafe contact tracing app works as few as one in every four times for some devices, documents tabled in the Senate have revealed.

CF0FD754-F88C-48BB-A69E-7A5E02577CE2

Their ‘moderate’ rating for Android to iPhone when locked suggests they might not be scanning for iPhone background advertisements correctly in the Android App (the UK app has an open issue for this too). But the iPhone locked to iPhone locked case matches our experience: it is flakey at best.

Edit: It seems the App has now been delayed until the Winter and is ‘no longer a priority’. I wonder how much is due to this issue, and the other technical and privacy issues.

@deathchurch
Copy link

deathchurch commented Jun 18, 2020

Good to see they have finally realised the Apple/Google model was the best way to go, lets hope this speeds up the public release of this app

@TheOtterlord
Copy link

TheOtterlord commented Jun 18, 2020

Just thought I'd add this article to the discussion: UK virus-tracing app switches to Apple-Google model.

@nrbrook
Copy link

nrbrook commented Jun 18, 2020

@TheOtterlord

The centralised version trialled on the Isle of Wight worked well at assessing the distance between two users, but was poor at recognising Apple's iPhones.

Specifically, the software registered about 75% of nearby Android handsets but only 4% of iPhones.

By contrast, the Apple-Google model logged 99% of both Android mobiles and iPhones. But its distance calculations were weaker.

So it seems it was this issue that caused the switch. I wonder how these figures were obtained – how was the number of missed contacts determined in a real world trial?

@lukeredpath
Copy link
Author

lukeredpath commented Jun 18, 2020

I can't say I'm really surprised - we all knew from the outset that the only way for this to work reliably (at least in terms of detection) is to use the Apple/Google frameworks as it circumvents all of the limitations of the various background modes.

The question still remains to be answered about the reliability of detecting proximity accurately enough for this to be useful in the real world but I imagine Apple and Google can refine this over time.

With all this said and done, once the switch to the decentralised model is confirmed, I think it's will be time to close this issue as "can't fix, won't fix". ;)

@TheOtterlord
Copy link

TheOtterlord commented Jun 18, 2020

@nrbrook I think it was a number of issues, including this one and others related to security and privacy. I don't think all the data from the trial was published as far as I've seen, but I may be wrong.

@lukeredpath Yeah, this was always going to be the case. The system used in the retired version is unreliable and it makes no sense to continue development with this. The Apple/Google version seems to work really well when you look at other apps already using it around the world. I am not sure why they did not use it sooner.

@lukeredpath
Copy link
Author

lukeredpath commented Jun 18, 2020

Well there we have it - Hancock did his best to try and throw Apple under the bus on this one, but I don't think it will work.

If it's true that something about the way this app is calculating proximity is better than what Apple/Google are doing then I would hope that Apple and Google will work with NHSX to try and incorporate those improvements into the Exposure Notification framework.

I think now we can officially put this issue to bed!

@thoutbeckers
Copy link

thoutbeckers commented Jun 19, 2020

Specifically, the software registered about 75% of nearby Android handsets but only 4% of iPhones.
By contrast, the Apple-Google model logged 99% of both Android mobiles and iPhones.

Let that be a lesson to other tracing apps, particularly if anyone involved is reading this thread. Testing sitting behind your desk fiddling with your phone will not give the same results as in the field when people have their phone in their pockets.

As much as I'd like to blame managers or politicians for this one, as this thread demonstrates developers are more than happy to make grand claims based on very little data and poor methodology. A healthy distrust of IT is needed for something as important as fighting a pandemic. While I don't think it ever should have gotten this far (as the abysmal test results indicate this was a waste of time) I've got to commend the NHS for at least doing a large scale test first (looking at you, Singapore/Australia/France).

I hope a similar skepticism will be leveled against Apple & Google. While the relatively easy problems of discoverability and decent-enough privacy are solved by them, distance measuring is something that can be vastly improved still (but will require protocol level changes most likely). The decisions about tradeoffs on this point should not be left to just software engineers, even if they are some of the best in the world.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests