Static experiment data in real setting and pre-GAEN data

[pdehaye]

All the data available so far seems to be collected in mocked up environments.
It is said here that static studies have also been realized in real environments ("e.g., an actual tram").

Please release that data as well.

[gannimo]

Thanks for the comment, your request is noted. We sufficiently validated the data during the experiments to make sure our experimental setup is close to the real world. Note that the BT experiments serve two purposes: a) demonstrate that reasonably precise contact estimation is possible and b) provide parameters to estimate the cut-offs as determined by epidemiologists and the COVID task force.

Many of our early outdoor (and indoor) experiments were conducted pre-GAEN using different calibration apps that we developed for testing. The collected data is therefore in proprietary formats. While we did measure with different TX power and scan intervals, processing and translating this data takes quite a bit of time---as does writing all the translation and analysis scripts.

Given the unfortunately highly political nature of the discussion around BT measurements, we cannot release the raw data without providing the corresponding translation scripts (e.g., the data is uncalibrated) and at least a lens for interpretation.

Note that, now that we are very confident that BT proximity estimation works at reasonably high precision, we are no longer investing as much time into raw BT measurements. Google will continue to improve the calibration of the devices, further increasing the precision of the proximity estimation and, similarly, the GAEN protocol will evolve to provide better estimation as well. We're currently faced with many concurrent requests and, while fun, Bluetooth measurements take a lot of our time. Requests for release of other older experimental data therefore will not receive high priority.

I'll keep this issue open so that we can track and discuss.

[pdehaye]

Thank you for the comment, your response is noted.

You say quite a few things in your response that go beyond the tram/real setting situation, let me break it down:

1. you have data from the pre-GAEN apps;
2. beyond calibration issues, the GAEN protocol itself will evolve to provide better estimation.

We know there were/are/will be four stages for the Bluetooth to distance inference part:

1. pre-GAEN,
2. GAEN 1.2,
3. version 1.6,
4. new improvements suggested above.

The Digital Contact Tracing sub-unit of the Task Force has claimed when switching from pre-GAEN to GAEN 1.2 that working with Google/Apple would improve precision. This is yet to be confirmed. All we know - because it is stated in the README here (and it could be recomputed from the data here) is that there was an improvement of 5% between GAEN 1.2 and version 1.6. We also know from measurements and research originating in other teams (PEPP-PT, French team, MIT) that it is likely that the pre-GAEN version was more precise than version 1.6 currently is.

While fully understanding of the time constraints on your side, I still cannot fathom how you would claim your setup was realistic enough, when other experiments - that you can only know about - have shown how important the realism of the experiment actually is.

You are correct that the conversation around BT measurements have become intensely political. There is good reason for this: claims by DP-3T members about BT measurements and the wisdom of associating with Google and Apple have contributed to a reorganization of the entire stack around contact tracing apps, sometimes putting entire governments in very uneasy situations. In this sense, statements about BT that have yet to be evidenced have contributed to a shift in the distribution of power, which perfectly fits the definition of politics as a struggle for power, in which DP-3T and associates were just (willing) pawns.

Previously there was growing distrust between authorities responding to the emergency, with division arguably stoked by DP-3T. Now, there is growing distrust from the general population in the authorities responding to the epidemic. I think you have a responsibility not to make it worse by lacking transparency, lest overeager efforts to convince SwissCovid is accurate spill over into detrimental effects on the public perception of vaccines, etc.

[doug-leith]

With respect Mathias, making strong assertions and then witholding the data that backs them up seems poor practice and rather unscholarly, to say the least. If the data is not available then I suggest the appropriate course would be to withdraw the assertions until supporting data can be made publicly available.

There are reasonable concerns regarding the validity of the assertions you make since other measurement studies taken in an actual bus and tram (e.g. cited by Paul above) would seem to be at odds with your claims. A tram/train/bus is a metal rich environment with metal walls, floor, ceiling, furniture. It is well known that radio signals are strongly reflected by metal surfaces, and previous (non-covid) studies in trains have highlighted that reflections can significantly affect the change in received signal strength with distance. Mocked up tests in an open space obviously fail to account for such effects, and so indeed it would be surprising if those measurements in fact lined up with measurements taken in an actual tram as you claim. These are not small matters, and should not be simply dismissed.

[gannimo]

Thanks for your comments. We believe that our dataset is currently the most realistic openly available dataset. Experiment 34 uses a range of phone models commonly found in Switzerland, RPIs were created and configured by the GAEN API, and packet dumps were collected from the phone's BT interfaces.