Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Javascript 120Hz devicemotion events for high end inertial applications #98

Open
lukes3315 opened this issue Apr 6, 2016 · 54 comments

Comments

9 participants
@lukes3315
Copy link

commented Apr 6, 2016

Browsers only sample sensor data at a varying 67Hz (drops down to 1Hz sometimes), which makes the readings unusable for high end use cases...
We need a robust 120Hz steady sampling rate for our human inertial navigation technology. iOS and Android do have this natively, but apparently current browser technology does not offer the same quality as the native sampling... Sadly :(

@tobie

This comment has been minimized.

Copy link
Member

commented Apr 6, 2016

Thanks for your input. This is addressed by the spec for sensors which support periodic reporting mode.

Which particular sensors would you need to poll at 120Hz?

Why 120Hz and not 1KHz or 67Hz?

What are you aiming to achieve with this? E.g. is this a latency issue, are you filtering the data and need a number of data points to do so? etc.

What are the precise use cases you're targeting?

Thanks for your input.

@lukes3315

This comment has been minimized.

Copy link
Author

commented Apr 6, 2016

We would need:

  • Accelerometer
  • Gyroscope
  • Magnetometer

And if possible:

  • Barometer

Well 120Hz is sufficient granularity for our application. 67Hz is not granular enough, 1Khz, iOS and Android will not let you poll at such a high frequency. But if you want to have a sample rate of 1Khz that isn't an issue as long as it is adjustable.

We are working on an indoor location technology based on inertial navigation. The current issue is the inconsistency in the data points, for example:
Timing between samples seems to be inconsistent and inaccurate.

Check us out:
www.navisens.com

@tobie

This comment has been minimized.

Copy link
Member

commented Apr 6, 2016

Thanks for this added information.

@tobie tobie added this to the Level 1 milestone Oct 27, 2016

@tobie tobie self-assigned this Oct 28, 2016

@tobie

This comment has been minimized.

Copy link
Member

commented Jan 26, 2017

@lukes3315

I have a couple of options for this, please LMK which are acceptable.

  1. polling at 120 Hz, getting roughly two data points together each time requestAnimationFrame is called?
  2. polling at 120 Hz, getting only the latest data point when requestAnimationFrame is called (this is good for latency, not so much if you want all data point, e.g. for filtering)?
  3. polling at 120Hz, receiving updates at 120Hz via event observers.

@tobie tobie removed the ED-ready label Jan 26, 2017

@tobie

This comment has been minimized.

Copy link
Member

commented Jan 26, 2017

Expecting feedback from Navisens shortly, see: https://twitter.com/navisens/status/824627997797801985.

@lukes3315

This comment has been minimized.

Copy link
Author

commented Jan 26, 2017

Hi Tobie,

We think there should be two acquisition methods supported:
1/ Polling sensor samples where you request 1 to N samples
which are then received at each polling loop
2/ Passing a callback function to receive 1 to N samples (preventing polling
for each individual sample).

Having the option to batch N samples would allow for a tradeoff between low latency
and greater efficiency depending on the application.

@lukes3315

This comment has been minimized.

Copy link
Author

commented Jan 26, 2017

Please feel free to reach out if you need more information.

@tobie

This comment has been minimized.

Copy link
Member

commented Jan 26, 2017

Thanks, @lukes3315.

What do you mean by "polling loop"? I'm also not sure what you mean by option 2 either. :-/

Maybe code examples or f2f would be helpful?

@tobie

This comment has been minimized.

Copy link
Member

commented Feb 15, 2017

Had a meeting with Luke (@lukes3315) and Ash (@adonikian) of Navisens. Navisens does indoors positioning without touching the GPS, relying instead on motion sensors in users' mobile phones. It collects readings on the device an sends them to remote servers for analysis where it notably corrects for drift using machine learning.

Currently, Navisens is available as native apps. The current max sampling rate of the DeviceOrientation API (~60 Hz) isn't enough for their requirements.

They's ideally need 120 Hz, but could work with 100 Hz.

The sampling intervals need to be consistant. And it's critical that no samples are lost. [Ed: I'd love to understand more about about why that's important.]

Latency, on the other hand, is less an issue, as the readings as sent to a server for analysis anyway.

The kind of API they have in mind would allow them to set the size of the sample batch they want to work with and get an event every time the batch is full so they can send the data over to their servers for analysis.

[Ed: it would be useful to get a better understanding of the kind of analysis that's being done on the readings to justify the high sampling rates requirements and the necessity of not dropping samples and have an idea of the batch sizes.]

@tobie

This comment has been minimized.

Copy link
Member

commented Feb 15, 2017

Hearing similar use cases from @padenot:

for example, gyroscope at 120Hz sending control information via websocket to a web audio api program, controling various parameter of a synthetizer

@padenot: would be amazingly helpful if you could provide requirements around:

  • latency (from time the data point is collected to time you get to use it),
  • frequency (how many samples per seconds you need), and
  • whether it's OK to batch the samples together (and if so, how often you need these batches sent).
@padenot

This comment has been minimized.

Copy link

commented Feb 16, 2017

latency (from time the data point is collected to time you get to use it),

This is very dependent on your application. It is generally considered that a normal human (i.e. not a professional drummer, for example) considers that sound happens "immediately" if the delay between doing the thing that makes your sensor react (be it a gyroscope, a fader, a knob, a keyboard press, etc.) is less than 20ms.

Now, you should experiment. Depending on the interaction, the type of sound, the parameter that is being modulated, etc., it is possible that you'll find that 100ms is acceptable. Keep in mind that the output latency of a normal computer using the Web Audio API is between 10ms and 40ms. It's possible to bring this down with special hardware and browser builds.

frequency (how many samples per seconds you need), and

It depends on the use case, for controlling parameters. Any rate is manageable with the Web Audio API, since you're simply doing js calls. There is a possibility to schedule a bunch of things in advance, or to react just in time, etc.

whether it's OK to batch the samples together (and if so, how often you need these batches sent).

This is ok, iff you can take the latency hit caused by the packetizing of the data.

@tobie

This comment has been minimized.

Copy link
Member

commented Feb 16, 2017

@padenot thanks for your comments.

(For the fun story, I was a professional drummer in a previous life, dabbled with controlling early versions of Live and NI software with a drumKAT and experienced sub 20ms issues first hand when studios moved from analog to digital recording.)

To give you a bit more context, there are security concerns about providing > 60 Hz frequency and even if we managed to work around this, there are perf concerns around providing the data samples faster than the animation frame rate.

If this is a concern for some use cases, I'd like to hear about it, to see if those are use cases we want to fight for or knowingly exclude.

@padenot

This comment has been minimized.

Copy link

commented Feb 16, 2017

Not for music in the normal sense, but critical for VR and related technologies. You can't really do anything competitive in VR with 60Hz data.

@tobie

This comment has been minimized.

Copy link
Member

commented Feb 16, 2017

@padenot OK, cool. (I'm aware of the VR requirements, they generally want higher frequency to reduce latency, but don't really care about more than one data point per animation frame. When they do, they're fine getting them batched).

@tobie

This comment has been minimized.

Copy link
Member

commented Feb 20, 2017

From the feedback gathered so far here, it seems a reporting frequency whose max is tied to the animation frame rate is sufficient.

On the other hand, a faster polling frequency seems necessary to either:

  1. gather more data points (indoor navigation use case), or
  2. decrease latency (VR use case).

There are privacy concerns with sampling > 60 Hz, though my understanding is research by @maryammjd seems to indicate this is an issue at much lower frequency rates already. (@maryammjd can you please confirm, plus point to relevant papers?)

When polling frequency is higher than reporting frequency and we don't want to loose samples (so not the VR case), we need to buffer samples. This needs discussion as to how best to achieve this. BYOB? Dedicated API? Etc.

@adonikian

This comment has been minimized.

Copy link

commented Feb 20, 2017

@tobie my response to:

The sampling intervals need to be consistent. And it's critical that no samples are lost. [Ed: I'd love to understand more about about why that's important.]

This is very important, because most applications using inertial sensors use incremental estimation where the current state (e.g. position, velocity, attitude) is composed of a a set of successive relative state estimates. For inertial sensors, this is not just dependent on the sensor readings but also having correct timing information, since the sensor readings are often integrated. Thus, if a sample is lost, it is generally unrecoverable (other than trying to compensate with global information such as the magnetometer).

@maryammjd

This comment has been minimized.

Copy link

commented Feb 21, 2017

Hi all, may I please ask you to clarify what is the exact problem? In terms of sampling rate, we have never experience 1 Hz with motion and orientation (we have been testing different browsers for years). Is this something browser-related or network-related (e.g. connection problems)?

From the security point of view, as long as the right permission is given to the eligible code, we don't mind the sampling rate. In fact, we believe increasing the sampling rate both on native apps and js in inevitable.

@tobie

This comment has been minimized.

Copy link
Member

commented Feb 21, 2017

@maryammjd, thanks for your reply. A couple of comments:

Hi all, may I please ask you to clarify what is the exact problem?\

Security and privacy concerns of using sensors at frequencies above 60 Hz.

In terms of sampling rate, we have never experience 1 Hz with motion and orientation (we have been testing different browsers for years). Is this something browser-related or network-related (e.g. connection problems)?

Not sure what you mean. I don't see any reference to 1 Hz in the whole thread. Did you misread 1 KHz way above?

From the security point of view, as long as the right permission is given to the eligible code, we don't mind the sampling rate. In fact, we believe increasing the sampling rate both on native apps and js in inevitable.

Vendor security teams do care however. I'm interested in determining whether they are right to care or not. That is, does increasing frequency create new attack scenarios (e.g. key logging of a nearby keyboard, pin logging on touch phones, crude voice recording, etc). And if so, at what frequency do they appear? What mitigation strategy exist beyond limiting the frequency (e.g. making sure the sensor doesn't report reading when the user is entering their pin code).

@maryammjd

This comment has been minimized.

Copy link

commented Feb 21, 2017

The very first comment of this thread says: "Browsers only sample sensor data at a varying 67Hz (drops down to 1Hz sometimes)".

Research-wise, sensors at any frequency (even 20 Hz) are able to reveal something about the user's activity (touch actions, PINs, etc.) or his surrounding environment. We can't say 5 hz or 60 hz or 200 hz is safe or not. However, the higher frequency, the better attack results; the lower frequency, the more expensive machine learning techniques, the attackers will find their way through anyway. That is why we advise to have design-level security models rather than implementation limitation solutions i.e. playing with the sampling rates.

@tobie

This comment has been minimized.

Copy link
Member

commented Feb 21, 2017

The very first comment of this thread says: "Browsers only sample sensor data at a varying 67Hz (drops down to 1Hz sometimes)".

Indeed. Sorry. :)

Research-wise, sensors at any frequency (even 20 Hz) are able to reveal something about the user's activity (touch actions, PINs, etc.) or his surrounding environment. We can't say 5 hz or 60 hz or 200 hz is safe or not. However, the higher frequency, the better attack results; the lower frequency, the more expensive machine learning techniques, the attackers will find their way through anyway. That is why we advise to have design-level security models rather than implementation limitation solutions i.e. playing with the sampling rates.

Do you have pointers to papers that discuss this? That would be super helpful!

@maryammjd

This comment has been minimized.

Copy link

commented Feb 21, 2017

TouchSignatures: Identification of User Touch Actions and PINs Based on Mobile Sensor Data via JavaScript

Results are presented for Touch Actions on iPhone (20 Hz), and PIN digits on iPhone (20 Hz) and Android (<60 Hz)

@tobie

This comment has been minimized.

Copy link
Member

commented Feb 21, 2017

Thanks are you aware of other papers that would show similar patterns for other privacy/security concerns?

@maryammjd

This comment has been minimized.

Copy link

commented Mar 6, 2017

As far as I know, Chrome is aware of this issue. As you can see the bug has been reported in 2014.

@tobie

This comment has been minimized.

Copy link
Member

commented Mar 6, 2017

As far as I know, Chrome is aware of this issue. As you can see the bug has been reported in 2014.

Yes they are indeed. :)

@timvolodine

This comment has been minimized.

Copy link

commented Mar 17, 2017

To chime in here, I remember two issues regarding increased frequency:

  • battery drain (which I guess is inevitable for VR and depends on hardware)
  • malicious 'attacks' e.g. fingerprinting or even 'eavesdropping'

IIRC in crbug.com/421691 one of the papers talks about audio attacks and increasing the sampling rate naturally increases the risk and accuracy of such attacks.. As was already mentioned it's not exactly clear what the reasonable threshold is though and the actual impact of marginal increase.

To reduce potential risks we can restrict the usage e.g. to top main frame only, and/or by requiring a user gesture. Additionally we could make the usage opt-in e.g. <iframe allow=sensors_hifrequency>, which would provide extra validation. It seems natural to also disable the sensors after some inactivity and of course when page is not visible or running in background..

I guess this would factor in the 'design-level' security approach mentioned by maryammjd@.

@tobie

This comment has been minimized.

Copy link
Member

commented Mar 20, 2017

Hey @timvolodine.

Thanks for chiming in.

There are two reasons why high frequencies might be warranted, which map directly to a bunch of key uses case we want to unlock.

  1. Faster frequency allows to lower latency, which is key to fight motion sickness in VR.
  2. You might just require more data points for certain things (e.g. inertial apps as described by OP).

As you mentioned, increased latency increased battery consumption, though I'm not sure how the numbers stack-up compared to say using a gyroscope vs. an accelerometer (i.e. from a battery usage perspective, is polling a gyroscope at 60 Hz cheaper or more expensive than polling an accelerometer at 240Hz). It would be nice to have more data on this.

The security concerns are more worrying. They're pretty serious and hard to explain to users which isn't a good combo.

As you mentioned, we have a number of lines of defence for those:

  • liming sensor access to secure contexts,
  • limiting sensor access to top level browsing contexts (and ultimately opening that up through Feature policy),
  • pausing sensors when page visibility changes to hidden,
  • caping polling frequency,
  • fuzzing frequency / timestamps,
  • limiting data accuracy,
  • relying on permissions (either by letting users set them on through whatever other heuristics),
  • disable sensor usage when user becomes inactive (that might be a tough one if you have both highly sensitive sensors and are only using sensors as user input).

You also added user gesture in there, which I hadn't thought about and aren't too familiar with. Pointers to how this would be helpful here would be welcomed.

Each of those have different advantages and disadvantages. The key is to tie it all back to the use cases we want to enable.

I was hoping to move forward with a binary approach to permissions, but it sounds like we might want to consider a more fine-grained strategy using dedicated PermissionDescriptors, for example to require non-fuzzy data, higher frequencies, or high-accuracy.

@maryammjd

This comment has been minimized.

Copy link

commented Apr 4, 2017

@tobie if you have concrete ideas on how to design new permission models and want us to test them at uni, we are happy to do that for you. We can simulate your solutions and run them with focused groups to study the usability issues.

@tobie

This comment has been minimized.

Copy link
Member

commented Apr 4, 2017

@maryammjd Amazing!

I'm in the process of doing a writeup on permissions which I'm planning to share shortly.

Maybe @owencm has suggestions here too.

@timvolodine

This comment has been minimized.

Copy link

commented Apr 10, 2017

@tobie by "user gesture" I meant an input from the user (e.g. permission or dialog notifying that sensors are being enabled) which may help to avoid activation "by default" and inform the user something is happening (which the user could refuse in some way). Not sure how helpful this is from the UI perspective though..

@alexshalamov

This comment has been minimized.

Copy link

commented Apr 11, 2017

@maryammjd Based on your research work and also papers from other security researchers (gyromic, ACCessory, Pin skimming, ALS Pin skimming).

It looks like the security & privacy risks come from the data exposed by low-level sensors (gyro,accel, als and magn).

What would be the risk estimation for a high-level orientation sensor?

We are collecting data for the security & privacy section of the Chromium implementation design document to find out optimal permission model for sensors. Your feedback would be very much appreciated.

@tobie

This comment has been minimized.

Copy link
Member

commented Apr 11, 2017

@alexshalamov the research @maryammjd carried out in "TouchSignatures: Identification of User Touch Actions and PINs Based on Mobile Sensor Data via JavaScript" relies on the DeviceOrientationEvent, itself pretty high-level. :)

@alexshalamov

This comment has been minimized.

Copy link

commented Apr 11, 2017

@tobie If I remember correctly, it uses motion + orientation. Would be nice to know security risks just for orientation sensor, without motion data.

@tobie

This comment has been minimized.

Copy link
Member

commented Apr 11, 2017

I think @maryammjd addresses this in the above-mentioned paper (p. 21) by comparing her team's work with TouchLogger which only relies on the orientation sensor:

Taplogger performs its experiments on Android devices and identifies 36.4%
of the digit positions in the first attempt by using accelerometer and orientation
sensors. On the other hand, TouchLogger is able to identify the digits with
71.5% accuracy on an Android device by using device orientation.

The paper goes on to note that once some of the discrepancies in methodology between both techniques are addressed, they both provide similar results.

@maryammjd

This comment has been minimized.

Copy link

commented Apr 11, 2017

I believe all of these high-level sensor data come from acc+gyr+mag. Any combination of this high and low sensor data would reveal sth about the user.

@alexshalamov

This comment has been minimized.

Copy link

commented Apr 11, 2017

@maryammjd Thanks.
@tobie Thanks for the hint, I will add reference to Taplogger paper and update risk values in the table.

@tobie

This comment has been minimized.

Copy link
Member

commented Apr 11, 2017

@alexshalamov it would be kind of cool to find an online reference to it. If by chance you came across something, do please share here. :)

@tobie

This comment has been minimized.

Copy link
Member

commented Apr 11, 2017

Oh, I had read this!

Need to read it again. :)

@alexshalamov

This comment has been minimized.

@tobie

This comment has been minimized.

Copy link
Member

commented Apr 11, 2017

I did not. o_O

This is one of the reasons this whole permission thing is hard. :(

@saabi

This comment has been minimized.

Copy link

commented Feb 14, 2018

Hi everybody,

I've been reading through the thread and it strikes me that basic usage (approximately what we already get through the old sensor API) should not need special permissions.

Reading sensor data on the background should definitely require permissions explaining about eavesdropping and that one should understand why permission should be granted.

High frequency sequencing is the gray area. So it should ask for permissions with a may-be-dangerous warning, or perhaps just the warning.

High frequency and background sensing should be different permissions.

EDIT: I should add that I'm actually referring to motion and gyro sensors only. I haven't payed attention to the other sensors.

EDIT2: Also, so far, at least the Android store permission system isn't warning users about these issues. That's however something that should be fixed elsewhere. Browser implementors should secure their side indistinctly.

@adonikian

This comment has been minimized.

Copy link

commented Feb 16, 2018

Hi everyone, we actually just release an in-browser demo on our website which uses the existing sensor API for tracking, check it out at: https://www.navisens.com

There are still many limitations compared to native sampling but we've tried to work around these as much as possible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.