AudioConstraints not well defined on chrome android

[BenjaminPoilve]

Hi,
I had some troubles transmitting data from my browser to my android phone, and it turned out it was a permission problem. The phone audio pre-processing wasn't turned-off which resulted in an effective cutoff frequency around 7500hz. Transmitting at 12000hz, it ruined my transmission!
My fix was to implement the adapter.js, and use only echoCancellation as constraints.
It now works!
My code is not very clean, I'll make it better then make a pull request! In the meantime, I wanted to keep anybody who runs into the problem up to date!
Incredible work by the way

[brian-armstrong]

Hi Benjamin,

I don't have an Android device to test on, so I'd love to hear more about the issues you've had with quiet-js on that platform. I have tried to get mobile Chrome to use the same constraints as used on desktop Chrome, which I felt would be a reasonable approach.

Are you saying that setting other constraints to disabled caused the pre-processing to activate? Also, are you using Chrome or AOSP?

Thanks

[BenjaminPoilve]

Well, I have a hard time debugging this issue, but setting the constraints as mandatory (opposed to optionnal) resulted in a overconstrained pipeline on chrome android, for every parameter. My understanding is that chrome android don't use the google legacy syntax for constraints.
I am using the standard chrome browser that came with my distribution.
Do you know any good tool for debuggin this? The getUserMedia spec is not really folllowed so it is hard to know what/if I am doing something wrong.
After further testing, it seems like I got my hopes a little bit up.
I can now receive messages ( which I couldn't before) but my upper frequency still is around 8Khz.

[BenjaminPoilve]

This is not a pull request (sorry) but for reference my code looks like that:

 function gUMConstraints() {
            return {
                "audio":{
                    "mandatory":{
                       echoCancellation: false
                   }
                }
            };
    };
    function createAudioInput() {
        console.log("trying to create an input");

        audioInput = 0; // prevent others from trying to create
        navigator.mediaDevices.getUserMedia(
            gUMConstraints()
        ).then(
          successCallback,
          errorCallback
        );
    };
    function successCallback(stream) {
        console.log("ok for stream");
        audioInput = audioCtx.createMediaStreamSource(stream);
        console.log(audioInput.getConstraints);
        // stash a very permanent reference so this isn't collected
        window.quiet_receiver_anti_gc = audioInput;
        audioInputReady();
    };

    function errorCallback(error) {
        console.log(error);
        audioInputFailed(error);
    }; 

With adapter.js

[BenjaminPoilve]

Still trying to make this work reliably!
The gmsk scheme seems to be very robust when the distance is small, but performances fall after a few centimeters. My guess being that phase shift is not very good for sound (echo and things..)
I would like to try to implement a FSK scheme which could be more resilient for audio..
Would that be a good idea? Do you have any pointers?

[brian-armstrong]

For testing purposes, I'd recommend trying your receiver in a variety of physical orientations with regard to the transmitter. The orientation can make a difference for sound transmission.

Also, if you haven't, try different volume levels on the transmitter. Often, a lower volume level can be better as it generates less noise and potentially avoids clipping.

As for FSK, GMSK is already a particular kind of FSK.

Do you know if your phone has two mics? From what I've read, many Androids have more than one microphone and do some kind of non-voice cancellation by subtracting one mic from another. I'm not sure if that'd be relevant here.

[BenjaminPoilve]

It might be possible but I didn't find any documentation of the feature..
My finding so far is that indeed transmission is very "orientation dependant". Probably some multiple path effect if there is no straight line between emitter and receiver. My basic understanding of physics tells me that using a lower frequency should fix the problem but when I do, I don't even get any results..
I think I don't understand enough the basics of modulation to be really efficient at this debugging.
I'll try to get up to speed by implementing a simple FSK modulator/demodulator! I found this to be very interesting and probably a good way to start!

[brian-armstrong]

I'm not sure how well this will work on Android, but have you tried the Lab? https://quiet.github.io/quiet-js/lab.html

If nothing else, it should give you a little more visibility into what frequencies work in your browser. It uses the same device for transmitting as receiving so you won't really be able to test various distances, but that might at least give you more visibility into your particular setup

[brian-armstrong]

As for implementing a FSK, I think the GMSK mode that comes with quiet.js should be pretty good in that regard already. Try increasing the interpolation factor samples_per_symbol by a few to see if that helps -- that will slow down the transmission, giving more discrete sound samples per bit of data.

[brian-armstrong]

As for why lower frequencies don't work, in my testing I have found that ambient room noise tends to occupy the space under ~1.5khz or so, which can really hamper your ability to receive there.

I'm sorry to hear that your experience with this library has not been very good so far. If you'd prefer to troubleshoot over email, I believe my address should be in the commits. I'm also happy to keep talking about it in this issue, whichever is easiest for you.

[BenjaminPoilve]

Well I have to answer now because I really wouldn't want you to believe this lib is anything short of amazing! Way better that anything I have seen (and I have been tinkering with the subject for at least 6 months).
The data-rate is incredible and very stable given the correct conditions! I didn't even know about the lab which is a great tool for debugging! I just think that a lot of my problem come from a lack of understanding the theory behind all this..
I'll be happy to keep discussing here, as long of you don't think this is a misuse of issue ticket (I am not good at github etiquette). But if this can helps some other people I think it is for the best.
I should probably fill you in on my goals. I am trying to setup a serverless filetransfert platform, using sound as a way to distribute hashes to people close to you. This is my after-work project and I think this could be pretty cool. I just need a not necessarily fast but reliable way to communicate 40 hex characters.

By the way, it is very kind of you to offer all this pointers, I am very grateful for that!

[brian-armstrong]

Thank you for the kind words. I really appreciate that. I feel like I'm pretty happy with how the library works, but it's also very dependent on individual hardware/setup, and I am limited in terms of what I can test, so I worry that it doesn't work well on setups I haven't tested.

I'm happy to discuss this here. As you pointed out, it can probably help others. I think you're probably one of the first people to test this library's receiving functionality on Android so your input is really helpful.

And yeah, I definitely agree that this library should work well your setup. Since your payloads are on the shorter side, I think increasing the interpolation factor could be helpful. It'll be a little slower but it should be more resilient. It looks like the audible profiles default to 6-10 samples per symbol, but you could go as high as 20. You wouldn't want to go higher than that since the modem will start to get too slow. Also, if you're using the clamp-frame feature, it's worth noting that increasing this value will decrease the length of each frame.

[BenjaminPoilve]

Hi,
The lab is really great as a tool! It really helped me to get a good profile going!
Right now my best profile is this one:

{ "mod_scheme": "gmsk",
      "checksum_scheme": "crc32",
      "inner_fec_scheme": "secded7264",
      "outer_fec_scheme": "rs8",
      "frame_length": 25,
      "modulation": {
        "center_frequency": 6800,
        "gain": 0.15
      },
      "interpolation": {
        "shape": "farcsech",
        "samples_per_symbol": 20,
        "symbol_delay": 1,
        "excess_bandwidth": 0.35
      },
      "encoder_filters": {
        "dc_filter_alpha": 0.01
      },
      "resampler": {
        "delay": 13,
        "bandwidth": 0.45,
        "attenuation": 60,
        "filter_bank_size": 64
      } 

Trouble is, it seems it is not supported by the emscriptem in the exemple folder (it get stuck on first packet and loops forever). When I use the emscriptem in the Lab demo, I get an error : quiet-emscripten.js:1631 Uncaught Assertion failed: , at: ,71, at Error

It seems like there might be some discrepancies between the files used in the lab and the ones in the github! I'll try to find a fix!

[brian-armstrong]

Hi Benjamin,

Well, it's clear I need better error reporting, if nothing else :) You're getting snagged by one of the rough edges on this library, unfortunately.

I can see why quiet.js doesn't like this profile. I will give some background on what's going on to hopefully help shine some light on this.

I was able to reproduce the behavior you see in the lab. What's important to note here is that if we switch Transmitter Frame Length Behavior from Allow Overlap of Multiple Blocks to Clamp to Single Sample Block, the transmitter quits. The reason for this is that the lab has a safeguard that will keep it from running if our frame length falls to less than 4 bytes long. Unfortunately, the actual library isn't quite as good at detecting this case yet.

Now, you might be wondering what this option does and why it matters. When quiet.js interacts with the sound hardware, it sends and receives samples in large batches. In this case, those batches are 16,384 samples long, about 0.37 seconds in length if we use a sample rate of 44,100 Hz. Once we hand a batch of samples off, the sound card driver ensures they'll play back seamlessly, so we can be sure that there will be no clicks or pops inside a single batch.

Now, the next thing to consider is what happens on the seam between two consecutive batches of samples. Let's imagine we're playing a simple sine wave, and imagine that this wave is at its peak at the end of the first batch. The very next sample in the next batch should be the sample that follows. If our encoder is running fast enough, that's what we'll get, the sine wave will continue playing without any clicks or pops.

But if the encoder falls behind, which can happen especially when the garbage collector runs in JS, then the sound card will have used up all the samples we've given it. In that case, because it is running in real time, it has to do something at this seam. And this is where we get clicks and pops. If this were a modem tone being played, it will almost definitely corrupt the rest of the entire frame.

So now we come back to this option. What this option does is that it forces quiet.js to constrain data frames to lie only within a single batch of samples. When the frame finishes, rather than start the next one, quiet.js will just start playing silence rather than start a new frame. This ensures that if there is a seam, the only thing that it can be filled with is silence. This option works great if our frame can fit inside a single batch, but if the frame is so long that it must overlap two batches, then it's clear that the option is incompatible with the profile.

The lab lets you choose which style you prefer. On a desktop, it's pretty rare that the encoder can't keep up, so the overlap works fine. On something with less memory and less cpu, the clamp option is a lot more useful. So, if you copy the example code, you'll have the clamping turned on -- quiet.js tries to be conservative here. And, as I'm typing this, I realize the version that's in the master branch is out of date. I added the option to toggle this in the version that's in the gh-pages branch, but I didn't push to master yet because I hadn't finished double-checking all the new API. You can see how this gets toggled here https://github.com/quiet/quiet-js/blob/gh-pages/javascripts/quiet.js#L255

There's an easy fix that doesn't require changing the clamp behavior for your profile. Simply change the Outer FEC to None. Although Reed-Solomon error correction is very powerful, it has a large overhead (this flavor of R-S will always add an additional 32 bytes, no matter how small the frame -- in your case, that's longer than the frame itself). Once I do that, I can confirm that the rest of the settings work great with clamping turned on in the lab.

I hope this helps, and I see that I need to do lots of work on the documentation :)

[BenjaminPoilve]

What an expensive response! It indeeds makes it much clearer! This is indeed the reason, trying out profiles that work with the clamped option work with the example code.
It really seems to me that there is a lot of possible application for this tech, so I'll be eager to help! I am not a great coder but if you think you can offload some tasks, I'll be happy to give it a try in my spare time!
Just by curiosity, did you had any specific use case in mind when developing this or did you do it just for the fun of it? What I can say at least is that it could be a drop-in replacement for many existing tech (QRcode at least, and for pairing process like SDP negotiation in WebRTC )!
I have been trying some other modulation scheme and some might be more robust than GMSK for my use case. The differential phase shift keying seems like it can take a lot of background noise.

[brian-armstrong]

I'm glad to hear that that helped, and thank you again for the nice feedback. I can't take all of the credit here, liquid DSP really made this all very easy, it's a fantastic library.

I think the best thing you can do to help out is what you're already doing. As far as I know, not many people have tried to build something with quiet yet. It would help tremendously to be able to point at projects built with quiet to show that it does work. Also, your feedback as you work through the process will help me know what needs improvement.

As far as GMSK goes, I suspect it will often not be the most robust method, but computationally the receiver is the most lightweight out of all of them. GMSK is the only mode that does not invoke any FFTs. For native apps built with libquiet this probably doesn't really matter, but if you wanted to run a receiver on mobile on top of quiet.js, I think it could make a difference. I know that the emscripten/js overhead is somewhat significant, and I'm not sure whether most mobile devices have enough CPU to decode the other modes without hiccups.

Personally, I'd like to build a simple app to allow me to pass my copy/paste buffer back and forth between devices. I often find a URL on my phone that I want to look at on my laptop, and if I could just leave something running in the system tray that would listen via libquiet, it'd be very easy to send it links or little text snippets.

[brian-armstrong]

I'm closing this issue for inactivity. If any new developments occur or if this problem persists, feel free to open a new issue.