not emitting error event

[maheshrevanwarPSL]

I am using .recognizeMicrophone function
stream = WatsonSpeech.SpeechToText.recognizeMicrophone({ token: token, continuous: false, outputElement: '<css selector>', content_type: 'audio/wav' });

I have added a "Push to Talk" button [When user click on it this button it will start recording till the button get released] but when I release this button before establishing WebSocket connection it gives bellow error

• recognize-stream.js:345 Uncaught InvalidStateError: Failed to execute 'send' on 'WebSocket': Still in CONNECTING state.

I have also added the stream.on('error') function for handling this error but it is not emitting that event.
stream.on('error', function(err) { console.log("error is produced when calling STT"); console.log(err); });

Same for bellow 2 errors:

• recognize-stream.js:260 Uncaught Error: Stream was 0 bytes but needs to be at least 100 bytes.
• recognize-stream.js:260 Uncaught Error: Session timed out due to inactivity after 30 seconds.

Please advice me how to handle this error.
And correct me if my approach is wrong.

Regards,
Mahesh Revanwar

[nfriedly]

Hi @maheshrevanwarPSL

I think that the error not being emitted is the same as #10, which I have an idea for how to resolve but haven't yet been able to implement it.

However, the cause of the error is this line: https://github.com/watson-developer-cloud/speech-javascript-sdk/blob/master/speech-to-text/recognize-stream.js#L407 - it should really just check that the state is OPEN rather then checking that it's not in any other state.

I have a ton on my plate right now, but if you're interested, I'd love a PR with:

1. A test case that replicates the issue - you can add it to end-to-end-spec.js or create a new recognize-stream-spec.js - the later might be easier due to recognizeMicrophone does not propigate errors from the RecognizeStream to the FormatStream #10. (The command npm test spins up a local websocket server so you don't need to worry about hitting the actual watson service, you can look at the end-to-end-spec.jsfor an example of how to connect to the server.
2. The actual fix, which should cause the above test to pass.

The test case is probably more work than the actual fix, so I'd settle for just the fix. But I'd really prefer it with a test.

One other thing to mention, that error looks like it would come from line 348 in the current codebase, so you may be on an older version of the library. But I don't think that updating to the latest would make the error go away.

[maheshrevanwarPSL]

Hi @nfriedly,

Thanks for your valuable inputs.

As you have suggested the code change, I have done that change locally, but this change only take care of first error.

Could you please guide me for other 2 errors that I have mentioned in my previous comment.

Regards,
Mahesh Revanwar

[nfriedly]

Hum, I was thinking that change would resolve all of the errors. I'll try and look at this more once I get a chance, but for now, I think you can just ignore those errors.

[kushan-chinthaka]

Hi @nfriedly,

I have been working on this issue for few days now. I would prefer, if you allow me to create a branch for committing only the fix, you mentioned. Furthermore, I was struggling to create a test case for the first issue, but it is still challenging to make the socket in CONNECTING state while the test is running and if you have any idea on that let me know.

The mentioned fix does not have any impact it seems. All the karma tests are passing as shown below in my local test run results.

22 09 2016 11:09:31.186:INFO [http.server]: Starting
22 09 2016 11:09:31.723:INFO [http.server]: setting up token & api servers for offline test
22 09 2016 11:09:31.723:INFO [http.server]: Done
22 09 2016 11:09:34.113:INFO [framework.browserify]: bundle built
22 09 2016 11:09:34.113:INFO [karma]: Karma v0.13.22 server started at http://localhost:9876/
22 09 2016 11:09:34.144:INFO [launcher]: Starting browser Chrome
22 09 2016 11:09:34.144:INFO [launcher]: Starting browser Firefox
22 09 2016 11:09:36.567:INFO [Firefox 48.0.0 (Windows 8.1 0.0.0)]: Connected on socket /#SW_n0xyZ8mH0rVXIAAAA with id 98078784
Firefox 48.0.0 (Windows 8.1 0.0.0): Executed 0 of 19 SUCCESS (0 secs / 0 secs)
Firefox 48.0.0 (Windows 8.1 0.0.0): Executed 1 of 19 SUCCESS (0 secs / 8.514 secs)
Firefox 48.0.0 (Windows 8.1 0.0.0): Executed 2 of 19 SUCCESS (0 secs / 8.74 secs)
Firefox 48.0.0 (Windows 8.1 0.0.0): Executed 19 of 19 SUCCESS (8.551 secs / 9.008 secs)
Firefox 48.0.0 (Windows 8.1 0.0.0): Executed 19 of 19 SUCCESS (8.551 secs / 9.008 secs)
Firefox 48.0.0 (Windows 8.1 0.0.0): Executed 19 of 19 SUCCESS (8.551 secs / 9.008 secs)
Firefox 48.0.0 (Windows 8.1 0.0.0): Executed 19 of 19 SUCCESS (8.551 secs / 9.008 secs)
Firefox 48.0.0 (Windows 8.1 0.0.0): Executed 19 of 19 SUCCESS (8.551 secs / 9.008 secs)
Chrome 53.0.2785 (Windows 8.1 0.0.0): Executed 19 of 19 SUCCESS (8.207 secs / 8.171 secs)
TOTAL: 38 SUCCESS

Regards,
Kushan

[kushan-chinthaka]

Hi @nfriedly,
I invested more time on the second part of the issue as well. I did following tests to capture the error event emitted by the socket from recognize-microphone.js and recognize-stream.js.

Test1. Add an event listener to recognize-microphone.js

function handleError(event) {
    console.error('Found the error ' + event);
   }
   recognizeStream.on('error', handleError);

Test2. Add an event listener recognize-stream.js

Test3. Emit a new error event from recognize-stream.js and capture it from recognize-microphone.js.

In recognize-stream.js

   function handleError(event) {
    console.error('Found the error ' + event);
    recognizeStream.emit('error1')
   }
   recognizeStream.on('error', handleError);

In recognize-microphone.js.

   function handleError1(event) {
    console.error('Found the error1 ' + event);

   }
   recognizeMicrophone.on('error1',handleError);

As per above tests I realized that there seems to be no issue with emitting or capturing error events between these two js files, as well as the error gets propagated successfully from recognize-stream.js to recognize-microphone.js.

Only concern I have now is that, why the sample client code(microphone-streaming.html) can not capture these error events.

Although I can not still confirm the fact, I suspect that the issue may be due to the Promises approach of handling async code from the client code. The reason I suspect is that, can we use same event listener mechanism as is in Promises code for capturing events.

fetch('/api/speech-to-text/token')
  .then(function(response) {
      return response.text();
  }).then(function (token) {
    var stream = WatsonSpeech.SpeechToText.recognizeMicrophone({
        token: token,
        outputElement: '#output' // CSS selector or DOM Element
    });

    **stream.on('error', function(err) {
        console.log(err);
    });**

However, I could not figure out why this is still failing. Do you think the approach I am taking would help fix this issue?

Regards,
Kushan

[kushan-chinthaka]

Hi @nfriedly,

Further investigating the issue, I was able to come up with a fix for this as I explained below.

Issue: Error event is not propagated to the front facing HTML file from socket timeouts and other error scenario. #17

1.  recognize-stream.js:260 Uncaught Error: Stream was 0 bytes but needs to be at least 100 bytes.
2.  recognize-stream.js:260 Uncaught Error: Session timed out due to inactivity after 30 seconds

Root-cause: EventEmitter reference is assigned to a different reference because the text stream is piped for text formatting in the middle of the recognizeMicrophone method.

Approach for the fix:

• Tests were done initially to make sure the error event propagation happens successfully from recognize-stream.js to recognize-microphone.js
• Suspected the async error handling of Promises technology may result in the issue. In line with that, a sample event based node-browserify client application was developed to test the following set of scenario,

1. Using javascript based client
2. Using promises based client
3. Using promises with jquery client
4. Using jquery client with promises catch block.

However, all the mentioned scenario got passed successfully in the sample app. Hence, it was possible to conclude the fact that the main issue should reside inside the recognize-microphone.js and nowhere else.

• In abstract terms, recognizeMicrophone() function of the recognize-microphone.js does following key tasks.

1. Establish the connection to the user microphone using microphone-stream node module, receive voice stream, convert to text stream.
2. Extract the textual representation with RecognizeStream
3. Do text formatting using FormatStream
4. Return the formatted RecognizeStream to the client app.

After step 2, there are two memory references to the same stream object in the flow as below.

var recognizeStream = new RecognizeStream(rsOpts);

        var stream = recognizeStream;

At step 3, the stream is piped to FormatStream for formatting needs such as capitalizing first letter as below.

stream = stream.pipe(new FormatStream(options));

Then the stream becomes a ‘FormatStream’ since pipe() makes the assignment to destination stream type. The stream refers new memory reference at this point due to new keyword and it brakes the reference from original RecognizeStream. Moreover, the stream is also the EventEmitter which propagates events to the caller, but it disconnects its reference as well at this point. Therefore, it is not possible to bubble up any errors henceforth.

Solution:
As a solution to this issue, we should manually propagate any errors from original RecognizeStream to new FormatStream as follows in the recognize-microphone.js

          function handleError(event) {
            // Emit a new error to new FormatStream
            stream.emit('error', new Error('new error'));
          }

          // Capture error from original stream
          recognizeStream.on('error', handleError);

I presume that this fix would result in the #10 issue to be fixed as well.

Please provide your feedback about this fix.

Regards,
Kushan

[germanattanasio]

This is amazing @kushan-chinthaka! Thanks for helping with this issue

[nfriedly]

Yes, I believe you are correct. Can you for, the repo and send a pull request with the change?

[kushan-chinthaka]

PR created.
#18

[nfriedly]

Cool, I'll take a look at it today. Thanks for working on this!