Handle custom JSON responses in intents

[danjenkins]

Based on custom JSON responses in intents generically

[JorgMuskens]

I already use this in my code and solved it this way with the event mqtt topic.

In dialgoflow i have the a custompayload

{
  "CustomPayloadCommand": "ContinueInDialplan"
}

In index.js i added the ChannelId to be include in json to be send to ari bridge

dialogflowConnector.on('message', async (data) => {
        log.info(`Got a message sending to ${mqttTopicPrefix}/events`);
        data["channelId"] = payload.channelId; //add ChannelID for handling corrent event for specific channel in ari-bridge
        await mqttClient.publish(`${mqttTopicPrefix}/events`, JSON.stringify(data));
    });

Then in the index.js on the ari-bridge i added a subscribe on the mqtttopic in the main() function

if (config.get('mqtt.url')) {
            log.info('trying to connect to mqtt');
            mqttClient = await mqtt.connectAsync(config.get('mqtt.url'))
            log.info('connected to mqtt');
        }

        await mqttClient.subscribe(`${mqttTopicPrefix}/events`);
        log.info('Subscribed to events topic');

Then in the client.on('StasisStart'.....

if (mqttClient) {
                bridge.on('newStream', async (data) => {
                    await mqttClient.publish(`${mqttTopicPrefix}/newStream`, JSON.stringify({
                        roomName: data.roomName,
                        port: data.port,
                        callerName: data.callerName,
                        channelId: data.channelId
                    }));

                });

                bridge.on('streamEnded', async (data) => {
                    await mqttClient.publish(`${mqttTopicPrefix}/streamEnded`, JSON.stringify({
                        name: data.roomName,
                        port: data.port,
                        callerName: data.callerName,
                        channelId: data.channelId
                    }));
                });
                
                mqttClient.on('message', (topic, message) => {
                    let payload = JSON.parse(message.toString());
                    
                    switch(topic) {
                        case `${mqttTopicPrefix}/events`:
                            // Check of we have the correct event
                            if(payload.transcript == null && payload.intent.fulfillmentMessages[1] != undefined) {
                                switch(payload.intent.fulfillmentMessages[1].payload.fields.CustomPayloadCommand.stringValue) {
                                    case `ContinueInDialplan`:
                                        console.log(payload.channelId, 'Received ContinueInDialplan');
                                        break;
                                    default:
                                        break;
                                }
                            }
                            break;
                        default:
                            break;
                    }
                });
            }

Most likely not the best way to do it but it handles the job.
We will need to see if this clean be clean up some more
But this will give you an idea how to handle the custome payloads.
I actually use this to transfer the incomming line to another destionation with an actually person.

[danjenkins]

yeah, its a difficult thing because each ARI app is different, and this isn't designed to be a complete solution - just a base for people to work from..... but the mqtt event topic makes sense, I can add in the basics for people to expand upon

[JorgMuskens]

Yeah i know the ari client is still not complete for me aswell.
I am trying to update the channel to do a continueindialplan but still running into some problems
Eventually this Payload is a custom option and mostlikey for everyone different so a absic start point would be fine.

[JorgMuskens]

Another thing to keep in mind is that the event mqtt handler which detects the custom payload is received up front. This means that when i play audio back from dialogflow like "Please wait while we connect you to an operator" this audio is cut and terminated as the ari bridge is already processing the custom command.

I was playing around with an extra mqtt topic to handle the rtp stream if it was live or not. This was intended for the other issue but now make sense to use here. Unless someone has another idea?
Looking for a proper way to handle this

[danjenkins]

yeah @JorgMuskens it makes sense, or to only give back the result of the intent/custom response to the other app once the audio has finished playing back

[danjenkins]

But then again.... if the audio is long, you want to give say an agent being given these transcripts/intent results sooner than after an audio prompt has finished playing. so yeah, probably an audioEnded event

[danjenkins]

@JorgMuskens also, I just pushed a new branch on each repo up - both called new functionality - designed to deal with no audio responses better, sending the mqtt events back to the ARI bridge so it can do more decision making etc

[JorgMuskens]

@danjenkins i will test this today and let you know

[danjenkins]

Thanks! Theyre both merged in now

[JorgMuskens]

Something broke big time.
When i dial in the stasis app i see the welcome message coming in from dialogflow but i hear nothing played.
Then immediately after the error thrown. Below the last couple of lines

,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"outputAudioConfig":{"audioEncoding":"OUTPUT_AUDIO_ENCODING_LINEAR_16","sampleRateHertz":8000,"synthesizeSpeechConfig":{"effectsProfileId":[],"speakingRate":1,"pitch":5,"volumeGainDb":0,"voice":{"name":"","ssmlGender":"SSML_VOICE_GENDER_FEMALE"}}}},"msg":"got data from dialogflow","v":1}
{"level":30,"time":1587112131187,"pid":29695,"hostname":"tron","name":"Dialogflow-AudioServer","id":"1587112128.20","msg":"Got audio to play back from dialogflow","v":1}
events.js:292
      throw er; // Unhandled 'error' event
      ^

Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    at ClientHttp2Stream.Writable.write (_stream_writable.js:286:11)
    at /usr/src/asterisk-dialogflow-rtp-audioserver/node_modules/@grpc/grpc-js/build/src/call-stream.js:416:34
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
Emitted 'error' event on Http2CallStream instance at:
    at errorOrDestroy (internal/streams/destroy.js:128:12)
    at onwriteError (_stream_writable.js:406:3)
    at onwrite (_stream_writable.js:427:7)
    at processTicksAndRejections (internal/process/task_queues.js:83:21) {
  code: 'ERR_STREAM_WRITE_AFTER_END'
}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
ict@tron:/usr/src/asterisk-dialogflow-rtp-audioserver$

[JorgMuskens]

It looks like there is no open connection between ari and rtp anymore. Is it still working on your end?

[danjenkins]

yeah I tested it all with a couple of different scenarios.....

I guess I'll go re-do that testing :S

[danjenkins]

Crap, I can reproduce :( looking into it now :)