This is a checker tool to validate a result phrase against each query sent to the Dialogflow.
To send a query to the Dialogflow, this uses the Dialogflow Query API.
You can get this tool from the following page:
https://github.com/yoichiro/dialogflow-query-checker/releases/latest
Download the archive file named dialogflow-query-checker-[version].zip
and extract the file to some directory where you want to install. The archive file has some executable files for each OS and Architecture.
You can copy an executable file to a directory included into your PATH
environment variable.
To check your intents, actions and fulfillment, you need to create a configuration file. The format of the file is YAML. The structure of the file is:
clientAccessToken: <CLIENT_ACCESS_TOKEN>
defaultLanguage: <DEFAULT_LANGUAGE>
defaultLocale: <DEFAULT_LOCALE>
dateMacroFormat: <DATE_MACRO_FORMAT>
defaultServiceAccessToken: <DEFAULT_SERVICE_ACCESS_TOKEN>
defaultScoreThreshold: <DEFAULT_SCORE_THRESHOLD>
tests:
-
condition:
contexts:
- <INPUT_CONTEXT>
language: <LANGUAGE>
locale: <LOCALE>
query: <QUERY_STRING>
eventName: <EVENT_NAME>
sessionId: <SESSION_ID>
serviceAccessToken: <SERVICE_ACCESS_TOKEN>
expect:
action: <ACTION_ID>
intentName: <INTENT_ID>
parameters:
<PARAMETER_NAME>: <PARAMETER_VALUE>
contexts:
- <OUTPUT_CONTEXT>
speeches:
- <SPEECH_REGULAR_EXPRESSION>
scoreThreshold: <SCORE_THRESHOLD>
CLIENT_ACCESS_TOKEN
- The client access token issued by the Dialogflow. You can get the token from the project configuration page of the your Dialogflow project. You can omit this value. Instead, you need to specify this value with aDIALOGFLOW_CLIENT_ACCESS_TOKEN
environment variable.DEFAULT_LANGUAGE
- This language value is used, if the language value in each test definition is not specified.DEFAULT_LOCALE
- This locale value is used, if the locale value in each test definition is not specified.DATE_MACRO_FORMAT
- You can use a date macro forquery
andspeeches
like${date.today}
. This is a format used when evaluating the macro. This format is based on Go Programming Language Date and Time format. For example, you can specify likeJan 1, 2006
.DEFAULT_SERVICE_ACCESS_TOKEN
- The access token issued by your service. You need to specify this access token to use Account Linking feature. If specified, this will be applied for all test-cases basically. Also, you can omit this and specify the access token with aDIALOGFLOW_SERVICE_ACCESS_TOKEN
environment variable.DEFAULT_SCORE_THRESHOLD
- This score value is a minimum value which all test-cases should exceed.- tests - This is an array which has each test case.
- condition - This defines the condition of the query represented by contexts and a query.
INPUT_CONTEXT
- The context ID when the query sends. You can specify multiple contexts, and also can omit.LANGUAGE
- The query language. When some concrete string is specified, the value is applied. When the "inherit" is specified or omitted, the previous value is applied (if any concrete value was not used at the time, the defaultLanguage value will be used).LOCALE
- The user's locale. When some concrete string is specified, the value is applied. When the "inherit" is specified or omitted, the previous value is applied (if any concrete value was not used at the time, the defaultLocale value will be used).QUERY_STRING
- The query string. This means "User says" in Dialogflow. This is required when the event is omitted.EVENT_NAME
- The event name string. For example, "GOOGLE_ASSISTANT_WELCOME". This is required when the query is omitted.SESSION_ID
- When some concrete string is specified, the value is applied as a Session ID. When the "inherit" is specified or omitted, the previous value is applied (if any concrete value was not used at the time, this tool will generate a random Session ID value). And, when the "new" is specified, the re-generated new value is applied.SERVICE_ACCESS_TOKEN
- The access token issued by your service. You need to specify this access token to use Account Linking feature. If specified, theDEFAULT_SERVICE_ACCESS_TOKEN
value is overridden.
- expect - This defines a expected result which should be returned from the Dialogflow.
ACTION_ID
- The action ID determined by an intent.INTENT_ID
- The intent ID guessed by the query.- parameters - This defines parameters which were parsed from the query by the Dialogflow.
PARAMETER_NAME
- The parameter's name.PARAMETER_VALUE
- The parameter's value retrieved from the query phrase. An object can also be specified (ex.$sys.unit-currency
consists of two properties:amount
andcurrency
).
OUTPUT_CONTEXT
- The context ID determined by the intent or the fulfillment which should be contained. You can specify multiple contexts, and also can omit.SPEECH_REGULAR_EXPRESSION
- The regular expression to validate the response from the Dialogflow. When you specify multiple regular expressions, the test is passed if matched at least one expression. That is, the condition is OR.SCORE_THRESHOLD
- This score value is a minimum value which this test-case should exceed. If the defaultScoreThreshold is specified, you can override the value using this.
- condition - This defines the condition of the query represented by contexts and a query.
The sample is like the following:
clientAccessToken: ...
defaultLanguage: en
defaultLocale: en-US
tests:
-
condition:
event: GOOGLE_ASSISTANT_WELCOME
expect:
action: "input.welcome"
intentName: "input.welcome"
speeches:
- "^Hello\\.$"
-
condition:
contexts:
- "input_condition"
query: "How many times is a Google I/O in this year\\?"
expect:
action: "event_info"
intentName: "input.condition"
parameters:
event: "Google I/O"
when: "2018"
contexts:
- "answered"
speeches:
- "^The event is the (1st)|(2nd)|(3rd)|([0-9]+th)\\.$"
...
In the QUERY_STRING
, the PARAMETER_VALUE
and the SPEECH_REGULAR_EXPRESSION
, you can use macros. In the latest version, the following macros are supported:
${date.today}
- This is replaced to today's date string.${date.tomorrow}
- This is replaced to tomorrow's date string.
When your assistant app is applying Account Linking, your app does not work without an access token. That is, your app works only when the access token issued by your service is passed from Account Linking feature on Actions on Google. If your app is called without any access token, an error should occur in your fulfillment.
This tool supports specifying an access token specified by one of following definitions:
DIALOGFLOW_SERVICE_ACCESS_TOKEN
environment variable.defaultServiceAccessToken
item in your YAML configuration file.serviceAccessToken
item for each test-case in your YAML configuration file.
If the access token specified, the access token is passed to the Dialogflow agent like the following:
{
...
"originalRequest": {
...
"data": {
...
"user": {
...
"accessToken": <SERVICE_ACCESS_TOKEN>
}
}
}
}
After writing the configuration file, you can execute this tool to validate each query and the response based on the configuration file. To use this tool, execute the following command:
$ dialogflow-query-checker run <CONFIGURATION_FILE_PATH>
You see like the following output:
........F.....F........F.......
[input_condition How many times is a Google I/O in this year?]
Failure: action is not same.
Expected: event_info
Actual: event_information
...
3 tests failed.
If all tests passed, exit status code is 0
. Otherwise, the code is 1
.
You can output test results with a JUnit XML format using --output
or -o
option as the following:
$ dialogflow-query-checker run --output <OUTPUT_FILE_PATH> <CONFIGURATION_FILE_PATH>
By using this option, you can integrate this tool for some services like CircleCI, TravisCI and so on. For example, when you use the CircleCI, you can use this tool like the following:
- Put the
dialogflow-query-checker
executable binary file into abin
directory of your project. - Create your configuration YAML file and put it into your project.
- Create
.circleci/config.yml
file like the following:
version: 2
jobs:
test:
docker:
- image: ubuntu
steps:
- run:
name: Update apt-get
command: apt-get update
- run:
name: Install ca-certificates
command: apt-get install -y ca-certificates
- checkout
- run:
name: dialogflow-query-checker
command: |
mkdir -p ~/repo/dialogflow-query-checker
bin/dialogflow-query-checker run -o ~/repo/dialogflow-query-checker/test_result.xml your_config_fileyml
- store_test_results:
path: ~/repo
- store_artifacts:
path: ~/repo
workflows:
version: 2
commit:
jobs:
- test
nightly:
triggers:
- schedule:
cron: "0 0 * * *"
filters:
branches:
only:
- master
jobs:
- test
You will see the integrated test summary information on your CircleCI project after steps above.
You can set the count of retrying using --retry
or -r
option as the following:
$ dialogflow-query-checker run --retry <RETRYING_COUNT> <CONFIGURATION_FILE_PATH>
When the Dialogflow returns a result status code which is not 200 and you set a positive value as the retry count, this tool will resend the same request to the Dialogflow.
You can set the interval per test-case using --sleep
or -s
option as the following:
$ dialogflow-query-checker run --sleep 500 <CONFIGURATION_FILE_PATH>
The unit of the interval value is millisecond. For example, if you specify --sleep 500
, this tool sleeps during 500ms per test-case.
You can build this tool by the following steps:
- Install Go language environment.
go get -u github.com/yoichiro/dialogflow-query-checker
go get github.com/mitchellh/gox
- Change the version number written in the
verion.go
file. - Run
./cross_build.sh