How to write agent plugin test case

Overview

This test cases are based on real instrumentation, and verify the segments which are generated by the plugin and agent core. By that, the test framework will call the HTTP endpoint provided by test case, and use mock-collector¹ to simulate the backend, which could receive and log all generated segments. Then the test framework will check the trace segments that whether they are same as expected.

Test case repository structure

Test case repository includes all the test cases. Each case is in an independent folder, and also an independent maven project. The maven project is used to build a docker image, which is required to provide a HTTP endpoint as an entry.

Each test case should look like this.

[plugin_testcase]
  |__ [config]
  | |__ docker-compse.yml
  | |__ expectedData.yaml
  |__ [src]
  | |__ [main]
  | |    ...
  | |__ [resources]
  | |    ...
  |__ pom.xml
  |__ testcase.yml

[] = directory

The usage of files in test case

The following setting files are required.

File Name	Usage
docker-compose.xml	Define the docker compose env used in test
expectedData.yaml	Define the check rules about the expected trace segment.
testcase.yml	Define the metadata of test, such as test case name, version

The workflow to write test case

Code the test case
Build and package test case. Test the image and make sure the agent and your plugin work.
Set your expected segment data check rules.
Build your test case metadata file.
Run the test case.

Example

Here we are using HttpClient plugin as an example.

HttpClient plugin is required to test two important points.

Span generated by HttpClient plugin.
The context propagation across HttpClient plugin works.

So the test including two servlets Case Servlet and ContextPropagateServlet. The Case Servlet use HttpClient componenent to a distributed call to ContextPropagateServlet.

+-------------+         +------------------+            +-------------------------+
|   Browser   |         |  Case Servlet    |            | ContextPropagateServlet |
|             |         |                  |            |                         |
+-----|-------+         +---------|--------+            +------------|------------+
      |                           |                                  |
      |                           |                                  |
      |       WebHttp            +-+                                 |
      +------------------------> |-|         HttpClient             +-+
      |                          |--------------------------------> |-|
      |                          |-|                                |-|
      |                          |-|                                |-|
      |                          |-| <--------------------------------|
      |                          |-|                                +-+
      | <--------------------------|                                 |
      |                          +-+                                 |
      |                           |                                  |
      |                           |                                  |
      |                           |                                  |
      |                           |                                  |
      +                           +                                  +

Code your case

Best practice in pom.xml:

The target component version should be set by maven properties
The image version should be set by maven properties
Keep image version as same as target component version

Take this as the example

Build the case docker image

Add maven docker plugin.

Pay attention the following fields.

Field	Comment
imageName	Image name, recommend style `skywalking/xxx-scenario`, Such as httpClient case should be named `skywalking/httpclient-scenario`
dockerDirectory	The folder used to build image, recommend `${project.basedir}/docker`
imageTags	The image version, use maven properties, and as same as target component version.

Prepare Dockerfile. This is totally based on your case scenario.

HttpClient test case DockerFile is here

Prepare docker-compose.xml. Create Docker-compose.xml in config folder
Run maven docker plugin

mvn package docker:build

Test the case

Run docker-compose up in config folder.
Open the browser and access the HTTP endpoint opened in your case.

Active SkyWalking agent

Add VOLUME of agent in Dockerfile, example is here
Add -javaagent in your startup script

HttpClient test is hosted in Tomcat, so the -javaagent should be added in ${project.basedir}/docker/catalina.sh. Example is here

Add mock-collector image¹

Add mock-collector in your docker-compose
Set your test case use mock-collector as backend

Example is here

Adjust the port and image version in docker-compose.xml to variables.

Set version to {CASES_IMAGE_VERSION}
Set port to {SERVER_OUTPUT_PORT}

Add volumes for agent in docker-compose

Example is here

Run test after we finish the expected data rule settings.

Expected data file

HttpClient test case expected data file --- expectedData.yaml

Expected data

Expected data file is used to describe the agent data, cinluding register and segment data. In segment verify, include verifying span, segment number, etc.

Before we introduce the expected data file format, we should new the verify OP(s)

Verify OP(s) for number

描述符	描述
`nq`	not equal
`eq`	equal(default)
`ge`	greater and equal
`gt`	greater than

Verify OP(s) for String

描述符	描述
`not null`	not null
`null`	null or empty string
`eq`	equal(default)

Now, let's see the expected data format.

Register expected data format

registryItems:
  applications:
  - SERVICE_CODE: SERVICE_ID(int)
  ...
  instances:
  - APPLICATION_CODE: INSTANCE_COUNT(int)
  ...
  operationNames:
  - APPLICATION_CODE: [ SPAN_OPERATION(string), ... ]
  ...

Field	Comment
applications	Service Code and its register id, but since it can't be expected in many cases, just set not 0
instances	The number of service instace
operationNames	Expected OperationNames of Span, which are sent to register

Segments expected data format

segments:
-
  applicationCode: SERVICE_CODE(string)
  segmentSize: SEGMENT_SIZE(int)
  segments:
  - segmentId: SEGMENT_ID(string)
    spans:
        ....

Field	Comment
applicationCode	Service code of segment, the application code is the name used in 5.x.
segmentSize	The number of segment in this service
segmentId	segment ID.
spans	segment span list

Span expected data format

Attention: Expected data span list should be sorted by the finished time.

    operationName: OPERATION_NAME(string)
    operationId: SPAN_ID(int)
    parentSpanId: PARENT_SPAN_ID(int)
    spanId: SPAN_ID(int)
    startTime: START_TIME(int)
    endTime: END_TIME(int)
    isError: IS_ERROR(string: true, false)
    spanLayer: SPAN_LAYER(string: DB, RPC_FRAMEWORK, HTTP, MQ, CACHE)
    spanType: SPAN_TYPE(string: Exit, Entry, Local )
    componentName: COMPONENT_NAME(string)
    componentId: COMPONENT_ID(int)
    tags:
    - {key: TAG_KEY(string), value: TAG_VALUE(string)}
    ...
    logs:
    - {key: LOG_KEY(string), value: LOG_VALUE(string)}
    ...
    peer: PEER(string)
    peerId: PEER_ID(int)
    refs:
    - {
       parentSpanId: PARENT_SPAN_ID(int),
       parentTraceSegmentId: PARENT_TRACE_SEGMENT_ID(string),
       entryServiceName: ENTRY_SERVICE_NAME(string),
       networkAddress: NETWORK_ADDRESS(string),
       parentServiceName: PARENT_SERVICE_NAME(string),
       entryApplicationInstanceId: ENTRY_APPLICATION_INSTANCE_ID(int)
     }
   ...

Field	Comment
operationName	Span Operation Name
operationId	OperationName id, usually expected 0, because no request after register.
parentSpanId	Paren span id. Attention: First Span's parentSpanId is -1
spanId	Span Id. Attention: ID starts with 0.
startTime	Span start time, not 0 should be enough
endTime	Span end time, not 0 should be enough
isError	Whether error happens.
componentName	Match the Component define，In most case, it is null. ID is used for performance consideration.
componentId	Match the Component define，
tags	Span Tag. Attention: Order sensitive, keep as same as you do in codes
logs	Span log. Attention: Order sensitive, keep as same as you do in codes
SpanLayer	Span Layer, including DB, RPC_FRAMEWORK, HTTP, MQ, CACHE
SpanType	Span type, including Exit, Entry, Local
peer	Remote address in exit span, should be not null.
peerId	0 for now, use peer in every first request.

Span ref expected data format

Field	Comment
parentSpanId	Span ID of Parent segment.
parentTraceSegmentId	Parent segment id. Format ${SERVICE_CODE[SEGMENT_INDEX]}, `SEGMENT_INDEX`in the INDEX in expected data.
entryEndpoint	OperationName of entry name in Segment. For example in HttpClient case, entryEndpoint is `/httpclient-case/case/httpclient`
networkAddress	The address is used to call this service. For example, CaseServlet uses 127.0.0.1:8080 to call ContextPropagateServlet. Then the value of this field in ContextPropagateServlet should be 127.0.0.1:8080
parentEndpoint	OperationName of entry name in parent segment
entryServiceInstanceId	The instance id of entry service. Not 0 should be enough

The workflow of writing expected data file

RegistryItems

HttpClient test case is based on Tomcat, only one instance is running. Service id is not null. And there are two entry endpoints, /httpclient-case/case/httpclient and /httpclient-case/case/context-propagate

registryItems:
  applications:
  - {httpclient-case: nq 0}
  instances:
  - {httpclient-case: 1}
  operationNames:
  - httpclient-case: [/httpclient-case/case/httpclient,/httpclient-case/case/context-propagate]

segmentItems

In HttpClient case, there are two segment created, one is caused by requesting CaseServlet, named SegmentA, and the other is caused by requesting ContextPropagateServlet, named SegmentB.

segments:
  - applicationCode: httpclient-case
    segmentSize: 2

Tomcat is supported, so in SegmentA, there is 2 spans. One is Tomcat, the other is HttpClient.

    - segmentId: not null
      spans:
        -
          operationName: /httpclient-case/case/context-propagate
          operationId: eq 0
          parentSpanId: 0
          spanId: 1
          startTime: nq 0
          endTime: nq 0
          isError: false
          spanLayer: Http
          spanType: Exit
          componentName: null
          componentId: eq 2
          tags:
            - {key: url, value: 'http://127.0.0.1:8080/httpclient-case/case/context-propagate'}
            - {key: http.method, value: GET}
          logs: []
          peer: null
          peerId: eq 0
        -
          operationName: /httpclient-case/case/httpclient
          operationId: eq 0
          parentSpanId: -1
          spanId: 0
          startTime: nq 0
          endTime: nq 0
          spanLayer: Http
          isError: false
          spanType: Entry
          componentName: null
          componentId: 1
          tags:
            - {key: url, value: 'http://localhost:{SERVER_OUTPUT_PORT}/httpclient-case/case/httpclient'}
            - {key: http.method, value: GET}
          logs: []
          peer: null
          peerId: eq 0

SegmentB is created by a internal request, but through network, HttpClient to ContextPropagateServlet. So, in SegmentB, SegmentRef is created because of trace context continuation.

- segmentId: not null
  spans:
  -
   operationName: /httpclient-case/case/context-propagate
   operationId: eq 0
   parentSpanId: -1
   spanId: 0
   tags:
   - {key: url, value: 'http://127.0.0.1:8080/httpclient-case/case/context-propagate'}
   - {key: http.method, value: GET}
   logs: []
   startTime: nq 0
   endTime: nq 0
   spanLayer: Http
   isError: false
   spanType: Entry
   componentName: null
   componentId: 1
   peer: null
   peerId: eq 0
   refs:
   - {parentSpanId: 1, parentTraceSegmentId: "${httpclient-case[0]}", entryServiceName: "/httpclient-case/case/httpclient", networkAddress: "127.0.0.1:8080",parentServiceName: "/httpclient-case/case/httpclient",entryApplicationInstanceId: nq 0 }

Provide metadata file

Add testcase.yml

testcase.yml Format:

testcase:
  request_url: TESTCASE_REQUEST_URL
  test_framework: TEST_FRAMEWORK_NAME
  support_versions:
    - VERSION

Field	Comment
request_url	URL used to request, use `{SERVER_OUTPUT_PORT}` as port.
test_framework	Case folder name
running_mode	TOGETHER(default), SINGLE, WITH_OPTIONAL. WITH_OPTIONAL means need include optional plugins in this case
support_versions	Tests should be run in these component versions.

testcase:
  request_url: http://localhost:{SERVER_OUTPUT_PORT}/httpclient-case/case/httpclient
  support_versions:
    - 4.3
    ...
    - 4.5.3

The whole example is here

Add Profile in maven

In profiles section in pom.xml. Each profile represents a version in support_versions in testcase.yaml. The format of profile id ${project.dir_name}-${support_version}.

Add 14 profiles in pom.xml in HttpClient case.

<profiles>
    <profile>
        <id>httpclient-4.3.x-scenario-4.5.3</id>
        <properties>
            <test.framework.version>4.5.3</test.framework.version>
        </properties>
    </profile>
    ....
    <profile>
        <id>httpclient-4.3.x-scenario-4.3</id>
        <properties>
            <test.framework.version>4.3</test.framework.version>
        </properties>
    </profile>
</profiles>

Run test cases

Intall required softwares

docker
docker-compose
maven
git

Run test case

# export project_name=httpclient-4.3.x-scenario
# bash ${SKYWALKING_AGENT_TESTCASES_HOME}/deploy-test.sh --scenario ${scenario_name} ${agent_repo} ${agent_repo_branch}

Parameter	Comment
--scenario scenario_name	Run specific test case. Default, all.
agent_repo_branch	Repository branch
agent_repo	Repository url

Check the status of testcase

The log is here.${SKYWALKING_AGENT_TESTCASES_HOME}/workspace/logs/test_report.log, If the test case validation failed， The Assert failed message should be found in the test_report.log log file。 e.g.,

assert failed.
SegmentNotFoundException:
expected:
  Segment:
  - span[-1, 0] /vertx-core-3-scenario/vertx-core/executeTest

actual:
  Segment[1.159.15542729582940000] e
  expected:	Span[-1, 0] /vertx-core-3-scenario/vertx-core/executeTest
  actual:	span[-1, 0] #local-message-receiver
  reason:	[operation name]: expected=>{/vertx-core-3-scenario/vertx-core/executeTest}, actual=>{#local-message-receiver}

Please read the document if you want fix those assert exception

¹ mock-collectoris a simulator backend, source code https://github.com/SkywalkingTest/skywalking-mock-collector.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

how-to-write-a-plugin-testcase.md

how-to-write-a-plugin-testcase.md

How to write agent plugin test case

Overview

Test case repository structure

The usage of files in test case

The workflow to write test case

Example

Code your case

Build the case docker image

Active SkyWalking agent

Expected data file

Expected data

The workflow of writing expected data file

Provide metadata file

Run test cases

Intall required softwares

Run test case

Check the status of testcase

Files

how-to-write-a-plugin-testcase.md

Latest commit

History

how-to-write-a-plugin-testcase.md

File metadata and controls

How to write agent plugin test case

Overview

Test case repository structure

The usage of files in test case

The workflow to write test case

Example

Code your case

Build the case docker image

Active SkyWalking agent

Expected data file

Expected data

The workflow of writing expected data file

Provide metadata file

Run test cases

Intall required softwares

Run test case

Check the status of testcase