add e2e test for kafka transporter by dmsolr · Pull Request #42 · apache/skywalking-java

dmsolr · 2021-10-03T15:29:42Z

If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.
Update the CHANGES log.

kezhenxu94 · 2021-10-03T15:31:06Z

test/e2e/case/kafka/expected/logs-list.yml

+# limitations under the License.
+
+logs:
+  {{- range .logs }}


Same issue as apache/skywalking#7849 (comment), please read that thread.

This is not resolved yet

dmsolr · 2021-10-03T15:32:24Z

Do we have any approach to set docker-compose file by the environment to reduce copy-codes? @kezhenxu94

kezhenxu94 · 2021-10-03T15:52:26Z

Do we have any approach to set docker-compose file by the environment to reduce copy-codes? @kezhenxu94

Not now, I'm also considering to provide this feature

wu-sheng · 2021-10-10T10:08:00Z

https://neo4j.com/docs/java-manual/current

Please fixe dead link.

wu-sheng · 2021-10-10T10:10:16Z

test/e2e/case/kafka/expected/trace-info-detail.yml

+# limitations under the License.
+
+spans:
+  {{- range .spans}}


range should be replaced by contains. We have similar suggestions in the main repo. Please check.

kezhenxu94

https://neo4j.com/docs/java-manual/current

Please fixe dead link.

@dmsolr should be https://neo4j.com/docs/java-manual/current/ (note the tailing /), I update this in another PR but it is not ready yet #41

kezhenxu94 · 2021-10-10T13:47:12Z

.github/workflows/e2e.yaml

+      matrix:
+        case:
+          - name: gRPC
+            path: test/e2e/case/base/e2e.yaml


What about changing this to test/e2e/case/grpc/e2e.yaml?

test/e2e/case/kafka/e2e.yaml

kezhenxu94 · 2021-10-10T13:49:18Z

test/e2e/case/kafka/e2e.yaml

+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql trace ls
+      expected: expected/traces-list.yml
+    # native tracing: trace detail
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql trace $(swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql trace ls|grep -A 5 'POST:/info'|tail -n1|awk -F ' ' '{print $2}')


Don't use grep, refer to how we do this in th main repo

kezhenxu94 · 2021-10-10T13:49:24Z

test/e2e/case/kafka/e2e.yaml

+      expected: expected/event-list.yml
+
+    # native log: logs list
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql logs list --service-id=$(swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql service ls|grep -B 1 'provider'|yq e '.[0].id' -) --trace-id=$(swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql trace ls|grep -A 5 'POST:/info'|tail -n1|awk -F ' ' '{print $2}')


kezhenxu94 · 2021-10-10T13:49:50Z

test/e2e/case/kafka/expected/event-list.yml

+# limitations under the License.
+
+events:
+{{- range .events }}


s/range/contains

kezhenxu94 · 2021-10-10T13:50:14Z

test/e2e/case/kafka/expected/logs-list.yml

+# limitations under the License.
+
+logs:
+  {{- range .logs }}


This is not resolved yet

kezhenxu94 · 2021-10-10T13:50:32Z

test/e2e/case/kafka/expected/service-endpoint.yml

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- range .}}


kezhenxu94 · 2021-10-10T13:50:39Z

test/e2e/case/kafka/expected/service-instance.yml

+# specific language governing permissions and limitations
+# under the License.
+
+{{- range .}}


kezhenxu94 · 2021-10-10T13:50:49Z

test/e2e/case/kafka/expected/trace-users-detail.yml

+# limitations under the License.
+
+spans:
+  {{- range .spans}}


kezhenxu94 · 2021-10-10T13:50:54Z

test/e2e/case/kafka/expected/traces-list.yml

+# limitations under the License.
+
+traces:
+{{- range .traces }}


Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>

test/e2e/case/grpc/expected/event-list.yml

test/e2e/case/grpc/expected/service-instance.yml

test/e2e/case/kafka/expected/service-endpoint.yml

test/e2e/case/kafka/expected/trace-info-detail.yml

kezhenxu94 · 2021-10-12T07:04:53Z

test/e2e/case/kafka/expected/trace-info-detail.yml

+      {{- if eq .servicecode "e2e-service-provider" }}
+      {{- cintains .refs }}
+        - traceid: {{ notEmpty .traceid }}
+          parentsegmentid: {{ notEmpty .parentsegmentid }}
+          parentspanid: 1
+          type: CROSS_PROCESS
+      {{- end }}
+      {{- end }}
+      {{- if eq .servicecode "e2e-service-consumer" }}


Suggested change

{{- if eq .servicecode "e2e-service-provider" }}

{{- cintains .refs }}

- traceid: {{ notEmpty .traceid }}

parentsegmentid: {{ notEmpty .parentsegmentid }}

parentspanid: 1

type: CROSS_PROCESS

{{- end }}

{{- end }}

{{- if eq .servicecode "e2e-service-consumer" }}

{{- if eq .servicecode "e2e-service-provider" }}

{{- cintains .refs }}

- traceid: {{ notEmpty .traceid }}

parentsegmentid: {{ notEmpty .parentsegmentid }}

parentspanid: 1

type: CROSS_PROCESS

{{- end }}

{{- end }}

{{- if eq .servicecode "e2e-service-consumer" }}

Please note, using {{- if eq .servicecode "e2e-service-provider" }} and {{- if eq .servicecode "e2e-service-consumer" }} inside contains, you actually only verify ONE of them, as soon as one of them satisfies the condition, the others won't be verified

kezhenxu94 · 2021-10-12T07:05:09Z

test/e2e/case/kafka/expected/traces-list.yml

+  {{- if eq (index .endpointnames 0) "HikariCP/Connection/getConnection" }}
+    - HikariCP/Connection/getConnection
+  {{- end }}
+  {{- if eq (index .endpointnames 0) "HikariCP/Connection/close" }}
+    - HikariCP/Connection/close
+  {{- end }}
+  {{- if eq (index .endpointnames 0) "H2/JDBI/Statement/execute" }}
+    - H2/JDBI/Statement/execute
+  {{- end}}
+  {{- if eq (index .endpointnames 0) "H2/JDBI/Statement/executeQuery" }}
+    - H2/JDBI/Statement/executeQuery
+  {{- end}}
+  {{- if eq (index .endpointnames 0) "H2/JDBI/PreparedStatement/executeQuery" }}
+    - H2/JDBI/PreparedStatement/executeQuery
+  {{- end }}
+  {{- if eq (index .endpointnames 0) "POST:/info" }}
+    - POST:/info
+  {{- end }}
+  {{- if eq (index .endpointnames 0) "POST:/users" }}
+    - POST:/users
+  {{- end }}


Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>

kezhenxu94 · 2021-10-12T11:46:41Z

test/e2e/case/grpc/expected/service-endpoint.yml


-{{- range .}}
+{{- contains .}}
 {{- if eq .name "POST:/info" }}


Remove this please

kezhenxu94

@dmsolr I'd suggest you overhaul all the if statements in the expected data file, they are mostly misused

kezhenxu94 · 2021-10-12T11:47:39Z

test/e2e/case/kafka/expected/traces-list.yml

+{{- contains .traces }}
+- segmentid: {{ notEmpty .segmentid }}
+  endpointnames:
+  {{- if eq (index .endpointnames 0) "POST:/info" }}


kezhenxu94 · 2021-10-12T11:47:54Z

test/e2e/case/kafka/expected/trace-info-detail.yml

+    {{- if eq .type "Exit" }}
+      /info
+    {{- end }}
+    {{- if eq .type "Entry" }}
+      POST:/info
+    {{- end }}
+    type: {{ notEmpty .type }}
+    peer:
+    {{- if eq .type "Exit" }}
+      provider:9090
+    {{ else }}
+      ""
+    {{- end }}
+    component:
+    {{- if eq .type "Exit" }}
+      SpringRestTemplate
+    {{- end }}
+    {{- if eq .type "Entry" }}
+      Tomcat
+    {{- end }}


These ifs

kezhenxu94 · 2021-10-12T11:47:58Z

test/e2e/case/kafka/expected/trace-info-detail.yml

+    spanid: {{ .spanid }}
+    parentspanid: {{ .parentspanid }}
+    refs:
+      {{- if eq .servicecode "e2e-service-provider" }}


test/e2e/script/env

* main: (28 commits) fix release doc (#61) Support Jedis' Transaction and fix duplicated enhancement (#57) Initialize 8.9.0 iteration (#59) Polish release shell and doc. (#58) fix rocketmq message header properties garbled characters issue (#54) Fix netty-socketio plugin test failure (#56) Add okhttp2.x plugin (#49) add e2e test for kafka transporter (#42) Fix version badge (#53) Add JDK17 supported declaration. (#52) Fix instrumentation v2 API doesn't work for constructor instrumentation. (#51) Add kylin jdbc plugin (#45) Fix version compatibility for JSON-RPC4J Plugin (#50) Feature add clickhouse jdbc plugin (#41) Update menu.yml (#48) Fix format. (#47) Add new menu for the document (#46) Doc: Update setup agent in kubernetes from 'containers' to 'initContainers'. (#44) The httpasyncclient-4.x-plugin does not take effect every time. (#40) Add an agent plugin to support Jackson (#39) ... # Conflicts: # .github/workflows/plugins-test.3.yaml # CHANGES.md # docs/en/setup/service-agent/java-agent/README.md # docs/menu.yml

add e2e test for kafka transporter

1a08e61

dmsolr added the testing label Oct 3, 2021

dmsolr requested review from kezhenxu94, mrproliu and rainbend October 3, 2021 15:29

kezhenxu94 reviewed Oct 3, 2021

View reviewed changes

dmsolr and others added 2 commits October 4, 2021 16:17

fix

91552d1

Merge branch 'main' into e2e/kafka

601e7bf

wu-sheng requested a review from kezhenxu94 October 10, 2021 10:03

wu-sheng added this to the 8.8.0 milestone Oct 10, 2021

wu-sheng reviewed Oct 10, 2021

View reviewed changes

kezhenxu94 requested changes Oct 10, 2021

View reviewed changes

kezhenxu94 mentioned this pull request Oct 11, 2021

[INFRA E2E][Feature] allow reusing the verify.cases apache/skywalking#7896

Closed

3 tasks

dmsolr and others added 4 commits October 12, 2021 13:28

fix

061983e

fix

2184878

Update test/e2e/case/kafka/e2e.yaml

8641738

Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>

Merge branch 'main' into e2e/kafka

77cc8f6

wu-sheng requested a review from kezhenxu94 October 12, 2021 06:43

kezhenxu94 reviewed Oct 12, 2021

View reviewed changes

test/e2e/case/grpc/expected/event-list.yml Outdated Show resolved Hide resolved

kezhenxu94 reviewed Oct 12, 2021

View reviewed changes

test/e2e/case/grpc/expected/service-instance.yml Show resolved Hide resolved

kezhenxu94 reviewed Oct 12, 2021

View reviewed changes

test/e2e/case/kafka/expected/service-endpoint.yml Outdated Show resolved Hide resolved

kezhenxu94 reviewed Oct 12, 2021

View reviewed changes

test/e2e/case/kafka/expected/trace-info-detail.yml Outdated Show resolved Hide resolved

kezhenxu94 reviewed Oct 12, 2021

View reviewed changes

dmsolr and others added 3 commits October 12, 2021 15:13

Update test/e2e/case/grpc/expected/event-list.yml

f0f9681

Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>

Update test/e2e/case/kafka/expected/service-endpoint.yml

e412a56

Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>

Update test/e2e/case/kafka/expected/trace-info-detail.yml

5de066b

Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>

dmsolr and others added 3 commits October 12, 2021 15:13

Update test/e2e/case/grpc/expected/service-instance.yml

7fc5c22

Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>

fix

0d1fcff

upgrade infra e2e

cb40cf9

kezhenxu94 reviewed Oct 12, 2021

View reviewed changes

dmsolr added 7 commits October 12, 2021 23:43

update

046e1a1

update

76ad59f

update

c662f5a

fix

a376c14

Merge branch 'main' into e2e/kafka

9d7a1ed

update cli version

95cb8ed

installer script

cd317d6

wu-sheng reviewed Oct 17, 2021

View reviewed changes

test/e2e/script/env Outdated Show resolved Hide resolved

upgrade go version 1.16

0a7ed22

wu-sheng requested a review from kezhenxu94 October 17, 2021 13:07

kezhenxu94 approved these changes Oct 18, 2021

View reviewed changes

wu-sheng merged commit fbdfc42 into apache:main Oct 18, 2021

GuoHaoZai pushed a commit to GuoHaoZai/skywalking-java that referenced this pull request Apr 24, 2025

Add events protocol message and service (apache#42)

64b2321

Conversation

dmsolr commented Oct 3, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmsolr commented Oct 3, 2021

Uh oh!

kezhenxu94 commented Oct 3, 2021

Uh oh!

wu-sheng commented Oct 10, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kezhenxu94 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kezhenxu94 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants