Add a doc to introduce how to use e2e to test itself

test/e2e/README.md

@@ -0,0 +1,105 @@
+# Use E2E to test itself
+
+After updating the features of e2e, you can use the files in the test/e2e/ directory to perform testing for both new and old features of e2e.
+You can perform testing locally. And when you submit a pull request (PR), GitHub Actions will automatically run the tests.
+
+## Files introduction
+- `test/e2e`
+    - `concurrency`
+        - `fail-fast` (concurrency & fail-fast mode)
+          - `internal`
+            - `expected.yaml` 
+            - `verify.yaml` (configuration file for the inner infra E2E(concurrency & fail-fast))
+          - `expected.yaml`
+        - `non-fail-fast` (concurrency & non-fail-fast mode)
+          - `internal`
+              - `expected.yaml`
+              - `verify.yaml` (configuration file for the inner infra E2E(concurrency & non-fail-fast))
+          - `expected.yaml`
+    - `non-concurrency` 
+        - `fail-fast` (non-concurrency & fail-fast mode)
+            - `internal`
+              - `expected.yaml`
+              - `verify.yaml` (configuration file for the inner infra E2E(non-concurrency & fail-fast))
+            - `expected.yaml`
+        - `non-fail-fast` (non-concurrency & non-fail-fast mode)
+            - `internal`
+                - `expected.yaml`
+                - `verify.yaml` (configuration file for the inner infra E2E(non-concurrency & non-fail-fast))
+            - `expected.yaml`
+    - `docker-compose.yaml` (run a httpbin container, which can return YAML data)
+    - `e2e.yaml` (configuration file for the outer infra E2E)
+
+## Tips
+### concurrency & fail-fast mode
+- We add `&& sleep=5` in `query`of the last few cases, to make sure that there will be cases to be skipped.
+examples: `'curl -s 127.0.0.1:8080/get?case=failure -H "accept: application/json" && sleep 5'`.
+- In the `expected.yaml`, we use yaml grammar in it:
+```yaml
+# It means that the `passed` field in the `expected.yaml` will be populated
+# with the elements (which contains a prefix of "passed") of `passed` field in the summary generated by infra E2E (concurrency & fail-fast mode).
+passed: 
+{{range .passed}}
+- {{ if hasPrefix . "passed" }}{{.}}{{ end }}
+{{end}}
+# It means that the `failed` field in the `expected.yaml` will be populated 
+# with the elements (which contain a prefix of "failed") of `failed` field in the summary generated by infra E2E (concurrency & fail-fast mode).
+failed:
+  {{range .failed}}
+- {{ if hasPrefix . "failed" }}{{.}}{{ end }}
+  {{end}}
+skipped:
+  {{range .skipped}}
+- {{.}}
+  {{end}}
+
+# if A <= B, the value of {{le A B}} is A.
+# you need to replace the `totalPassedCount` with your real total count of passed cases. 
+passedCount: {{le .passedCount totalPassedCount}}
+# you need to replace the `totalFailedCount` with your real total count of failed cases. 
+failedCount: {{le .failedCount totalFailedCount}}
+# the subtrtactor is a function. The value of {{subtrator A B C D ...}} is (A-B-C-D-...)
+# you need to replace the totalCount with your real count of cases.
+skippedCount: {{subtractor totalCount .passedCount .failedCount}}
+```
+
+### concurrency & non-fail-fast mode
+- In the `expected.yaml`, we use yaml grammar in it:
+```yaml
+# unlike in concurrency & fail-fast mode, the summary of infra E2E (concurrency & non-fail-fast) is predictable.
+# but the sequence of the elements of `passed` and `failed` field of the summary isn't predictable, so we use `contains`.
+# `contains` selects the elements from the collection of `contains` that belong to the collection of `passed` (the collection  of `passed` field of summary),
+# and arranges them in the order they appear in collection of `passed`. Then, it populates the `passed` field in `expected.yaml` with these elements.
+passed:
+{{- contains .passed}}
+- passed-case-1
+- passed-case-2
+- passed-case-3
+...
+{{- end}}
+# `contains` is the same in `failed`.
+failed:
+{{- contains .failed }}
+- failed-case-1
+- failed-case-2
+- failed-case-3
+...
+{{- end }}
+skipped: []
+# you need to replace the `totalPassedCount` with your real total count of passed cases. 
+passedCount: totalPassedCount
+# you need to replace the `totalFailedCount` with your real total count of failed cases. 
+failedCount: totalFailedCount
+skippedCount: 0
+```
+
+## Local Test
+After `make build`, you can use `skywalking-infra-e2e/test/e2e/e2e.yaml` as the configuration file to run `e2e run`.
+
+
+
+
+
+
+
+

test/e2e/concurrency/fail-fast/expected.yaml

@@ -29,4 +29,4 @@ skipped:
 {{end}}
 passedCount: {{le .passedCount 4}}
 failedCount: {{le .failedCount 5}}
-skippedCount: {{.skippedCount}}
+skippedCount: {{subtractor 9 .passedCount .failedCount}}

test/e2e/e2e.yaml

@@ -17,6 +17,7 @@
 
 setup:
   env: compose
+  # Run a httpbin container, which can return YAML data
   file: docker-compose.yml
   timeout: 20m
 
@@ -32,9 +33,9 @@ verify:
     # the interval between two attempts, e.g. 10s, 1m.
     interval: 1s
 
-  # when a case fails, whether to stop verifying other cases. This property defaults to true.
+  # when a case fails, whether to stop verifying other cases. This property is recommended to false.
   fail-fast: false
-  # Whether to verify cases concurrently. This property defaults to false.
+  # Whether to verify cases concurrently. This property is recommended to false.
   concurrency: false