feat: expose more prometheus metrics #670

Sindweller · 2021-09-07T09:16:07Z

Please answer these questions before submitting a pull request

Why submit this pull request?
Bugfix
New feature provided
Improve performance
Backport patches
Related issues

New feature or improvement

Describe the details and related test reports.
Expose several metrics for apisix-ingress-controller. This PR has a prior PR: feat: add metric 'check_cluster_health_total' and 'sync_operation_total' #627

HELP apisix_ingress_controller_is_leader Whether the role of controller instance is leader
TYPE apisix_ingress_controller_is_leader guage
Labels:

controller_pod
controller_namespace

HELP apisix_ingress_controller_apisix_bad_status_codes Bad status codes of requests to APISIX
TYPE apisix_ingress_controller_apisix_bad_status_codes guage
Labels:

controller_pod
controller_namespace
status_code
resource

HELP apisix_ingress_controller_apisix_request_latencies Request latencies with APISIX
TYPE apisix_ingress_controller_apisix_request_latencies summary
Labels:

controller_pod
controller_namespace

HELP apisix_ingress_controller_apisix_requests Number of requests to APISIX
TYPE apisix_ingress_controller_apisix_requests counter
Labels:

controller_pod
controller_namespace
resource

HELP apisix_ingress_controller_check_cluster_health_total Number of cluster health check operations
TYPE apisix_ingress_controller_check_cluster_health_total counter
Labels:

controller_pod
controller_namespace
name

HELP apisix_ingress_controller_sync_operation_total Number of sync operations
TYPE apisix_ingress_controller_sync_operation_total counter
Labels:

controller_pod
controller_namespace
resource
result

HELP apisix_ingress_controller_cache_sync_total Number of cache sync operations
TYPE apisix_ingress_controller_cache_sync_total counter
Labels:

controller_pod
controller_namespace
result

HELP apisix_ingress_controller_events_total Number of events handled by the controller
TYPE apisix_ingress_controller_events_total counter
Labels:

controller_pod
controller_namespace
resource
operation

pkg/apisix/cluster.go

pkg/ingress/apisix_tls.go

… codes, also fix TLS typo

Sindweller · 2021-09-08T14:20:18Z

@tokers done.

tokers · 2021-09-09T01:06:33Z

@Sindweller Please no longer force push, you can just use merge (from master).

tokers · 2021-09-09T01:11:58Z

@Sindweller Changes LGTM for me but some CI failures.

tao12345666333 · 2021-09-09T03:09:05Z

please check the lint and unit test errors

Sindweller · 2021-09-09T08:40:18Z

Should have resolved the error in unit-test. It's due to the lack of cluster metricCollector.
But I need help to solve the e2e-test failed errors, like:
Error: Expected nil, but got: &errors.errorString{s:"timed out waiting for the condition"}

• Failure in Spec Setup (BeforeEach) [86.687 seconds]
rewrite annotations
/Users/sindweller/apisix-ingress-controller/test/e2e/annotations/rewrite.go:28
  enable in ingress networking/v1beta1 [BeforeEach]
  /Users/sindweller/apisix-ingress-controller/test/e2e/annotations/rewrite.go:62

  
  	Error Trace:	scaffold.go:325
  	            				runner.go:113
  	            				runner.go:64
  	            				setup_nodes.go:15
  	            				spec.go:193
  	            				spec.go:138
  	            				spec_runner.go:200
  	            				spec_runner.go:170
  	            				spec_runner.go:66
  	            				suite.go:79
  	            				ginkgo_dsl.go:238
  	            				ginkgo_dsl.go:213
  	            				e2e_test.go:26
  	Error:      	Expected nil, but got: &errors.errorString{s:"timed out waiting for the condition"}
  	Test:       	rewrite annotations enable in ingress networking/v1beta1
  	Messages:   	waiting for ingress apisix controller ready

It seems my changes may not affect these files and cause the error.

tokers · 2021-09-09T12:52:05Z

Should have resolved the error in unit-test. It's due to the lack of cluster metricCollector.
But I need help to solve the e2e-test failed errors, like:
Error: Expected nil, but got: &errors.errorString{s:"timed out waiting for the condition"}

• Failure in Spec Setup (BeforeEach) [86.687 seconds]
rewrite annotations
/Users/sindweller/apisix-ingress-controller/test/e2e/annotations/rewrite.go:28
  enable in ingress networking/v1beta1 [BeforeEach]
  /Users/sindweller/apisix-ingress-controller/test/e2e/annotations/rewrite.go:62

  
  	Error Trace:	scaffold.go:325
  	            				runner.go:113
  	            				runner.go:64
  	            				setup_nodes.go:15
  	            				spec.go:193
  	            				spec.go:138
  	            				spec_runner.go:200
  	            				spec_runner.go:170
  	            				spec_runner.go:66
  	            				suite.go:79
  	            				ginkgo_dsl.go:238
  	            				ginkgo_dsl.go:213
  	            				e2e_test.go:26
  	Error:      	Expected nil, but got: &errors.errorString{s:"timed out waiting for the condition"}
  	Test:       	rewrite annotations enable in ingress networking/v1beta1
  	Messages:   	waiting for ingress apisix controller ready

It seems my changes may not affect these files and cause the error.

That's right, it's due to the speed of GitHub action, we have some plans to optimize it, for now, please just retry it.

gxthrj · 2021-09-09T13:13:16Z

Since #627 has been merged, please do with conflicting files.

Sindweller · 2021-09-10T04:05:10Z

I'm used to using git fetch upstream and then git rebase upstream/master on my local branch so that I can apply my changes on the newest codes from upstream. But then I need to use git push --force my branch-name to force pushed the commit to keep a clearly linear commit tree. (clear for me)
This time I try to use git pull instead of git rebase to avoid force push. But the commits seems weird. The accepted pr has overwrite my changes, and I pushed a new commit to apply all my new changes.

So what's the correct steps of this situation? Can I still use git rebase?
btw, if I want to use git commit --amend, I need to use git push --force as well. Is it acceptable?

codecov-commenter · 2021-09-10T04:23:42Z

Codecov Report

Merging #670 (60cd7b0) into master (4a862e2) will increase coverage by 0.50%.
The diff coverage is 55.44%.

@@            Coverage Diff             @@
##           master     #670      +/-   ##
==========================================
+ Coverage   31.70%   32.21%   +0.50%     
==========================================
  Files          66       66              
  Lines        6640     6768     +128     
==========================================
+ Hits         2105     2180      +75     
- Misses       4280     4333      +53     
  Partials      255      255

Impacted Files	Coverage Δ
pkg/ingress/apisix_cluster_config.go	`0.00% <0.00%> (ø)`
pkg/ingress/apisix_consumer.go	`0.00% <0.00%> (ø)`
pkg/ingress/apisix_route.go	`0.00% <0.00%> (ø)`
pkg/ingress/apisix_tls.go	`0.00% <0.00%> (ø)`
pkg/ingress/apisix_upstream.go	`0.00% <0.00%> (ø)`
pkg/ingress/controller.go	`1.00% <0.00%> (-0.01%)`	⬇️
pkg/ingress/endpoint.go	`0.00% <0.00%> (ø)`
pkg/ingress/endpointslice.go	`0.00% <0.00%> (ø)`
pkg/ingress/ingress.go	`8.10% <0.00%> (-0.19%)`	⬇️
pkg/ingress/secret.go	`0.00% <0.00%> (ø)`
... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 774077a...60cd7b0. Read the comment docs.

tokers · 2021-09-11T14:54:48Z

I'm used to using git fetch upstream and then git rebase upstream/master on my local branch so that I can apply my changes on the newest codes from upstream. But then I need to use git push --force my branch-name to force pushed the commit to keep a clearly linear commit tree. (clear for me)
This time I try to use git pull instead of git rebase to avoid force push. But the commits seems weird. The accepted pr has overwrite my changes, and I pushed a new commit to apply all my new changes.

So what's the correct steps of this situation? Can I still use git rebase?

btw, if I want to use git commit --amend, I need to use git push --force as well. Is it acceptable?

You don't need release, just merge the master branch is OK, don't be overwhelmed by the extra Merge commit, we use squash merge strategy so finally only a commit will be merged into the master branch and the tree will be clean.

… codes, also fix TLS typo

tao12345666333

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0x155bc1b]

goroutine 680 [running]:
github.com/apache/apisix-ingress-controller/pkg/apisix.(*cluster).getList(0xc0004f1680, 0x1c687d0, 0xc000044118, 0xc0003873b0, 0x30, 0x19bdf54, 0x6, 0x0, 0x0, 0x0, ...)
        /home/tao/go/src/github.com/apache/apisix-ingress-controller/pkg/apisix/cluster.go:711 +0x19b
github.com/apache/apisix-ingress-controller/pkg/apisix.(*pluginClient).List(0xc0005c1440, 0x1c687d0, 0xc000044118, 0x0, 0x0, 0x0, 0x0, 0x2)
        /home/tao/go/src/github.com/apache/apisix-ingress-controller/pkg/apisix/plugin.go:44 +0x3b9
github.com/apache/apisix-ingress-controller/pkg/apisix.(*cluster).syncSchemaOnce(0xc0004f1680, 0x1c687d0, 0xc000044118, 0x0, 0xc0008af798)
        /home/tao/go/src/github.com/apache/apisix-ingress-controller/pkg/apisix/cluster.go:365 +0x74f
github.com/apache/apisix-ingress-controller/pkg/apisix.(*cluster).syncSchema(0xc0004f1680, 0x1c687d0, 0xc000044118, 0x13a52453c000)
        /home/tao/go/src/github.com/apache/apisix-ingress-controller/pkg/apisix/cluster.go:329 +0xa5
created by github.com/apache/apisix-ingress-controller/pkg/apisix.newCluster
        /home/tao/go/src/github.com/apache/apisix-ingress-controller/pkg/apisix/cluster.go:150 +0x852

Please fix the e2e test case.

tao12345666333 · 2021-10-18T10:32:01Z

you can run

cd test/e2e
ginkgo --focus='support ingress https should support ingress v1 with kube style tls secret'

for local test.

Sindweller · 2021-10-19T02:02:35Z

you can run

cd test/e2e
ginkgo --focus='support ingress https should support ingress v1 with kube style tls secret'

for local test.

thx.
I've hit this error, working on it.

Sindweller · 2021-10-25T02:12:07Z

hi, this problem is caused by c.metricsCollector.RecordAPISIXLatency(time.Since(start), "getList"). In e2e test case maybe there's no metricsCollector in cluster since it's passed by controller.go. In unit test, I called NewPrometheusCollector for every test case so that the test passed. Could you help me find out the relative file to instantiate metricsCollector for cluster in e2e test?

tokers · 2021-10-25T11:44:40Z

hi, this problem is caused by c.metricsCollector.RecordAPISIXLatency(time.Since(start), "getList"). In e2e test case maybe there's no metricsCollector in cluster since it's passed by controller.go. In unit test, I called NewPrometheusCollector for every test case so that the test passed. Could you help me find out the relative file to instantiate metricsCollector for cluster in e2e test?

This is not the logic of E2E test cases but the apisix-ingress-controller itself, see https://github.com/apache/apisix-ingress-controller/blob/master/pkg/ingress/controller.go#L382 for the details.

tao12345666333 · 2021-11-09T06:21:41Z

CI failed

Sindweller · 2021-11-17T12:38:58Z

Here is the CI log:

2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/zipkin", "url": "http://localhost:30046/apisix/admin/schema//plugins/zipkin", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/zipkin: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "zipkin", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/zipkin: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/stream_route.go:117	failed to list stream_routes: Get http://localhost:30046/apisix/admin/stream_routes: dial tcp [::1]:30046: connect: connection refused
2021-11-09T02:50:22Z	error	apisix/cluster.go:215	failed to list stream_routes in APISIX: Get http://localhost:30046/apisix/admin/stream_routes: dial tcp [::1]:30046: connect: connection refused
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/request-id", "url": "http://localhost:30046/apisix/admin/schema//plugins/request-id", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/request-id: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "request-id", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/request-id: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/fault-injection", "url": "http://localhost:30046/apisix/admin/schema//plugins/fault-injection", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/fault-injection: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "fault-injection", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/fault-injection: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/serverless-pre-function", "url": "http://localhost:30046/apisix/admin/schema//plugins/serverless-pre-function", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/serverless-pre-function: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "serverless-pre-function", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/serverless-pre-function: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/batch-requests", "url": "http://localhost:30046/apisix/admin/schema//plugins/batch-requests", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/batch-requests: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "batch-requests", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/batch-requests: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/cors", "url": "http://localhost:30046/apisix/admin/schema//plugins/cors", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/cors: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "cors", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/cors: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/ip-restriction", "url": "http://localhost:30046/apisix/admin/schema//plugins/ip-restriction", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/ip-restriction: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "ip-restriction", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/ip-restriction: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/ua-restriction", "url": "http://localhost:30046/apisix/admin/schema//plugins/ua-restriction", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/ua-restriction: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "ua-restriction", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/ua-restriction: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/referer-restriction", "url": "http://localhost:30046/apisix/admin/schema//plugins/referer-restriction", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/referer-restriction: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "referer-restriction", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/referer-restriction: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/uri-blocker", "url": "http://localhost:30046/apisix/admin/schema//plugins/uri-blocker", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/uri-blocker: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "uri-blocker", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/uri-blocker: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/request-validation", "url": "http://localhost:30046/apisix/admin/schema//plugins/request-validation", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/request-validation: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "request-validation", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/request-validation: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/openid-connect", "url": "http://localhost:30046/apisix/admin/schema//plugins/openid-connect", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/openid-connect: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "openid-connect", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/openid-connect: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/authz-casbin", "url": "http://localhost:30046/apisix/admin/schema//plugins/authz-casbin", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/authz-casbin: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "authz-casbin", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/authz-casbin: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/wolf-rbac", "url": "http://localhost:30046/apisix/admin/schema//plugins/wolf-rbac", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/wolf-rbac: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "wolf-rbac", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/wolf-rbac: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/ldap-auth", "url": "http://localhost:30046/apisix/admin/schema//plugins/ldap-auth", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/ldap-auth: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "ldap-auth", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/ldap-auth: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/hmac-auth", "url": "http://localhost:30046/apisix/admin/schema//plugins/hmac-auth", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/hmac-auth: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "hmac-auth", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/hmac-auth: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/basic-auth", "url": "http://localhost:30046/apisix/admin/schema//plugins/basic-auth", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/basic-auth: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "basic-auth", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/basic-auth: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/jwt-auth", "url": "http://localhost:30046/apisix/admin/schema//plugins/jwt-auth", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/jwt-auth: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "jwt-auth", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/jwt-auth: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/key-auth", "url": "http://localhost:30046/apisix/admin/schema//plugins/key-auth", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/key-auth: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "key-auth", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/key-auth: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/consumer-restriction", "url": "http://localhost:30046/apisix/admin/schema//plugins/consumer-restriction", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/consumer-restriction: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "consumer-restriction", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/consumer-restriction: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/authz-keycloak", "url": "http://localhost:30046/apisix/admin/schema//plugins/authz-keycloak", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/authz-keycloak: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "authz-keycloak", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/authz-keycloak: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/proxy-mirror", "url": "http://localhost:30046/apisix/admin/schema//plugins/proxy-mirror", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/proxy-mirror: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "proxy-mirror", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/proxy-mirror: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/proxy-cache", "url": "http://localhost:30046/apisix/admin/schema//plugins/proxy-cache", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/proxy-cache: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "proxy-cache", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/proxy-cache: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/proxy-rewrite", "url": "http://localhost:30046/apisix/admin/schema//plugins/proxy-rewrite", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/proxy-rewrite: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "proxy-rewrite", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/proxy-rewrite: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/api-breaker", "url": "http://localhost:30046/apisix/admin/schema//plugins/api-breaker", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/api-breaker: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "api-breaker", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/api-breaker: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/limit-conn", "url": "http://localhost:30046/apisix/admin/schema//plugins/limit-conn", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/limit-conn: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "limit-conn", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/limit-conn: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/limit-count", "url": "http://localhost:30046/apisix/admin/schema//plugins/limit-count", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/limit-count: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "limit-count", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/limit-count: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/limit-req", "url": "http://localhost:30046/apisix/admin/schema//plugins/limit-req", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/limit-req: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "limit-req", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/limit-req: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/gzip", "url": "http://localhost:30046/apisix/admin/schema//plugins/gzip", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/gzip: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "gzip", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/gzip: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/server-info", "url": "http://localhost:30046/apisix/admin/schema//plugins/server-info", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/server-info: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "server-info", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/server-info: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/traffic-split", "url": "http://localhost:30046/apisix/admin/schema//plugins/traffic-split", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/traffic-split: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "traffic-split", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/traffic-split: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/redirect", "url": "http://localhost:30046/apisix/admin/schema//plugins/redirect", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/redirect: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "redirect", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/redirect: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/response-rewrite", "url": "http://localhost:30046/apisix/admin/schema//plugins/response-rewrite", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/response-rewrite: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "response-rewrite", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/response-rewrite: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/grpc-transcode", "url": "http://localhost:30046/apisix/admin/schema//plugins/grpc-transcode", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/grpc-transcode: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "grpc-transcode", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/grpc-transcode: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/prometheus", "url": "http://localhost:30046/apisix/admin/schema//plugins/prometheus", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/prometheus: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "prometheus", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/prometheus: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/echo", "url": "http://localhost:30046/apisix/admin/schema//plugins/echo", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/echo: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "echo", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/echo: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/http-logger", "url": "http://localhost:30046/apisix/admin/schema//plugins/http-logger", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/http-logger: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "http-logger", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/http-logger: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/sls-logger", "url": "http://localhost:30046/apisix/admin/schema//plugins/sls-logger", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/sls-logger: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "sls-logger", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/sls-logger: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/tcp-logger", "url": "http://localhost:30046/apisix/admin/schema//plugins/tcp-logger", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/tcp-logger: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "tcp-logger", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/tcp-logger: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/kafka-logger", "url": "http://localhost:30046/apisix/admin/schema//plugins/kafka-logger", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/kafka-logger: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "kafka-logger", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/kafka-logger: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/syslog", "url": "http://localhost:30046/apisix/admin/schema//plugins/syslog", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/syslog: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "syslog", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/syslog: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/udp-logger", "url": "http://localhost:30046/apisix/admin/schema//plugins/udp-logger", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/udp-logger: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "udp-logger", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/udp-logger: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/example-plugin", "url": "http://localhost:30046/apisix/admin/schema//plugins/example-plugin", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/example-plugin: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "example-plugin", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/example-plugin: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/serverless-post-function", "url": "http://localhost:30046/apisix/admin/schema//plugins/serverless-post-function", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/serverless-post-function: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "serverless-post-function", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/serverless-post-function: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	error	apisix/schema.go:69	failed to get schema from APISIX	{"name": "plugins/ext-plugin-post-req", "url": "http://localhost:30046/apisix/admin/schema//plugins/ext-plugin-post-req", "cluster": "default", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/ext-plugin-post-req: dial tcp [::1]:30046: connect: connection refused"}
2021-11-09T02:50:22Z	warn	apisix/cluster.go:373	failed to get plugin schema	{"plugin": "ext-plugin-post-req", "error": "Get http://localhost:30046/apisix/admin/schema//plugins/ext-plugin-post-req: dial tcp [::1]:30046: connect: connection refused"}
retries is a positive number 2021-11-09T02:50:22Z logger.go:66: warning: deleting cluster-scoped resources, not scoped to the provided namespace
retries is a positive number 2021-11-09T02:50:22Z logger.go:66: clusterrolebinding.rbac.authorization.k8s.io "ingress-apisix-e2e-tests-default-499295280-clusterrolebinding" deleted
retries is a positive number 2021-11-09T02:50:22Z logger.go:66: Running command kubectl with args [--kubeconfig /home/runner/.kube/config --namespace ingress-apisix-e2e-tests-default-499295280 delete -f /tmp/retries%20is%20a%20positive%20number909682229]
retries is a positive number 2021-11-09T02:50:22Z logger.go:66: warning: deleting cluster-scoped resources, not scoped to the provided namespace
retries is a positive number 2021-11-09T02:50:22Z logger.go:66: clusterrole.rbac.authorization.k8s.io "ingress-apisix-e2e-tests-default-499295280-apisix-view-clusterrole" deleted
2021-11-09T02:50:24Z	error	apisix/route.go:119	failed to list routes: Get http://localhost:30046/apisix/admin/routes: dial tcp [::1]:30046: connect: connection refused
2021-11-09T02:50:24Z	error	apisix/cluster.go:200	failed to list route in APISIX: Get http://localhost:30046/apisix/admin/routes: dial tcp [::1]:30046: connect: connection refused

Could you help to resolve error dial tcp [::1]:30046: connect: connection refused"? What does it mean?
And what should I do to resolve it?

tao12345666333 · 2021-11-17T14:04:15Z

Could you merge master branch?

Sindweller · 2021-11-18T01:38:22Z

Could you merge master branch?

Already done. Here is the git log locally.

commit 60cd7b0d651f9228cc84379b961d1af391f9a79b (HEAD -> other-metrics, origin/other-metrics)
Author: Sindweller <sindweller5530@gmail.com>
Date:   Wed Nov 17 20:52:11 2021 +0800

    fix some err

commit 11e39a33c6a30175bf9e4d443d65faf8fabdaca9
Merge: d825c18 774077a
Author: Sindweller <sindweller5530@gmail.com>
Date:   Wed Nov 17 19:46:01 2021 +0800

    resolve conflict

commit 774077a527e43775bcd6346bebdb2ae0b3f80c22 (upstream/master)
Author: Jintao Zhang <zhangjintao9020@gmail.com>
Date:   Sat Nov 13 18:40:50 2021 +0800

    docs: Customize the namespace used for installation (#747)

commit 4a862e206602ae9c7ac534fdfd9a557748b9ad26
Author: Nic <qianyong@api7.ai>
Date:   Fri Nov 12 11:59:54 2021 +0800

    fix: use independent dns service for UDP e2e test (#753)
...

tao12345666333 · 2021-11-18T18:19:12Z

It's Green!

tokers · 2021-11-19T09:22:35Z

@Sindweller Merged, thanks for your great job!

Sindweller added 2 commits September 7, 2021 16:25

add metric: check_cluster_health and sync_operation_total

d21fb0c

add prometheus metrics

da1ece0

tokers reviewed Sep 7, 2021

View reviewed changes

pkg/apisix/cluster.go Show resolved Hide resolved

pkg/apisix/cluster.go Outdated Show resolved Hide resolved

pkg/apisix/cluster.go Outdated Show resolved Hide resolved

tokers reviewed Sep 8, 2021

View reviewed changes

pkg/ingress/apisix_tls.go Outdated Show resolved Hide resolved

add operation label to apisix_request_latencies and record all status…

bbe13c9

… codes, also fix TLS typo

Sindweller force-pushed the other-metrics branch from d76168b to bbe13c9 Compare September 8, 2021 14:17

tokers approved these changes Sep 9, 2021

View reviewed changes

fix test error of nil pointer

2a1c62c

expose more metrics

e244660

add metricCollector in pod_test.go

d60022d

change time calculation method

faff77c

resolve conflict

ed4fd31

Sindweller mentioned this pull request Sep 28, 2021

request help: metrics was exposed but not exist in Prometheus #696

Closed

gxthrj approved these changes Sep 29, 2021

View reviewed changes

Sindweller added 6 commits September 30, 2021 09:52

add prometheus metrics

2655a42

add operation label to apisix_request_latencies and record all status…

d6d3115

… codes, also fix TLS typo

fix test error of nil pointer

282ec12

add metricCollector in pod_test.go

fa5dcfb

change time calculation method

c64e4eb

pass MetricsCollector to cluster

a366e7f

tao12345666333 removed the do-not-merge/hold label Oct 18, 2021

tao12345666333 added this to In progress in v1.4 Planning via automation Oct 18, 2021

tao12345666333 added this to the 1.4.0 milestone Oct 18, 2021

tao12345666333 requested changes Oct 18, 2021

View reviewed changes

v1.4 Planning automation moved this from In progress to Review in progress Oct 18, 2021

Sindweller mentioned this pull request Nov 1, 2021

docs: Grafana Dashboard Configuration #731

Merged

4 tasks

Sindweller added 2 commits November 9, 2021 10:31

fix e2e test nil pointer err

4461888

merge conflicts

d825c18

resolve conflict

11e39a3

fix some err

60cd7b0

tao12345666333 requested review from gxthrj and tokers November 19, 2021 01:35

tao12345666333 approved these changes Nov 19, 2021

View reviewed changes

v1.4 Planning automation moved this from Review in progress to Reviewer approved Nov 19, 2021

tokers approved these changes Nov 19, 2021

View reviewed changes

tokers merged commit 580e7d4 into apache:master Nov 19, 2021

v1.4 Planning automation moved this from Reviewer approved to Done Nov 19, 2021

tao12345666333 added the enhancement New feature or request label Nov 19, 2021

Sindweller added a commit to Sindweller/apisix-ingress-controller that referenced this pull request Nov 25, 2021

feat: expose more prometheus metrics (apache#670)

b0ca273

Sindweller mentioned this pull request Dec 18, 2021

request help: Grafana dashboard config for apisix-ingress-controller #691

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: expose more prometheus metrics #670

feat: expose more prometheus metrics #670

Sindweller commented Sep 7, 2021

Sindweller commented Sep 8, 2021

tokers commented Sep 9, 2021

tokers commented Sep 9, 2021

tao12345666333 commented Sep 9, 2021

Sindweller commented Sep 9, 2021

tokers commented Sep 9, 2021

gxthrj commented Sep 9, 2021

Sindweller commented Sep 10, 2021

codecov-commenter commented Sep 10, 2021 •

edited

Loading

tokers commented Sep 11, 2021

tao12345666333 left a comment

tao12345666333 commented Oct 18, 2021

Sindweller commented Oct 19, 2021 •

edited

Loading

Sindweller commented Oct 25, 2021

tokers commented Oct 25, 2021

tao12345666333 commented Nov 9, 2021

Sindweller commented Nov 17, 2021

tao12345666333 commented Nov 17, 2021

Sindweller commented Nov 18, 2021

tao12345666333 commented Nov 18, 2021

tokers commented Nov 19, 2021

feat: expose more prometheus metrics #670

feat: expose more prometheus metrics #670

Conversation

Sindweller commented Sep 7, 2021

New feature or improvement

Sindweller commented Sep 8, 2021

tokers commented Sep 9, 2021

tokers commented Sep 9, 2021

tao12345666333 commented Sep 9, 2021

Sindweller commented Sep 9, 2021

tokers commented Sep 9, 2021

gxthrj commented Sep 9, 2021

Sindweller commented Sep 10, 2021

codecov-commenter commented Sep 10, 2021 • edited Loading

Codecov Report

tokers commented Sep 11, 2021

tao12345666333 left a comment

Choose a reason for hiding this comment

tao12345666333 commented Oct 18, 2021

Sindweller commented Oct 19, 2021 • edited Loading

Sindweller commented Oct 25, 2021

tokers commented Oct 25, 2021

tao12345666333 commented Nov 9, 2021

Sindweller commented Nov 17, 2021

tao12345666333 commented Nov 17, 2021

Sindweller commented Nov 18, 2021

tao12345666333 commented Nov 18, 2021

tokers commented Nov 19, 2021

codecov-commenter commented Sep 10, 2021 •

edited

Loading

Sindweller commented Oct 19, 2021 •

edited

Loading