bug: Upstream scheme used is https when using k8s Ingress resource

[shantanu10]

Current Behavior

I am using apisix ingress controller in the composite mode architecture in a k8s cluster. Whenever I apply any ingress with http scheme (no TLS), some how apisix ingress controller is saving the upstream with scheme as "https". Due to that the requests are returning 502 when proxied through apisix. I also see this log for the failing request inside apisix container:

2024/05/22 07:23:10 [crit] 51#51: *5144 SSL_do_handshake() failed (SSL: error:0A0000C6:SSL routines::packet length too long error:0A000139:SSL routines::record layer failure) while SSL handshaking to upstream, client: 10.1.45.13, server: _, request: "GET /api/webhook HTTP/1.1", upstream: "https://10.1.33.14:7000/api/webhook", host: "argocd.mycompany.com"

This is probably due to the fact that apisix is trying to use ssl while connecting to the upstream's port which is not configured to do so. The root cause is the use of https by apisix instead of http despite I used http in the ingress configuration. This I verified by using the admin api to fetch all upstreams and also in the apisix logs:

{"time": "2024-05-22T07:23:10+00:00","opentelemetry_context_traceparent": "00-59ab8aeb89007e53bc5d7d1c3b4b0561-e5cb39185ea6d409-00","trace_id": "59ab8aeb89007e53bc5d7d1c3b4b0561","span_id": "e5cb39185ea6d409","remote_addr": "10.1.38.19","host": "argocd.mycompany.com","uri": "/api/webhook", "status": "502", "upstream_addr": "10.1.35.158:8080", "upstream_status": "502", "upstream_response_time": "0.005", "upstream_url": "https://10.1.35.158:8080"}

Here are some other observations:

1. This is not happening when i am using the ApisixRoute crd. Although I won't be able to use it permanently since there are some annotations defined here which i need to use which can only be used with Ingress resource.

2. It started coming after almost 1 month of using apisix ingress controller. This while i just upgraded to version 1.8.1 from 1.7.1 last week. But the issue appeared yesterday. I can ensure that I made no change in the other part of my infrastructure ,i.e, k8s version, load balancer etc.

Expected Behavior

Ideally the upstream should be saved with scheme as "http" on applying any ingress resource in the cluster.

Error Logs

Container: apisix

2024/05/22 07:23:10 [crit] 51#51: *5144 SSL_do_handshake() failed (SSL: error:0A0000C6:SSL routines::packet length too long error:0A000139:SSL routines::record layer failure) while SSL handshaking to upstream, client: 10.1.45.13, server: _, request: "GET /api/webhook HTTP/1.1", upstream: "https://10.1.33.14:7000/api/webhook", host: "argocd.mycompany.com"
{"time": "2024-05-22T07:23:10+00:00","opentelemetry_context_traceparent": "00-59ab8aeb89007e53bc5d7d1c3b4b0561-e5cb39185ea6d409-00","trace_id": "59ab8aeb89007e53bc5d7d1c3b4b0561","span_id": "e5cb39185ea6d409","remote_addr": "10.1.38.19","host": "argocd.mycompany.com","uri": "/api/webhook", "status": "502", "upstream_addr": "10.1.35.158:8080", "upstream_status": "502", "upstream_response_time": "0.005", "upstream_url": "https://10.1.35.158:8080"}

No error logs in apisix-ingress-controller container

Initialisation logs

2024-05-22T19:45:49+08:00	info	ingress/ingress.go:128	init apisix ingress controller
2024-05-22T19:45:49+08:00	info	ingress/ingress.go:129	version:
Version: 1.8.1
Git SHA: no-git-module
Go Version: go1.20.14
Building OS/Arch: linux/amd64
Running OS/Arch: linux/amd64

2024-05-22T19:45:49+08:00	info	ingress/ingress.go:139	use configuration
{
  "cert_file": "/etc/webhook/certs/cert.pem",
  "key_file": "/etc/webhook/certs/key.pem",
  "log_level": "info",
  "log_output": "stderr",
  "log_rotate_output_path": "",
  "log_rotation_max_size": 100,
  "log_rotation_max_age": 0,
  "log_rotation_max_backups": 0,
  "http_listen": ":8080",
  "https_listen": ":8443",
  "ingress_publish_service": "apisix/apisix-private-apisix-ingress-controller-apisix-gateway",
  "ingress_status_address": [],
  "enable_profiling": true,
  "kubernetes": {
    "kubeconfig": "",
    "resync_interval": "6h0m0s",
    "namespace_selector": [],
    "election_id": "ingress-apisix-leader",
    "ingress_class": "apisix-private",
    "ingress_version": "networking/v1",
    "watch_endpoint_slices": false,
    "api_version": "apisix.apache.org/v2",
    "enable_gateway_api": false,
    "disable_status_updates": false,
    "enable_admission": false
  },
  "apisix": {
    "admin_api_version": "v2",
    "default_cluster_name": "default",
    "default_cluster_base_url": "http://127.0.0.1:9180/apisix/admin",
    "default_cluster_admin_key": "******"
  },
  "apisix_resource_sync_interval": "1h0m0s",
  "apisix_resource_sync_comparison": true,
  "plugin_metadata_cm": "",
  "etcdserver": {
    "enabled": true,
    "prefix": "/apisix",
    "listen_address": ":12379",
    "ssl_key_encrypt_salt": "*******"
  }
}
2024-05-22T19:45:49+08:00	info	providers/controller.go:169	start leader election
I0522 19:45:49.160877       1 leaderelection.go:250] attempting to acquire leader lease apisix/ingress-apisix-leader...
2024-05-22T19:45:49+08:00	info	providers/controller.go:154	start api server
2024-05-22T19:45:49+08:00	warn	providers/controller.go:219	found a new leader apisix-public-apisix-ingress-controller-6b548bf57-l2mbm
2024-05-22T19:45:49+08:00	info	providers/controller.go:221	controller now is running as a candidate	{"namespace": "apisix", "pod": "apisix-private-apisix-ingress-controller-7b7b699b55-dzfc6"}
2024-05-22T19:45:49+08:00	info	providers/controller.go:386	controller tries to leading ...	{"namespace": "apisix", "pod": "apisix-private-apisix-ingress-controller-7b7b699b55-dzfc6"}
start etcd server
2024-05-22T19:45:49+08:00	info	providers/controller.go:430	creating controller
2024-05-22T19:45:49+08:00	info	adapter/server.go:123	register grpc gateway
2024-05-22T19:45:49+08:00	info	providers/controller.go:505	init namespaces
2024-05-22T19:45:49+08:00	info	providers/controller.go:512	wait for resource sync
2024-05-22T19:45:50+08:00	info	providers/controller.go:520	init providers
2024-05-22T19:45:50+08:00	info	providers/controller.go:532	try to run providers
2024-05-22T19:45:50+08:00	info	configmap/configmap.go:87	configmap controller started
2024-05-22T19:45:50+08:00	info	apisix/apisix_global_rule.go:65	ApisixGlobalRule controller started
2024-05-22T19:45:50+08:00	info	apisix/apisix_upstream.go:92	ApisixUpstream controller started
2024-05-22T19:45:50+08:00	info	apisix/apisix_route.go:104	ApisixRoute controller started
2024-05-22T19:45:50+08:00	info	endpoint/endpoint.go:70	endpoints controller started
2024-05-22T19:45:50+08:00	info	apisix/apisix_cluster_config.go:65	ApisixClusterConfig controller started
2024-05-22T19:45:50+08:00	info	apisix/apisix_consumer.go:69	ApisixConsumer controller started
2024-05-22T19:45:50+08:00	info	apisix/apisix_plugin_config.go:70	ApisixPluginConfig controller started
2024-05-22T19:45:50+08:00	info	k8s/secret.go:79	secret controller started
2024-05-22T19:45:50+08:00	info	adapter/etcd.go:147	created object{revision 11 15  <nil>} {key 15 0 /apisix/global_rules/88900b32 <nil>}
2024-05-22T19:45:50+08:00	info	adapter/etcd.go:147	created object{revision 11 16  <nil>} {key 15 0 /apisix/global_rules/776b4ce1 <nil>}
2024-05-22T19:45:50+08:00	info	adapter/etcd.go:147	created object{revision 11 17  <nil>} {key 15 0 /apisix/plugin_configs/6012b23f <nil>}
2024-05-22T19:45:50+08:00	info	ingress/ingress.go:85	ingress controller started
2024-05-22T19:45:50+08:00	info	apisix/apisix_tls.go:78	ApisixTls controller started
2024-05-22T19:45:50+08:00	info	adapter/etcd.go:147	created object{revision 11 18  <nil>} {key 15 0 /apisix/plugin_configs/b8217bfd <nil>}
2024-05-22T19:45:50+08:00	info	adapter/etcd.go:147	created object{revision 11 19  <nil>} {key 15 0 /apisix/upstreams/3aa1222b <nil>}
2024-05-22T19:45:51+08:00	info	adapter/etcd.go:147	created object{revision 11 20  <nil>} {key 15 0 /apisix/plugin_configs/55e5be08 <nil>}
2024-05-22T19:45:51+08:00	info	adapter/etcd.go:147	created object{revision 11 21  <nil>} {key 15 0 /apisix/plugin_configs/72f99ba8 <nil>}
2024-05-22T19:45:51+08:00	info	adapter/etcd.go:147	created object{revision 11 22  <nil>} {key 15 0 /apisix/plugin_configs/68847a54 <nil>}
2024-05-22T19:45:51+08:00	info	adapter/etcd.go:147	created object{revision 11 23  <nil>} {key 15 0 /apisix/routes/80bb6249 <nil>}
2024-05-22T19:45:51+08:00	info	adapter/etcd.go:147	created object{revision 11 24  <nil>} {key 15 0 /apisix/plugin_configs/139a164c <nil>}
2024-05-22T19:45:51+08:00	info	adapter/etcd.go:147	created object{revision 11 25  <nil>} {key 15 0 /apisix/upstreams/3cd561aa <nil>}
2024-05-22T19:45:51+08:00	info	adapter/etcd.go:147	created object{revision 11 26  <nil>} {key 15 0 /apisix/routes/5dd3745c <nil>}
2024-05-22T19:45:51+08:00	info	adapter/etcd.go:147	created object{revision 11 27  <nil>} {key 15 0 /apisix/upstreams/e9b16f05 <nil>}
2024-05-22T19:45:51+08:00	info	adapter/etcd.go:147	created object{revision 11 28  <nil>} {key 15 0 /apisix/routes/17669e15 <nil>}
2024-05-22T19:45:51+08:00	info	adapter/etcd.go:147	created object{revision 11 29  <nil>} {key 15 0 /apisix/upstreams/15633f64 <nil>}
2024-05-22T19:45:51+08:00	info	adapter/etcd.go:147	created object{revision 11 30  <nil>} {key 15 0 /apisix/routes/91a52f6c <nil>}
2024-05-22T19:45:51+08:00	info	adapter/etcd.go:147	created object{revision 11 31  <nil>} {key 15 0 /apisix/upstreams/8496ae <nil>}
2024-05-22T19:45:51+08:00	info	adapter/etcd.go:147	created object{revision 11 32  <nil>} {key 15 0 /apisix/routes/70b630c9 <nil>}
2024-05-22T19:45:51+08:00	info	adapter/etcd.go:147	created object{revision 11 33  <nil>} {key 15 0 /apisix/upstreams/5c0f40f9 <nil>}
2024-05-22T19:45:51+08:00	info	adapter/etcd.go:147	created object{revision 11 34  <nil>} {key 15 0 /apisix/routes/b97496fe <nil>}
2024-05-22T19:45:51+08:00	info	adapter/etcd.go:147	created object{revision 11 35  <nil>} {key 15 0 /apisix/upstreams/4845b0f5 <nil>}
2024-05-22T19:45:51+08:00	info	adapter/etcd.go:147	created object{revision 11 36  <nil>} {key 15 0 /apisix/routes/3cd74641 <nil>}

Steps to Reproduce

1. Install apisix ingress controller with its helm chart (version 1.8.1)
2. Apply any ingress resource attached to apisix

Environment

• APISIX Ingress controller version (run apisix-ingress-controller version --long)

 Version: 1.8.1
Git SHA: no-git-module
Go Version: go1.20.14
Building OS/Arch: linux/amd64
Running OS/Arch: linux/amd64

• Kubernetes cluster version (run kubectl version)

Client Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.10-eks-48e63af", GitCommit:"9176fb99b52f8d5ff73d67fea27f3a638f679f8a", GitTreeState:"clean", BuildDate:"2023-01-24T19:21:38Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"darwin/amd64"}
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.10-eks-48e63af", GitCommit:"9176fb99b52f8d5ff73d67fea27f3a638f679f8a", GitTreeState:"clean", BuildDate:"2023-01-24T19:21:38Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.4

• OS version if running APISIX Ingress controller in a bare-metal environment (run uname -a)

[github-actions]

This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.

[github-actions]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.