Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pod communication is broken when specify multiple root cert in mesh config #31987

Closed
irisdingbj opened this issue Apr 7, 2021 · 10 comments
Closed
Assignees
Labels
area/security lifecycle/automatically-closed Indicates a PR or issue that has been closed automatically. lifecycle/stale Indicates a PR or issue hasn't been manipulated by an Istio team member for a while

Comments

@irisdingbj
Copy link
Member

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: default
  meshConfig:
    caCertificates:
    - pem: |
        -----BEGIN CERTIFICATE-----
        MIID7TCCAtWgAwIBAgIJAOIRDhOcxsx6MA0GCSqGSIb3DQEBCwUAMIGLMQswCQYD
        VQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTESMBAGA1UEBwwJU3Vubnl2YWxl
        MQ4wDAYDVQQKDAVJc3RpbzENMAsGA1UECwwEVGVzdDEQMA4GA1UEAwwHUm9vdCBD
        QTEiMCAGCSqGSIb3DQEJARYTdGVzdHJvb3RjYUBpc3Rpby5pbzAgFw0xODAxMjQx
        OTE1NTFaGA8yMTE3MTIzMTE5MTU1MVowgYsxCzAJBgNVBAYTAlVTMRMwEQYDVQQI
        DApDYWxpZm9ybmlhMRIwEAYDVQQHDAlTdW5ueXZhbGUxDjAMBgNVBAoMBUlzdGlv
        MQ0wCwYDVQQLDARUZXN0MRAwDgYDVQQDDAdSb290IENBMSIwIAYJKoZIhvcNAQkB
        FhN0ZXN0cm9vdGNhQGlzdGlvLmlvMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
        CgKCAQEA38uEfAatzQYqbaLou1nxJ348VyNzumYMmDDt5pbLYRrCo2pS3ki1ZVDN
        8yxIENJFkpKw9UctTGdbNGuGCiSDP7uqF6BiVn+XKAU/3pnPFBbTd0S33NqbDEQu
        IYraHSl/tSk5rARbC1DrQRdZ6nYD2KrapC4g0XbjY6Pu5l4y7KnFwSunnp9uqpZw
        uERv/BgumJ5QlSeSeCmhnDhLxooG8w5tC2yVr1yDpsOHGimP/mc8Cds4V0zfIhQv
        YzfIHphhE9DKjmnjBYLOdj4aycv44jHnOGc+wvA1Jqsl60t3wgms+zJTiWwABLdw
        zgMAa7yxLyoV0+PiVQud6k+8ZoIFcwIDAQABo1AwTjAdBgNVHQ4EFgQUOUYGtUyh
        euxO4lGe4Op1y8NVoagwHwYDVR0jBBgwFoAUOUYGtUyheuxO4lGe4Op1y8NVoagw
        DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEANXLyfAs7J9rmBamGJvPZ
        ltx390WxzzLFQsBRAaH6rgeipBq3dR9qEjAwb6BTF+ROmtQzX+fjstCRrJxCto9W
        tC8KvXTdRfIjfCCZjhtIOBKqRxE4KJV/RBfv9xD5lyjtCPCQl3Ia6MSf42N+abAK
        WCdU6KCojA8WB9YhSCzza3aQbPTzd26OC/JblJpVgtus5f8ILzCsz+pbMimgTkhy
        AuhYRppJaQ24APijsEC9+GIaVKPg5IwWroiPoj+QXNpshuvqVQQXvGaRiq4zoSnx
        xAJz+w8tjrDWcf826VN14IL+/Cmqlg/rIfB5CHdwVIfWwpuGB66q/UiPegZMNs8a
        3g==
        -----END CERTIFICATE-----
    - pem: |
        -----BEGIN CERTIFICATE-----
        MIIFCTCCAvGgAwIBAgIJAL4uxHfykeWSMA0GCSqGSIb3DQEBBQUAMCIxDjAMBgNV
        BAoMBUlzdGlvMRAwDgYDVQQDDAdSb290IENBMB4XDTIxMDIxNzIyNDA1N1oXDTMx
        MDIxNTIyNDA1N1owIjEOMAwGA1UECgwFSXN0aW8xEDAOBgNVBAMMB1Jvb3QgQ0Ew
        ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCkESA1E1psP/v9wkdimcqZ
        X832eMRKomDxFFwbk9ayMF/XrGMAUmsvqeN9a73m5UD3MpArBiRc97XXzW1K1hnW
        sCtcN42C25NDXgHGjzyhplNogR6/SsKYg2oZx2iBRJUxwroi3/iTv7KPousQwGpF
        a/leoNxfr0+twbA5Y9nS17zO8CfJLlJz+c8MIbSdCTckcRxvVSXWsUlH1BJS/Bfh
        TnlaVqk/YGWBxhtm8BowB0hzaxFrQnwuxsRXgnFmlAV0iZ35jvrhM6vmU2RqvUUo
        BEgTTPuToC/2VRmyhFw/9cWcjzxgkvkjLsmVg5icuNvKQ4PgJL07zguRjk0XFchz
        SZuqimjDYSRQv3I0TOn+eT0b2KX8neg1pqh7w81YotyqFcJ7SdpQaau7CeMbus92
        P7XsCpCSVe82Y8BRcdtPgDEzn7AOA2IlgxDC1hex80+10aL8naWGdxxUEom8wQwS
        gvRHrdDsRigVvcygvVhfcoMak4RUxFeaQK5c1ruMlNvuuwZ20C4mUvZTvlaz7RmN
        yazzjqQYT4GHbR2e1kwBqe6YtlOrHY1Fpg5V6+S1rQkbbZrfQVQOXz7VQ7jOsmEr
        kNkrtgS8ZjAwgnOrf878Rr1g8Ac+I4q7Mpei2humdAydO3cEaGskcoozsxjPAKvd
        8be76nUjjkBv6eURp1ziEQIDAQABo0IwQDAdBgNVHQ4EFgQUPyNoAnWNHwP+2NFi
        zWLW0hz3Cw0wDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAuQwDQYJKoZI
        hvcNAQEFBQADggIBAIQx3aCt5GFuWxLLYlL2wbrO8tFoQnN4Poa/uli65YF47abb
        zZkDm6OomYIsWVce4tdoJZy1TLlyKZPb+MDDnelOzNhpljjpw2ZdhEtnv703513q
        o1zCgVrO1YWvk6Xv1gt3wVhQvJhq87BqrYFcCo899k09haXU4ddtP+YMPjyIngVb
        ucxML2xqjzS1Cfs+CD/OpwntISzWOEi5r/3IkbPlMT15hFa2oAVKBhOkyk0QQP8t
        bV9i4AC32gvshwIiGjbXUmnlRwBxUi8GBq5ZyR66nqoV9wBHPoqJZ3z+j6DNZSYm
        QGaO0wwWgSePRNPodzPAw6vofDjBe/hcyCk2d2uRrOLJICWbAdx76+j6h3zX2sPS
        FVSK1eVZaPUylL9rE+AyVGgl8/FqLNTwOHdSSovgIVVID7eXSpebnFtQEtlCSnik
        naaVSrG+sTH77WD9mQO9LmYS8JVceLE+ErSEAXkFKim131317sS5Z310U/L021M4
        xGH6zZHK9W9dx1X4gZKfoqGwSAHhs4rjEZCU7CKR1ouJBPWQ/cGrrk8n8ZdmKxmz
        OHNB4GteIEKJKrJTKQil8hsdSIqSUX4H4tw4GXlpyBSmZNt9iOjo4tWUGoUlQRIp
        QDpfEx1ep9pVDwQNGXVf+m9iqbc3DAiSN+1CGSZI5Kv0RzZSih5zIaxB2gJ7
        -----END CERTIFICATE-----
  • deploy httpbin and sleep into the same namespace ns1 with auto side car injeciton

  • sleep fails to call httpbin with below error:

curl  http://httpbin.ns1:8000/ip
upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 67108979:RSA routines:OPENSSL_internal:DATA_TOO_LARGE_FOR_MODULUS 184549382:X.509 certificate routines:OPENSSL_internal:public key routines 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED/
@irisdingbj
Copy link
Member Author

@shankgan suppose we should support define multiple pem in meshconfig like above? any suggestions to figure this issue out?

Below is some log snippet from the httpbin pod:

2021-04-07T08:28:45.882559Z	info	FLAG: --concurrency="2"
2021-04-07T08:28:45.882582Z	info	FLAG: --domain="ns1.svc.cluster.local"
2021-04-07T08:28:45.882587Z	info	FLAG: --help="false"
2021-04-07T08:28:45.882592Z	info	FLAG: --log_as_json="false"
2021-04-07T08:28:45.882594Z	info	FLAG: --log_caller=""
2021-04-07T08:28:45.882597Z	info	FLAG: --log_output_level="debug"
2021-04-07T08:28:45.882599Z	info	FLAG: --log_rotate=""
2021-04-07T08:28:45.882601Z	info	FLAG: --log_rotate_max_age="30"
2021-04-07T08:28:45.882604Z	info	FLAG: --log_rotate_max_backups="1000"
2021-04-07T08:28:45.882607Z	info	FLAG: --log_rotate_max_size="104857600"
2021-04-07T08:28:45.882612Z	info	FLAG: --log_stacktrace_level="default:none"
2021-04-07T08:28:45.882619Z	info	FLAG: --log_target="[stdout]"
2021-04-07T08:28:45.882624Z	info	FLAG: --meshConfig="./etc/istio/config/mesh"
2021-04-07T08:28:45.882628Z	info	FLAG: --outlierLogPath=""
2021-04-07T08:28:45.882631Z	info	FLAG: --proxyComponentLogLevel="misc:error"
2021-04-07T08:28:45.882635Z	info	FLAG: --proxyLogLevel="warning"
2021-04-07T08:28:45.882638Z	info	FLAG: --serviceCluster="httpbin.ns1"
2021-04-07T08:28:45.882642Z	info	FLAG: --stsPort="0"
2021-04-07T08:28:45.882644Z	info	FLAG: --templateFile=""
2021-04-07T08:28:45.882647Z	info	FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2021-04-07T08:28:45.882650Z	info	Version 1.10-dev-e7c3dcf3653c0bdb5458f055b94d8e049689ef7d-dirty-Modified
2021-04-07T08:28:45.882783Z	info	Proxy role	ips=[172.16.138.25] type=sidecar id=httpbin-74fb669cc6-blw76.ns1 domain=ns1.svc.cluster.local
2021-04-07T08:28:45.882834Z	info	Apply proxy config from env {}

2021-04-07T08:28:45.883900Z	info	Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: httpbin.ns1
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
  zipkin:
    address: zipkin.istio-system:9411

2021-04-07T08:28:45.883912Z	info	JWT policy is third-party-jwt
2021-04-07T08:28:45.883919Z	info	Pilot SAN: [istiod.istio-system.svc]
2021-04-07T08:28:45.883922Z	info	CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2021-04-07T08:28:45.883946Z	info	Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2021-04-07T08:28:45.884044Z	info	citadelclient	Citadel client using custom root cert: istiod.istio-system.svc:15012
2021-04-07T08:28:45.902869Z	info	ads	All caches have been synced up in 21.883896ms, marking server ready
2021-04-07T08:28:45.903126Z	info	sds	SDS server for workload certificates started, listening on "./etc/istio/proxy/SDS"
2021-04-07T08:28:45.903151Z	info	xdsproxy	Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"
2021-04-07T08:28:45.903203Z	info	sds	Start SDS grpc server
2021-04-07T08:28:45.903299Z	info	Starting proxy agent
2021-04-07T08:28:45.903314Z	info	Epoch 0 starting
2021-04-07T08:28:45.903313Z	info	Opening status port 15020
2021-04-07T08:28:45.953945Z	info	cache	generated new workload certificate	latency=50.857122ms ttl=23h59m59.046066308s
2021-04-07T08:28:45.953970Z	info	cache	Root cert has changed, start rotating root cert
2021-04-07T08:28:45.953984Z	info	ads	XDS: Incremental Pushing:0 ConnectedEndpoints:0 Version:
2021-04-07T08:28:45.954017Z	info	cache	returned workload trust anchor from cache	ttl=23h59m59.045985985s
2021-04-07T08:28:45.971178Z	info	Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --drain-strategy immediate --parent-shutdown-time-s 60 --service-cluster httpbin.ns1 --service-node sidecar~172.16.138.25~httpbin-74fb669cc6-blw76.ns1~ns1.svc.cluster.local --local-address-ip-version v4 --bootstrap-version 3 --disable-hot-restart --log-format %Y-%m-%dT%T.%fZ	%l	envoy %n	%v -l warning --component-log-level misc:error --concurrency 2]
2021-04-07T08:28:46.039168Z	info	xdsproxy	connected to upstream XDS server: istiod.istio-system.svc:15012
2021-04-07T08:28:46.101135Z	info	ads	ADS: new connection for node:sidecar~172.16.138.25~httpbin-74fb669cc6-blw76.ns1~ns1.svc.cluster.local-1
2021-04-07T08:28:46.101134Z	info	ads	ADS: new connection for node:sidecar~172.16.138.25~httpbin-74fb669cc6-blw76.ns1~ns1.svc.cluster.local-2
2021-04-07T08:28:46.101211Z	info	cache	returned workload certificate from cache	ttl=23h59m58.89879286s
2021-04-07T08:28:46.101228Z	info	cache	returned workload trust anchor from cache	ttl=23h59m58.898775041s
2021-04-07T08:28:46.101367Z	info	sds	SDS: PUSH	resource=default
2021-04-07T08:28:46.101451Z	info	sds	SDS: PUSH	resource=ROOTCA
2021-04-07T08:28:46.109491Z	debug	received new certificates to add to mesh trust domain: [-----BEGIN CERTIFICATE-----
MIID7TCCAtWgAwIBAgIJAOIRDhOcxsx6MA0GCSqGSIb3DQEBCwUAMIGLMQswCQYD
VQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTESMBAGA1UEBwwJU3Vubnl2YWxl
MQ4wDAYDVQQKDAVJc3RpbzENMAsGA1UECwwEVGVzdDEQMA4GA1UEAwwHUm9vdCBD
QTEiMCAGCSqGSIb3DQEJARYTdGVzdHJvb3RjYUBpc3Rpby5pbzAgFw0xODAxMjQx
OTE1NTFaGA8yMTE3MTIzMTE5MTU1MVowgYsxCzAJBgNVBAYTAlVTMRMwEQYDVQQI
DApDYWxpZm9ybmlhMRIwEAYDVQQHDAlTdW5ueXZhbGUxDjAMBgNVBAoMBUlzdGlv
MQ0wCwYDVQQLDARUZXN0MRAwDgYDVQQDDAdSb290IENBMSIwIAYJKoZIhvcNAQkB
FhN0ZXN0cm9vdGNhQGlzdGlvLmlvMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
CgKCAQEA38uEfAatzQYqbaLou1nxJ348VyNzumYMmDDt5pbLYRrCo2pS3ki1ZVDN
8yxIENJFkpKw9UctTGdbNGuGCiSDP7uqF6BiVn+XKAU/3pnPFBbTd0S33NqbDEQu
IYraHSl/tSk5rARbC1DrQRdZ6nYD2KrapC4g0XbjY6Pu5l4y7KnFwSunnp9uqpZw
uERv/BgumJ5QlSeSeCmhnDhLxooG8w5tC2yVr1yDpsOHGimP/mc8Cds4V0zfIhQv
YzfIHphhE9DKjmnjBYLOdj4aycv44jHnOGc+wvA1Jqsl60t3wgms+zJTiWwABLdw
zgMAa7yxLyoV0+PiVQud6k+8ZoIFcwIDAQABo1AwTjAdBgNVHQ4EFgQUOUYGtUyh
euxO4lGe4Op1y8NVoagwHwYDVR0jBBgwFoAUOUYGtUyheuxO4lGe4Op1y8NVoagw
DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEANXLyfAs7J9rmBamGJvPZ
ltx390WxzzLFQsBRAaH6rgeipBq3dR9qEjAwb6BTF+ROmtQzX+fjstCRrJxCto9W
tC8KvXTdRfIjfCCZjhtIOBKqRxE4KJV/RBfv9xD5lyjtCPCQl3Ia6MSf42N+abAK
WCdU6KCojA8WB9YhSCzza3aQbPTzd26OC/JblJpVgtus5f8ILzCsz+pbMimgTkhy
AuhYRppJaQ24APijsEC9+GIaVKPg5IwWroiPoj+QXNpshuvqVQQXvGaRiq4zoSnx
xAJz+w8tjrDWcf826VN14IL+/Cmqlg/rIfB5CHdwVIfWwpuGB66q/UiPegZMNs8a
3g==
-----END CERTIFICATE-----
 -----BEGIN CERTIFICATE-----
MIIFCTCCAvGgAwIBAgIJAL4uxHfykeWSMA0GCSqGSIb3DQEBBQUAMCIxDjAMBgNV
BAoMBUlzdGlvMRAwDgYDVQQDDAdSb290IENBMB4XDTIxMDIxNzIyNDA1N1oXDTMx
MDIxNTIyNDA1N1owIjEOMAwGA1UECgwFSXN0aW8xEDAOBgNVBAMMB1Jvb3QgQ0Ew
ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCkESA1E1psP/v9wkdimcqZ
X832eMRKomDxFFwbk9ayMF/XrGMAUmsvqeN9a73m5UD3MpArBiRc97XXzW1K1hnW
sCtcN42C25NDXgHGjzyhplNogR6/SsKYg2oZx2iBRJUxwroi3/iTv7KPousQwGpF
a/leoNxfr0+twbA5Y9nS17zO8CfJLlJz+c8MIbSdCTckcRxvVSXWsUlH1BJS/Bfh
TnlaVqk/YGWBxhtm8BowB0hzaxFrQnwuxsRXgnFmlAV0iZ35jvrhM6vmU2RqvUUo
BEgTTPuToC/2VRmyhFw/9cWcjzxgkvkjLsmVg5icuNvKQ4PgJL07zguRjk0XFchz
SZuqimjDYSRQv3I0TOn+eT0b2KX8neg1pqh7w81YotyqFcJ7SdpQaau7CeMbus92
P7XsCpCSVe82Y8BRcdtPgDEzn7AOA2IlgxDC1hex80+10aL8naWGdxxUEom8wQwS
gvRHrdDsRigVvcygvVhfcoMak4RUxFeaQK5c1ruMlNvuuwZ20C4mUvZTvlaz7RmN
yazzjqQYT4GHbR2e1kwBqe6YtlOrHY1Fpg5V6+S1rQkbbZrfQVQOXz7VQ7jOsmEr
kNkrtgS8ZjAwgnOrf878Rr1g8Ac+I4q7Mpei2humdAydO3cEaGskcoozsxjPAKvd
8be76nUjjkBv6eURp1ziEQIDAQABo0IwQDAdBgNVHQ4EFgQUPyNoAnWNHwP+2NFi
zWLW0hz3Cw0wDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAuQwDQYJKoZI
hvcNAQEFBQADggIBAIQx3aCt5GFuWxLLYlL2wbrO8tFoQnN4Poa/uli65YF47abb
zZkDm6OomYIsWVce4tdoJZy1TLlyKZPb+MDDnelOzNhpljjpw2ZdhEtnv703513q
o1zCgVrO1YWvk6Xv1gt3wVhQvJhq87BqrYFcCo899k09haXU4ddtP+YMPjyIngVb
ucxML2xqjzS1Cfs+CD/OpwntISzWOEi5r/3IkbPlMT15hFa2oAVKBhOkyk0QQP8t
bV9i4AC32gvshwIiGjbXUmnlRwBxUi8GBq5ZyR66nqoV9wBHPoqJZ3z+j6DNZSYm
QGaO0wwWgSePRNPodzPAw6vofDjBe/hcyCk2d2uRrOLJICWbAdx76+j6h3zX2sPS
FVSK1eVZaPUylL9rE+AyVGgl8/FqLNTwOHdSSovgIVVID7eXSpebnFtQEtlCSnik
naaVSrG+sTH77WD9mQO9LmYS8JVceLE+ErSEAXkFKim131317sS5Z310U/L021M4
xGH6zZHK9W9dx1X4gZKfoqGwSAHhs4rjEZCU7CKR1ouJBPWQ/cGrrk8n8ZdmKxmz
OHNB4GteIEKJKrJTKQil8hsdSIqSUX4H4tw4GXlpyBSmZNt9iOjo4tWUGoUlQRIp
QDpfEx1ep9pVDwQNGXVf+m9iqbc3DAiSN+1CGSZI5Kv0RzZSih5zIaxB2gJ7
-----END CERTIFICATE-----
 -----BEGIN CERTIFICATE-----
MIIFFDCCAvygAwIBAgIUDnmBxhDLiS368KiL7SjJbvKIf0owDQYJKoZIhvcNAQEL
BQAwIjEOMAwGA1UECgwFSXN0aW8xEDAOBgNVBAMMB1Jvb3QgQ0EwHhcNMjEwNDA3
MDEyODA1WhcNMjEwNDA4MDEyODA1WjAiMQ4wDAYDVQQKDAVJc3RpbzEQMA4GA1UE
AwwHUm9vdCBDQTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAN+nqJSi
d2eAzhRev2Nr8memipLFTWr3Rq+kEbhZ9R5nX2HmGSIzzkjPWfCOq8RJswYCw5RO
frh6xegt0WTp612L1h5V+0s20L8i5mPXv2x/Qu74mEAMIxlcd88Y8HLfjJCn3Dpt
CvYJz2jP/VXIZ944oUV67FX5gLhXzfm6YOnYb/+GcVGEaNPDZOCFd0epbOl/6qMt
/cyGshbfXf3/rWj7I0Y9nFDEhEVfH0V3dsHsr5Rdb9wPva0wt/j/MCT+mSvKH/vO
+AYY/skF+I/tA4ihbgodQJn+SNN1FNyDr0O365WwO+iG/Gse4pqpNZpY9iYtiJND
Cp6/WM3cQAawSQ7SqgptZga+na+RHazsshwboERVT8BjWstRkiNvghr6woaMRlj0
NEuxS2LyZiRE/nP/+v7SbvicWVMQmKMJiOIGIZSl5Uv/WTmhwp4Wok9LFgFJSw/D
QfXRrteQin+n5bzYGSdIzVt2GSYEB+ognDR2EEqBwaIltwcoXcJP6zIVreIGh4YG
h3aNxUl+jhpDl88t8T0z6gTPje2aFWSb8mEIEyEskcfUtJQ4myPXcrksqrOZw5zL
zcmdC/6UB7wKlEkJatIgoGm52sSI0J+G9aGPMtZItlYfOtw6lZfKEtF6pu9DJeg4
mLaClI8eNw2l1h4cymuh0UZo3hsAU6yn4NTFAgMBAAGjQjBAMB0GA1UdDgQWBBT1
TPsMy0ReR8mEdE87hnKwIxsOiDAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQE
AwIC5DANBgkqhkiG9w0BAQsFAAOCAgEArefraGmfYOXQm2gMoWbN074e1PdbItWl
n4S9DgyITMvw47T1qnod+2ZyMzQH7oxiHW95iTkE7672mBz2Cpmw/tkpI34qz0iv
hmyUaELhN8oGkXKAuvFLx16RqH2ABD71w1twFhqL09B/Hgu5s2W0mLGTqbY1EfNB
NsxaL+j5yRQT8y15s7/DmQBp5zbjjtpXpxixmHdltXd8ScfaaA+sJV7DP6TpRe2g
uvyt+95wJIx2GWZShfbDy3MHlnEq7iFiiSK7WPDkwE85m38ECnHUoZw/yLZr0jFL
3cKWEQdKugqc33HrcprEQL2GhP6VZrU/VfV2Ab4caVB856hdMACMTCoMrdf32WgZ
Gp9SMwHLxHq17qh+2WkuuzBStymRSzhmxnn16CRYJxPyAVye+hwf1LQz9TIwwQqy
mGL/n3aYJPBX8Ybjz/YdTitGEKUFwpnxuQq2WUnLSlZ4mVB/RUh2bTK2kYhsZnKZ
XKcRuu6K9+PpgGM5AfuIJTMqGoQxMULS8/9K1KZu4BkOwGUNogSk+JMXN73MrFTq
hveDAUbKWFNCtuG2iN5MtORmoedCNA7Zd626o11Q0MM1eh53NMCi/lF23MEkHr+R
+LT/YM29jA5j0pwWvPQRX72SarFVs2aaDXUAvQ5hMv0oexrfxPce8Mb7XTNxl+v7
mDsf9AabQss=
-----END CERTIFICATE-----
]
2021-04-07T08:28:46.109526Z	info	ads	XDS: Incremental Pushing:0 ConnectedEndpoints:2 Version:
2021-04-07T08:28:46.109723Z	info	cache	returned workload trust anchor from cache	ttl=23h59m58.8902872s
2021-04-07T08:28:46.109807Z	info	sds	SDS: PUSH	resource=ROOTCA
2021-04-07T08:28:47.391940Z	info	Initialization took 1.510545234s
2021-04-07T08:28:47.391977Z	info	Envoy proxy is ready

@irisdingbj
Copy link
Member Author

irisdingbj commented Apr 8, 2021

After the fix for duplicate certificates in #32032 it fails with below error:

upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 67108971:RSA routines:OPENSSL_internal:BLOCK_TYPE_IS_NOT_01 67109000:RSA routines:OPENSSL_internal:PADDING_CHECK_FAILED 184549382:X.509 certificate routines:OPENSSL_internal:public key routines 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

This seems be caused by the multi root we set in ROOTCA, besides the one coming from istio CA source, all others can not work.

@shankgan what is the expected usage method for multi root? any specific requirements for the root certs we defined in mesh config? Thanks.

@irisdingbj
Copy link
Member Author

@myidpt @howardjohn @hzxuzhonghu ^^^ in case you have some thoughts for this, thanks.

@hzxuzhonghu
Copy link
Member

I wish we should provide an integration test for this feature

@shankgan
Copy link
Contributor

@hzxuzhonghu - already did add an integration test. However, that adds only one additional root. (ca_custom_root). It appears in this case there is some padding mismatch in what x509 libarary supports vs what envoy supports. Will add this as an additional test.
@irisdingbj - It appears that your padding is incorrect routines:OPENSSL_internal:PADDING_CHECK_FAILED 184549382:X.509. Can you execute istioctl pc secret <pod-name> -o json and get the base64 encoded contents of the ROOTCA and decode to see what is going on here?

@irisdingbj
Copy link
Member Author

@shankgan this is the ROOTCA I decode from the pod:

-----BEGIN CERTIFICATE-----
MIIEYjCCAkqgAwIBAgIRAO1EsjaVVCMPzSBa6HSVGkkwDQYJKoZIhvcNAQELBQAw
PTEOMAwGA1UECgwFSXN0aW8xGDAWBgNVBAMMD0ludGVybWVkaWF0ZSBDQTERMA8G
A1UEBwwIY2x1c3RlcjEwHhcNMjEwNDA3MDc1MjI5WhcNMjEwNDA4MDc1MjI5WjAA
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAus9soDSxW+TXeFlxie07
ilpEW0QxNytCUHAWooqschildXQltYcEAECU5twvNWwqMrsC8ziACBRWxnwNFb5/
+9jw1FnAJQVhgEiHzWjn2yWIxbpUmy0SOdrsfl1B4lm+Xv13gnr+Q4j1cWay/IKR
AdjGLPmpAHnZe0pJQwS/MeTpTED2z4OoZPipcsnHDbe2ls9Q3ZJ4yAbxjZYdUSlR
4gAB5QU3NxlbydpSphvFcsj0fREfTNpZHTxuSQqY+s15TSNBsVLWtnm0fyGpCeDN
VaRyerGQtDlco25DhGtAOnLDdqCbQBrNO1LaY0ET253ZIF72LZuhpAsfZ8YPeYNr
WwIDAQABo4GZMIGWMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcD
AQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBTlWaxePQpWpMHJ
IVGUVTpjViINGTA2BgNVHREBAf8ELDAqhihzcGlmZmU6Ly9jbHVzdGVyLmxvY2Fs
L25zL25zMS9zYS9odHRwYmluMA0GCSqGSIb3DQEBCwUAA4ICAQBwaJkXTP999pQE
SCDZOo6ifZxqAODDATzYgrfsUPJjxk57voqfgE5LXLPTWlI2yi7DD10X6EpNyAHY
/21SSifHyP05ORW7t+iLrmAH2JiQVYBZdOICnbtK2VlaRVCjDqPYEzqEb5R0a5CV
XFPTbXsVwOBC/kEIPE/UM1Ifzmd7EE4mGUGmg68/gtRHHDEgNndEdUt+BvZC53sU
RvqWyD835z/ImkFOYRi9FmgjDP6+TU3XLpIpX2fE4e3J6EzVUp2InhkPJgiJAn+l
sazMdkJctbjGuMQFIY9K1dS0SH0Qb00eUQBdgVsEIGNDa08S1mQTRi0BvrQoEPma
NSZTKAnfc4MLm8qWK5Ic/PY4mHZ0fnNPRGpPGEWi2m2DIL76wgUyTUSGUxesKnPm
rQ0m0A1M2s2UDpBg9DCf0eGk5wbHYtb0JxtYkON97UxIxrA/KEGO1+MOAAwc5xZB
qhZIl1tpWGxhCYw8bdxS9O+4n0oPf/wSyx3JZgs8dyzugvvhJDLj2GqyU2atezvG
UFFxuOBWG4c+FBCPE/jjtpjrShHNvcn0j8Eu/LaVruSVxxJ4qmCsOCTOuk0iNBfX
uCtG53vscCAVRqwq+pc4rF/+W9YuBJBv9uk/wlYudpl7xtkrCxlaFidxeBTrwc5z
u6VNjp5NX07MrglWylYxvKroByLdcA==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFVjCCAz6gAwIBAgIUW5mQyeu9RzzxYXpUuvhdYgPFIJ0wDQYJKoZIhvcNAQEL
BQAwIjEOMAwGA1UECgwFSXN0aW8xEDAOBgNVBAMMB1Jvb3QgQ0EwHhcNMjEwNDA3
MDEyOTEwWhcNMjEwNDA4MDEyOTEwWjA9MQ4wDAYDVQQKDAVJc3RpbzEYMBYGA1UE
AwwPSW50ZXJtZWRpYXRlIENBMREwDwYDVQQHDAhjbHVzdGVyMTCCAiIwDQYJKoZI
hvcNAQEBBQADggIPADCCAgoCggIBAL6kD5sPyrBYgSpyhoC2dVbzf5ry91E9S/ap
9kD5KS4Xjvw0PrgeuvQEex843woEJCOvz+Kfx2xMTVms2vRtTXKTf5Zb+evIIUmK
dnMnhUIyP2KhQ5VNzP1Rq2E2Si2QY9lCmL0GixWa6N0ss/8//6gsSftx9R/ayXQ9
0w5qb/kzn7zIzmmZsYMUBRawy9bgQDC/38amfnNBcmhRBbWKfj89weOifnSW1aQv
xzVI3Km4AzT2D1KNTPIkQ/VQOEmzt+78t1+QN2cZwmb89NqU5yDyTIswVsgFE4J/
x72KNSQ4DTC3FpKO3pVbIoXl54aFop2MpZV0yTp20u1Y5H9gqSqdCPPhQ7esCa2R
k/+qBnao9wz1722+9AnxhoLaXQTqGg5grl/SBg0HuAdFzJu+5iBZVsuZqqT4q0/2
9nygOmoqTZz8dQqdXmcFohkJ1wKZj38gGmAsigb7+bBdn30etOZ7aD6Q13EaEifF
dEVLM7dDQ0BtLJHR6f/1pYStCK8whlW5dBtZMCdERW54E7z8WMQpxL1lzl0O2erO
BAKDEJNeZDfCJhQE5jbzc11wJ4pjUr/204Vhv6ETCkdxmZg/hHsDSwAKC1qKE3wi
kQIuQ+GrX0hHsYkpJmrfU5u/pLuNozr2JA6ZpaNqMEkXm38BumMBtTXmxrR7aJPJ
BoePU09lAgMBAAGjaTBnMB0GA1UdDgQWBBTlWaxePQpWpMHJIVGUVTpjViINGTAS
BgNVHRMBAf8ECDAGAQH/AgEAMA4GA1UdDwEB/wQEAwIC5DAiBgNVHREEGzAZghdp
c3Rpb2QuaXN0aW8tc3lzdGVtLnN2YzANBgkqhkiG9w0BAQsFAAOCAgEArnKpHxG4
g4kxA0aMF9tBnB6VE+zEmZJunvFDJGBaU2NCiWPHU6ozpgCIOz4b/NamaIDLHn6g
zOlTvtgj9GwMQ5zyNV3vkxv+6fLprpd4QnWE8026IGBH/j3XalmDc/+qBo26stYK
WS221Gbiq3KdHh583e3XheWxQLO495aCLBEZHIuvbfw8ozpdB+/MWJ0wn5DcCrH9
eK3b1PsFuwjWPCTf07MsS8EWOO0vngFNe++YNBod7QZLcODUOerE6WTyIU54y25O
s9R1m6dJlzdXjGwxF6m2606dcey7TQYR6h4+X1PNjqgPNbrBF+OUfXlHRUUzPcat
/afaJRkPfgTwKdveWjCDdnVXUNFh3jjtTXZAAW1vhv+H4d5mxCuvO3Lp3ix/rD6u
5sp2oDmXcW5dkg4CGkP1X2jHY9gcEmYBD8FQjsHjo2HY2mYRuSKZK9IAuuzGgXH3
rME+IBoY2jRWjnjtoJqUVLMdOEjNdctkHdQ7U+h3gjbFCveYTJqzesSEXZEr1uTV
wDQIg6n1sviS5o9uHUUTkkUyFWvRcPbERg4pwQQ5Mnk1bxuUuRGIRQXthlxdKAlr
VGtgjdj1f0+ASY+SBK7c5VVqVOmTWTKWU8KPbU9AwCb1OFkAiSCiLDf3UCSOf1QC
4JiufQ3dfdKgQFGMiYkCeZ0xWmtf011BfyE=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFFDCCAvygAwIBAgIUDnmBxhDLiS368KiL7SjJbvKIf0owDQYJKoZIhvcNAQEL
BQAwIjEOMAwGA1UECgwFSXN0aW8xEDAOBgNVBAMMB1Jvb3QgQ0EwHhcNMjEwNDA3
MDEyODA1WhcNMjEwNDA4MDEyODA1WjAiMQ4wDAYDVQQKDAVJc3RpbzEQMA4GA1UE
AwwHUm9vdCBDQTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAN+nqJSi
d2eAzhRev2Nr8memipLFTWr3Rq+kEbhZ9R5nX2HmGSIzzkjPWfCOq8RJswYCw5RO
frh6xegt0WTp612L1h5V+0s20L8i5mPXv2x/Qu74mEAMIxlcd88Y8HLfjJCn3Dpt
CvYJz2jP/VXIZ944oUV67FX5gLhXzfm6YOnYb/+GcVGEaNPDZOCFd0epbOl/6qMt
/cyGshbfXf3/rWj7I0Y9nFDEhEVfH0V3dsHsr5Rdb9wPva0wt/j/MCT+mSvKH/vO
+AYY/skF+I/tA4ihbgodQJn+SNN1FNyDr0O365WwO+iG/Gse4pqpNZpY9iYtiJND
Cp6/WM3cQAawSQ7SqgptZga+na+RHazsshwboERVT8BjWstRkiNvghr6woaMRlj0
NEuxS2LyZiRE/nP/+v7SbvicWVMQmKMJiOIGIZSl5Uv/WTmhwp4Wok9LFgFJSw/D
QfXRrteQin+n5bzYGSdIzVt2GSYEB+ognDR2EEqBwaIltwcoXcJP6zIVreIGh4YG
h3aNxUl+jhpDl88t8T0z6gTPje2aFWSb8mEIEyEskcfUtJQ4myPXcrksqrOZw5zL
zcmdC/6UB7wKlEkJatIgoGm52sSI0J+G9aGPMtZItlYfOtw6lZfKEtF6pu9DJeg4
mLaClI8eNw2l1h4cymuh0UZo3hsAU6yn4NTFAgMBAAGjQjBAMB0GA1UdDgQWBBT1
TPsMy0ReR8mEdE87hnKwIxsOiDAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQE
AwIC5DANBgkqhkiG9w0BAQsFAAOCAgEArefraGmfYOXQm2gMoWbN074e1PdbItWl
n4S9DgyITMvw47T1qnod+2ZyMzQH7oxiHW95iTkE7672mBz2Cpmw/tkpI34qz0iv
hmyUaELhN8oGkXKAuvFLx16RqH2ABD71w1twFhqL09B/Hgu5s2W0mLGTqbY1EfNB
NsxaL+j5yRQT8y15s7/DmQBp5zbjjtpXpxixmHdltXd8ScfaaA+sJV7DP6TpRe2g
uvyt+95wJIx2GWZShfbDy3MHlnEq7iFiiSK7WPDkwE85m38ECnHUoZw/yLZr0jFL
3cKWEQdKugqc33HrcprEQL2GhP6VZrU/VfV2Ab4caVB856hdMACMTCoMrdf32WgZ
Gp9SMwHLxHq17qh+2WkuuzBStymRSzhmxnn16CRYJxPyAVye+hwf1LQz9TIwwQqy
mGL/n3aYJPBX8Ybjz/YdTitGEKUFwpnxuQq2WUnLSlZ4mVB/RUh2bTK2kYhsZnKZ
XKcRuu6K9+PpgGM5AfuIJTMqGoQxMULS8/9K1KZu4BkOwGUNogSk+JMXN73MrFTq
hveDAUbKWFNCtuG2iN5MtORmoedCNA7Zd626o11Q0MM1eh53NMCi/lF23MEkHr+R
+LT/YM29jA5j0pwWvPQRX72SarFVs2aaDXUAvQ5hMv0oexrfxPce8Mb7XTNxl+v7
mDsf9AabQss=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFFDCCAvygAwIBAgIUDnmBxhDLiS368KiL7SjJbvKIf0owDQYJKoZIhvcNAQEL
BQAwIjEOMAwGA1UECgwFSXN0aW8xEDAOBgNVBAMMB1Jvb3QgQ0EwHhcNMjEwNDA3
MDEyODA1WhcNMjEwNDA4MDEyODA1WjAiMQ4wDAYDVQQKDAVJc3RpbzEQMA4GA1UE
AwwHUm9vdCBDQTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAN+nqJSi
d2eAzhRev2Nr8memipLFTWr3Rq+kEbhZ9R5nX2HmGSIzzkjPWfCOq8RJswYCw5RO
frh6xegt0WTp612L1h5V+0s20L8i5mPXv2x/Qu74mEAMIxlcd88Y8HLfjJCn3Dpt
CvYJz2jP/VXIZ944oUV67FX5gLhXzfm6YOnYb/+GcVGEaNPDZOCFd0epbOl/6qMt
/cyGshbfXf3/rWj7I0Y9nFDEhEVfH0V3dsHsr5Rdb9wPva0wt/j/MCT+mSvKH/vO
+AYY/skF+I/tA4ihbgodQJn+SNN1FNyDr0O365WwO+iG/Gse4pqpNZpY9iYtiJND
Cp6/WM3cQAawSQ7SqgptZga+na+RHazsshwboERVT8BjWstRkiNvghr6woaMRlj0
NEuxS2LyZiRE/nP/+v7SbvicWVMQmKMJiOIGIZSl5Uv/WTmhwp4Wok9LFgFJSw/D
QfXRrteQin+n5bzYGSdIzVt2GSYEB+ognDR2EEqBwaIltwcoXcJP6zIVreIGh4YG
h3aNxUl+jhpDl88t8T0z6gTPje2aFWSb8mEIEyEskcfUtJQ4myPXcrksqrOZw5zL
zcmdC/6UB7wKlEkJatIgoGm52sSI0J+G9aGPMtZItlYfOtw6lZfKEtF6pu9DJeg4
mLaClI8eNw2l1h4cymuh0UZo3hsAU6yn4NTFAgMBAAGjQjBAMB0GA1UdDgQWBBT1
TPsMy0ReR8mEdE87hnKwIxsOiDAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQE
AwIC5DANBgkqhkiG9w0BAQsFAAOCAgEArefraGmfYOXQm2gMoWbN074e1PdbItWl
n4S9DgyITMvw47T1qnod+2ZyMzQH7oxiHW95iTkE7672mBz2Cpmw/tkpI34qz0iv
hmyUaELhN8oGkXKAuvFLx16RqH2ABD71w1twFhqL09B/Hgu5s2W0mLGTqbY1EfNB
NsxaL+j5yRQT8y15s7/DmQBp5zbjjtpXpxixmHdltXd8ScfaaA+sJV7DP6TpRe2g
uvyt+95wJIx2GWZShfbDy3MHlnEq7iFiiSK7WPDkwE85m38ECnHUoZw/yLZr0jFL
3cKWEQdKugqc33HrcprEQL2GhP6VZrU/VfV2Ab4caVB856hdMACMTCoMrdf32WgZ
Gp9SMwHLxHq17qh+2WkuuzBStymRSzhmxnn16CRYJxPyAVye+hwf1LQz9TIwwQqy
mGL/n3aYJPBX8Ybjz/YdTitGEKUFwpnxuQq2WUnLSlZ4mVB/RUh2bTK2kYhsZnKZ
XKcRuu6K9+PpgGM5AfuIJTMqGoQxMULS8/9K1KZu4BkOwGUNogSk+JMXN73MrFTq
hveDAUbKWFNCtuG2iN5MtORmoedCNA7Zd626o11Q0MM1eh53NMCi/lF23MEkHr+R
+LT/YM29jA5j0pwWvPQRX72SarFVs2aaDXUAvQ5hMv0oexrfxPce8Mb7XTNxl+v7
mDsf9AabQss=
-----END CERTIFICATE-----

@shankgan
Copy link
Contributor

shankgan commented Apr 15, 2021

I tested a very similar config (derived from your config) and it worked for me. Note that the latest builds require you to explicitly enable the feature using flags in IstioOperator

apiVersion: install.istio.io/v1alpha1 
kind: IstioOperator
spec:
  values: 
    pilot: 
      env: 
        ISTIO_MULTIROOT_MESH: true 
  meshConfig:
    defaultConfig: 
      proxyMetadata: 
        PROXY_CONFIG_XDS_AGENT: "true" 
    caCertificates:
    - pem: |
        -----BEGIN CERTIFICATE-----
        MIID7TCCAtWgAwIBAgIJAOIRDhOcxsx6MA0GCSqGSIb3DQEBCwUAMIGLMQswCQYD
        VQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTESMBAGA1UEBwwJU3Vubnl2YWxl
        MQ4wDAYDVQQKDAVJc3RpbzENMAsGA1UECwwEVGVzdDEQMA4GA1UEAwwHUm9vdCBD
        QTEiMCAGCSqGSIb3DQEJARYTdGVzdHJvb3RjYUBpc3Rpby5pbzAgFw0xODAxMjQx
        OTE1NTFaGA8yMTE3MTIzMTE5MTU1MVowgYsxCzAJBgNVBAYTAlVTMRMwEQYDVQQI
        DApDYWxpZm9ybmlhMRIwEAYDVQQHDAlTdW5ueXZhbGUxDjAMBgNVBAoMBUlzdGlv
        MQ0wCwYDVQQLDARUZXN0MRAwDgYDVQQDDAdSb290IENBMSIwIAYJKoZIhvcNAQkB
        FhN0ZXN0cm9vdGNhQGlzdGlvLmlvMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
        CgKCAQEA38uEfAatzQYqbaLou1nxJ348VyNzumYMmDDt5pbLYRrCo2pS3ki1ZVDN
        8yxIENJFkpKw9UctTGdbNGuGCiSDP7uqF6BiVn+XKAU/3pnPFBbTd0S33NqbDEQu
        IYraHSl/tSk5rARbC1DrQRdZ6nYD2KrapC4g0XbjY6Pu5l4y7KnFwSunnp9uqpZw
        uERv/BgumJ5QlSeSeCmhnDhLxooG8w5tC2yVr1yDpsOHGimP/mc8Cds4V0zfIhQv
        YzfIHphhE9DKjmnjBYLOdj4aycv44jHnOGc+wvA1Jqsl60t3wgms+zJTiWwABLdw
        zgMAa7yxLyoV0+PiVQud6k+8ZoIFcwIDAQABo1AwTjAdBgNVHQ4EFgQUOUYGtUyh
        euxO4lGe4Op1y8NVoagwHwYDVR0jBBgwFoAUOUYGtUyheuxO4lGe4Op1y8NVoagw
        DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEANXLyfAs7J9rmBamGJvPZ
        ltx390WxzzLFQsBRAaH6rgeipBq3dR9qEjAwb6BTF+ROmtQzX+fjstCRrJxCto9W
        tC8KvXTdRfIjfCCZjhtIOBKqRxE4KJV/RBfv9xD5lyjtCPCQl3Ia6MSf42N+abAK
        WCdU6KCojA8WB9YhSCzza3aQbPTzd26OC/JblJpVgtus5f8ILzCsz+pbMimgTkhy
        AuhYRppJaQ24APijsEC9+GIaVKPg5IwWroiPoj+QXNpshuvqVQQXvGaRiq4zoSnx
        xAJz+w8tjrDWcf826VN14IL+/Cmqlg/rIfB5CHdwVIfWwpuGB66q/UiPegZMNs8a
        3g==
        -----END CERTIFICATE-----
    - pem: |
        -----BEGIN CERTIFICATE-----
        MIIFCTCCAvGgAwIBAgIJAL4uxHfykeWSMA0GCSqGSIb3DQEBBQUAMCIxDjAMBgNV
        BAoMBUlzdGlvMRAwDgYDVQQDDAdSb290IENBMB4XDTIxMDIxNzIyNDA1N1oXDTMx
        MDIxNTIyNDA1N1owIjEOMAwGA1UECgwFSXN0aW8xEDAOBgNVBAMMB1Jvb3QgQ0Ew
        ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCkESA1E1psP/v9wkdimcqZ
        X832eMRKomDxFFwbk9ayMF/XrGMAUmsvqeN9a73m5UD3MpArBiRc97XXzW1K1hnW
        sCtcN42C25NDXgHGjzyhplNogR6/SsKYg2oZx2iBRJUxwroi3/iTv7KPousQwGpF
        a/leoNxfr0+twbA5Y9nS17zO8CfJLlJz+c8MIbSdCTckcRxvVSXWsUlH1BJS/Bfh
        TnlaVqk/YGWBxhtm8BowB0hzaxFrQnwuxsRXgnFmlAV0iZ35jvrhM6vmU2RqvUUo
        BEgTTPuToC/2VRmyhFw/9cWcjzxgkvkjLsmVg5icuNvKQ4PgJL07zguRjk0XFchz
        SZuqimjDYSRQv3I0TOn+eT0b2KX8neg1pqh7w81YotyqFcJ7SdpQaau7CeMbus92
        P7XsCpCSVe82Y8BRcdtPgDEzn7AOA2IlgxDC1hex80+10aL8naWGdxxUEom8wQwS
        gvRHrdDsRigVvcygvVhfcoMak4RUxFeaQK5c1ruMlNvuuwZ20C4mUvZTvlaz7RmN
        yazzjqQYT4GHbR2e1kwBqe6YtlOrHY1Fpg5V6+S1rQkbbZrfQVQOXz7VQ7jOsmEr
        kNkrtgS8ZjAwgnOrf878Rr1g8Ac+I4q7Mpei2humdAydO3cEaGskcoozsxjPAKvd
        8be76nUjjkBv6eURp1ziEQIDAQABo0IwQDAdBgNVHQ4EFgQUPyNoAnWNHwP+2NFi
        zWLW0hz3Cw0wDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAuQwDQYJKoZI
        hvcNAQEFBQADggIBAIQx3aCt5GFuWxLLYlL2wbrO8tFoQnN4Poa/uli65YF47abb
        zZkDm6OomYIsWVce4tdoJZy1TLlyKZPb+MDDnelOzNhpljjpw2ZdhEtnv703513q
        o1zCgVrO1YWvk6Xv1gt3wVhQvJhq87BqrYFcCo899k09haXU4ddtP+YMPjyIngVb
        ucxML2xqjzS1Cfs+CD/OpwntISzWOEi5r/3IkbPlMT15hFa2oAVKBhOkyk0QQP8t
        bV9i4AC32gvshwIiGjbXUmnlRwBxUi8GBq5ZyR66nqoV9wBHPoqJZ3z+j6DNZSYm
        QGaO0wwWgSePRNPodzPAw6vofDjBe/hcyCk2d2uRrOLJICWbAdx76+j6h3zX2sPS
        FVSK1eVZaPUylL9rE+AyVGgl8/FqLNTwOHdSSovgIVVID7eXSpebnFtQEtlCSnik
        naaVSrG+sTH77WD9mQO9LmYS8JVceLE+ErSEAXkFKim131317sS5Z310U/L021M4
        xGH6zZHK9W9dx1X4gZKfoqGwSAHhs4rjEZCU7CKR1ouJBPWQ/cGrrk8n8ZdmKxmz
        OHNB4GteIEKJKrJTKQil8hsdSIqSUX4H4tw4GXlpyBSmZNt9iOjo4tWUGoUlQRIp
        QDpfEx1ep9pVDwQNGXVf+m9iqbc3DAiSN+1CGSZI5Kv0RzZSih5zIaxB2gJ7
        -----END CERTIFICATE-----

Following the code given in this example, I created pods in two different namespaces and ensured communication between them works

kubectl exec "$(kubectl get pod -l app=sleep -n foo -o jsonpath={.items..metadata.name})" -c sleep -n foo -- curl http://httpbin.bar:8000/ip -s -o /dev/null -w "%{http_code}\n"
200

@irisdingbj
Copy link
Member Author

@shankgan Have you pluggin cert before you install istio? ( https://istio.io/latest/docs/tasks/security/cert-management/plugin-ca-cert/). Then using above cert in mesh config, You will get pod communication broken issue.

If you use self-signed ca certs for isito, it works fine.

@shankgan
Copy link
Contributor

shankgan commented Apr 16, 2021

Tried with plugin CA certs. Pod - Pod communication appears to work just fine. The CA certs below contain the plugin CA cert.

@irisdingbj - I wonder if there is some misconfiguration of your meshConfig. Perhaps an indent that is incorrect or a LACK of a newline char?

-----BEGIN CERTIFICATE-----
MIID7TCCAtWgAwIBAgIJAOIRDhOcxsx6MA0GCSqGSIb3DQEBCwUAMIGLMQswCQYD
VQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTESMBAGA1UEBwwJU3Vubnl2YWxl
MQ4wDAYDVQQKDAVJc3RpbzENMAsGA1UECwwEVGVzdDEQMA4GA1UEAwwHUm9vdCBD
QTEiMCAGCSqGSIb3DQEJARYTdGVzdHJvb3RjYUBpc3Rpby5pbzAgFw0xODAxMjQx
OTE1NTFaGA8yMTE3MTIzMTE5MTU1MVowgYsxCzAJBgNVBAYTAlVTMRMwEQYDVQQI
DApDYWxpZm9ybmlhMRIwEAYDVQQHDAlTdW5ueXZhbGUxDjAMBgNVBAoMBUlzdGlv
MQ0wCwYDVQQLDARUZXN0MRAwDgYDVQQDDAdSb290IENBMSIwIAYJKoZIhvcNAQkB
FhN0ZXN0cm9vdGNhQGlzdGlvLmlvMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
CgKCAQEA38uEfAatzQYqbaLou1nxJ348VyNzumYMmDDt5pbLYRrCo2pS3ki1ZVDN
8yxIENJFkpKw9UctTGdbNGuGCiSDP7uqF6BiVn+XKAU/3pnPFBbTd0S33NqbDEQu
IYraHSl/tSk5rARbC1DrQRdZ6nYD2KrapC4g0XbjY6Pu5l4y7KnFwSunnp9uqpZw
uERv/BgumJ5QlSeSeCmhnDhLxooG8w5tC2yVr1yDpsOHGimP/mc8Cds4V0zfIhQv
YzfIHphhE9DKjmnjBYLOdj4aycv44jHnOGc+wvA1Jqsl60t3wgms+zJTiWwABLdw
zgMAa7yxLyoV0+PiVQud6k+8ZoIFcwIDAQABo1AwTjAdBgNVHQ4EFgQUOUYGtUyh
euxO4lGe4Op1y8NVoagwHwYDVR0jBBgwFoAUOUYGtUyheuxO4lGe4Op1y8NVoagw
DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEANXLyfAs7J9rmBamGJvPZ
ltx390WxzzLFQsBRAaH6rgeipBq3dR9qEjAwb6BTF+ROmtQzX+fjstCRrJxCto9W
tC8KvXTdRfIjfCCZjhtIOBKqRxE4KJV/RBfv9xD5lyjtCPCQl3Ia6MSf42N+abAK
WCdU6KCojA8WB9YhSCzza3aQbPTzd26OC/JblJpVgtus5f8ILzCsz+pbMimgTkhy
AuhYRppJaQ24APijsEC9+GIaVKPg5IwWroiPoj+QXNpshuvqVQQXvGaRiq4zoSnx
xAJz+w8tjrDWcf826VN14IL+/Cmqlg/rIfB5CHdwVIfWwpuGB66q/UiPegZMNs8a
3g==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFCTCCAvGgAwIBAgIJAKPdJXONP0kXMA0GCSqGSIb3DQEBBQUAMCIxDjAMBgNV
BAoMBUlzdGlvMRAwDgYDVQQDDAdSb290IENBMB4XDTIxMDQxNjAyNDMzMVoXDTMx
MDQxNDAyNDMzMVowIjEOMAwGA1UECgwFSXN0aW8xEDAOBgNVBAMMB1Jvb3QgQ0Ew
ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCxtJoY3NehyaDzP4OvH0r0
c7I3uAPZ62Z0cWS8gt/Ig/fvstt84FQ/QA/hoW2YTJexu2UvLbQJrdkpKceikZHG
xHfDf4qTJ+0hXBfmeLl7AhAiqMzoPKU8X6/aSggQH8uA1lp7WUUCLjBZ23HGbrh0
RywPlD1ipEfhJpKE85uWHLt8kJrC56BbXi6VRzqk5bUEu+duJPK7huNeky/py9+h
ICHvM1bHqjL5erOf21q9dQBj3A4Ug9Dnd81eVxWcjHKS6lkNOREkxkRaqMjivHpL
UKMoFtTXgfuVnRIRRWVTCjE+r7YiKcvuTm+7UB+ulMZu/pcWb3/Mmiu4ZuY19Bx7
Udv/SUzUCRROEmZrogQHDeflntV4iRofn/wdNx19f+FsRQ6BgvQfsUf1YgI0G2vt
Q+92duPWwV3OIgsrUODhccGlirJYJYHSIA32rYeqO9q9vnjynMMnEinr350XILDC
iSNPcs1A2SoKricxPj79P/UhduBJ72S/1Fof73AeQ6MMBYFNqorG0kCJLl8tHS8C
H27GDPzxD3s5ew64ukimUSyXI9yabVlOY4GYUbWqVl51XpIvFExkq0eTLpR+veTJ
L4IuqRXOGW1NREBtsrwjSGFWb0mNetdiOU9oYEc/UwHoUYMpGIs8QM7VwinWemdU
mEO9do7DiLGLCO7Vk9gqrQIDAQABo0IwQDAdBgNVHQ4EFgQUuZBGb+NQq8HvfAD5
BRDg5Cq4p0kwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAuQwDQYJKoZI
hvcNAQEFBQADggIBAFyk/QXeWRmvcUNN5xvXr9iFudmy6Sc6PmKR5HAMl2QgGrvF
OlBD/myxxrP9+biBRHaXHy0Zc63DWPq4sXmdOdEi7jUse4iYFEi2oqmZs1Q86rlu
/p4ezAwcMaGSkniq7xZk2IgMtoLmGU5y3jp7iPGZdHD4L1Tpiq5n/mzDRqd3XGwO
BjU75uvcFWKnFr11u1yfTaSpWs/ufmfomzKIyVh4ZBi0xRC9NQD12MTdaBl5mOtX
lTtbt/NqwNjWuSXYnf5tKPlZb1ZLtuaLfz0NqU3aFgw+bKDzx6RRB+L+cuJ2KgS8
thP0dI3HvloOOtcUVy9sgb0RIs2uIZe1Oh2T40qYNknfEmgOtvGDerfIZwcYXb3F
mpGn5JX0HHg/PTCSNOtR2p6U2WM/hEfyR+1gwSkYkjGzukfAy7WSia24tYUDaP+3
mRjT6ztfuVL6Qqmwoe9BImfgIYQEhU6scCG4fX7D5jEl+8vbhNcBaWfCPjJzXNCu
t4Vlb5aDrxmcMSKazOUew+f08cO8P7H5KLNrI5AIql+PKZ84oSe4a8xQLwN5JUHu
TmzAChnaGUTXzsr/rYLL1aCPwo/v01FfJuhVsUdYa7q0maGCb4UnhhGCLJrr8UvC
XpMS4lojk/6P3dkx/CMGR+y5xYASq6VxJG7nKG6+jQU469zZMjASBgp4cLd4
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFCTCCAvGgAwIBAgIJAL4uxHfykeWSMA0GCSqGSIb3DQEBBQUAMCIxDjAMBgNV
BAoMBUlzdGlvMRAwDgYDVQQDDAdSb290IENBMB4XDTIxMDIxNzIyNDA1N1oXDTMx
MDIxNTIyNDA1N1owIjEOMAwGA1UECgwFSXN0aW8xEDAOBgNVBAMMB1Jvb3QgQ0Ew
ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCkESA1E1psP/v9wkdimcqZ
X832eMRKomDxFFwbk9ayMF/XrGMAUmsvqeN9a73m5UD3MpArBiRc97XXzW1K1hnW
sCtcN42C25NDXgHGjzyhplNogR6/SsKYg2oZx2iBRJUxwroi3/iTv7KPousQwGpF
a/leoNxfr0+twbA5Y9nS17zO8CfJLlJz+c8MIbSdCTckcRxvVSXWsUlH1BJS/Bfh
TnlaVqk/YGWBxhtm8BowB0hzaxFrQnwuxsRXgnFmlAV0iZ35jvrhM6vmU2RqvUUo
BEgTTPuToC/2VRmyhFw/9cWcjzxgkvkjLsmVg5icuNvKQ4PgJL07zguRjk0XFchz
SZuqimjDYSRQv3I0TOn+eT0b2KX8neg1pqh7w81YotyqFcJ7SdpQaau7CeMbus92
P7XsCpCSVe82Y8BRcdtPgDEzn7AOA2IlgxDC1hex80+10aL8naWGdxxUEom8wQwS
gvRHrdDsRigVvcygvVhfcoMak4RUxFeaQK5c1ruMlNvuuwZ20C4mUvZTvlaz7RmN
yazzjqQYT4GHbR2e1kwBqe6YtlOrHY1Fpg5V6+S1rQkbbZrfQVQOXz7VQ7jOsmEr
kNkrtgS8ZjAwgnOrf878Rr1g8Ac+I4q7Mpei2humdAydO3cEaGskcoozsxjPAKvd
8be76nUjjkBv6eURp1ziEQIDAQABo0IwQDAdBgNVHQ4EFgQUPyNoAnWNHwP+2NFi
zWLW0hz3Cw0wDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAuQwDQYJKoZI
hvcNAQEFBQADggIBAIQx3aCt5GFuWxLLYlL2wbrO8tFoQnN4Poa/uli65YF47abb
zZkDm6OomYIsWVce4tdoJZy1TLlyKZPb+MDDnelOzNhpljjpw2ZdhEtnv703513q
o1zCgVrO1YWvk6Xv1gt3wVhQvJhq87BqrYFcCo899k09haXU4ddtP+YMPjyIngVb
ucxML2xqjzS1Cfs+CD/OpwntISzWOEi5r/3IkbPlMT15hFa2oAVKBhOkyk0QQP8t
bV9i4AC32gvshwIiGjbXUmnlRwBxUi8GBq5ZyR66nqoV9wBHPoqJZ3z+j6DNZSYm
QGaO0wwWgSePRNPodzPAw6vofDjBe/hcyCk2d2uRrOLJICWbAdx76+j6h3zX2sPS
FVSK1eVZaPUylL9rE+AyVGgl8/FqLNTwOHdSSovgIVVID7eXSpebnFtQEtlCSnik
naaVSrG+sTH77WD9mQO9LmYS8JVceLE+ErSEAXkFKim131317sS5Z310U/L021M4
xGH6zZHK9W9dx1X4gZKfoqGwSAHhs4rjEZCU7CKR1ouJBPWQ/cGrrk8n8ZdmKxmz
OHNB4GteIEKJKrJTKQil8hsdSIqSUX4H4tw4GXlpyBSmZNt9iOjo4tWUGoUlQRIp
QDpfEx1ep9pVDwQNGXVf+m9iqbc3DAiSN+1CGSZI5Kv0RzZSih5zIaxB2gJ7
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFCTCCAvGgAwIBAgIJAKPdJXONP0kXMA0GCSqGSIb3DQEBBQUAMCIxDjAMBgNV
BAoMBUlzdGlvMRAwDgYDVQQDDAdSb290IENBMB4XDTIxMDQxNjAyNDMzMVoXDTMx
MDQxNDAyNDMzMVowIjEOMAwGA1UECgwFSXN0aW8xEDAOBgNVBAMMB1Jvb3QgQ0Ew
ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCxtJoY3NehyaDzP4OvH0r0
c7I3uAPZ62Z0cWS8gt/Ig/fvstt84FQ/QA/hoW2YTJexu2UvLbQJrdkpKceikZHG
xHfDf4qTJ+0hXBfmeLl7AhAiqMzoPKU8X6/aSggQH8uA1lp7WUUCLjBZ23HGbrh0
RywPlD1ipEfhJpKE85uWHLt8kJrC56BbXi6VRzqk5bUEu+duJPK7huNeky/py9+h
ICHvM1bHqjL5erOf21q9dQBj3A4Ug9Dnd81eVxWcjHKS6lkNOREkxkRaqMjivHpL
UKMoFtTXgfuVnRIRRWVTCjE+r7YiKcvuTm+7UB+ulMZu/pcWb3/Mmiu4ZuY19Bx7
Udv/SUzUCRROEmZrogQHDeflntV4iRofn/wdNx19f+FsRQ6BgvQfsUf1YgI0G2vt
Q+92duPWwV3OIgsrUODhccGlirJYJYHSIA32rYeqO9q9vnjynMMnEinr350XILDC
iSNPcs1A2SoKricxPj79P/UhduBJ72S/1Fof73AeQ6MMBYFNqorG0kCJLl8tHS8C
H27GDPzxD3s5ew64ukimUSyXI9yabVlOY4GYUbWqVl51XpIvFExkq0eTLpR+veTJ
L4IuqRXOGW1NREBtsrwjSGFWb0mNetdiOU9oYEc/UwHoUYMpGIs8QM7VwinWemdU
mEO9do7DiLGLCO7Vk9gqrQIDAQABo0IwQDAdBgNVHQ4EFgQUuZBGb+NQq8HvfAD5
BRDg5Cq4p0kwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAuQwDQYJKoZI
hvcNAQEFBQADggIBAFyk/QXeWRmvcUNN5xvXr9iFudmy6Sc6PmKR5HAMl2QgGrvF
OlBD/myxxrP9+biBRHaXHy0Zc63DWPq4sXmdOdEi7jUse4iYFEi2oqmZs1Q86rlu
/p4ezAwcMaGSkniq7xZk2IgMtoLmGU5y3jp7iPGZdHD4L1Tpiq5n/mzDRqd3XGwO
BjU75uvcFWKnFr11u1yfTaSpWs/ufmfomzKIyVh4ZBi0xRC9NQD12MTdaBl5mOtX
lTtbt/NqwNjWuSXYnf5tKPlZb1ZLtuaLfz0NqU3aFgw+bKDzx6RRB+L+cuJ2KgS8
thP0dI3HvloOOtcUVy9sgb0RIs2uIZe1Oh2T40qYNknfEmgOtvGDerfIZwcYXb3F
mpGn5JX0HHg/PTCSNOtR2p6U2WM/hEfyR+1gwSkYkjGzukfAy7WSia24tYUDaP+3
mRjT6ztfuVL6Qqmwoe9BImfgIYQEhU6scCG4fX7D5jEl+8vbhNcBaWfCPjJzXNCu
t4Vlb5aDrxmcMSKazOUew+f08cO8P7H5KLNrI5AIql+PKZ84oSe4a8xQLwN5JUHu
TmzAChnaGUTXzsr/rYLL1aCPwo/v01FfJuhVsUdYa7q0maGCb4UnhhGCLJrr8UvC
XpMS4lojk/6P3dkx/CMGR+y5xYASq6VxJG7nKG6+jQU469zZMjASBgp4cLd4
-----END CERTIFICATE-----

@istio-policy-bot istio-policy-bot added the lifecycle/stale Indicates a PR or issue hasn't been manipulated by an Istio team member for a while label Jul 15, 2021
@istio-policy-bot
Copy link

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2021-04-16. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/security lifecycle/automatically-closed Indicates a PR or issue that has been closed automatically. lifecycle/stale Indicates a PR or issue hasn't been manipulated by an Istio team member for a while
Projects
None yet
Development

No branches or pull requests

5 participants