Unable to get GCS backend working #3741

day0ops · 2019-02-08T00:24:01Z

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.): GCS, ExternalName

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT
NGINX Ingress controller version: 0.22.0
Kubernetes version (use kubectl version): 1.11.5

Environment:

Cloud provider or hardware configuration: EKS
OS (e.g. from /etc/os-release): Debian GNU/Linux buster/sid
Kernel (e.g. uname -a): Linux 4.14.88-88.76.amzn2.x86_64
Install tools: Helm

What happened:
Trying to setup the proxy to GCS storage however running into a problem in EKS. Looked at #1809 as well.
So in EKS, I have Nginx ingress running and trying to proxy the requests to .well-known/assetlinks.json where .well-known/assetlinks.json is stored in a GCS bucket test-api.xxx.co.

Ingress,

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/backend-protocol: HTTP
    nginx.ingress.kubernetes.io/enable-rewrite-log: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /test-api.xxx.co/.well-known
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/upstream-vhost: storage.googleapis.com
  labels:
    app: mobile-universal-links
    chart: mobile-universal-links-0.1.0
    heritage: Tiller
    release: apps
  name: apps-mobile-universal-links
  namespace: apps
spec:
  rules:
  - host: test-api.xxx.co
    http:
      paths:
      - backend:
          serviceName: apps-mobile-universal-links
          servicePort: 80
        path: /.well-known/
  tls:
  - hosts:
    - test-api.xxx.co
    secretName: apps-mobile-universal-links-tls
status:
  loadBalancer:
    ingress:
    - hostname: xxx

Service,

apiVersion: v1
kind: Service
metadata:
  labels:
    app: mobile-universal-links
    chart: mobile-universal-links-0.1.0
    heritage: Tiller
    release: apps
  name: apps-mobile-universal-links
  namespace: apps
spec:
  externalName: storage.googleapis.com
  sessionAffinity: None
  type: ExternalName
status:
  loadBalancer: {}

If I curl the endpoint curl -iv https://test-api.xxx.co/.well-known/assetlinks.json I keep getting,

<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>Anonymous caller does not have storage.objects.get access to test-api.xxx.co/.well-known.</Details></Error>

And the logs,

2019/02/08 00:20:37 [notice] 486#486: *274726 "(?i)/.well-known/" matches "/.well-known/assetlinks.json", client: xxx.xxx.xxx.xxx, server: test-api.xxx.co, request: "GET /.well-known/assetlinks.json HTTP/1.1", host: "test-api.xxx.co"
2019/02/08 00:20:37 [notice] 486#486: *274726 rewritten data: "/test-api.xxx.co/.well-known", args: "", client: xxx.xxx.xxx.xxx, server: test-api.xxx.co, request: "GET /.well-known/assetlinks.json HTTP/1.1", host: "test-api.xxx.co"
xxx.xxx.xxx.xxx - [xxx.xxx.xxx.xxx] - - [08/Feb/2019:00:20:37 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 403 225 "-" "curl/7.54.0" 356 0.035 [apps-apps-mobile-universal-links-80] 74.125.193.128:80 225 0.032 403 8947aba6bf6b6e5b0f4c9f3969b044a3

FYI, this bucket is publicly available. I have tested it directly in GCS with curl -iv https://storage.googleapis.com/test-api.xxx.co/.well-known/assetlinks.json

If I use nginx.ingress.kubernetes.io/backend-protocol: HTTPS I get the same 403 back from GCS.

What you expected to happen:
Bucket to be available for proxying via Nginx ingress.

How to reproduce it (as minimally and precisely as possible):
As described above I have given the ingress config.

Anything else we need to know:

The text was updated successfully, but these errors were encountered:

aledbf · 2019-02-08T01:16:17Z

@nixgadget please use the configuration snippet to make sure SSLv3 is not used

nginx.ingress.kubernetes.io/configuration-snippet: |
  proxy_ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

day0ops · 2019-02-08T01:30:19Z

@aledbf I thought so too at first. But doesnt seem to be the case.

*   Trying xxx.xxx.xxx.xxx...
* TCP_NODELAY set
* Connected to test-api.xxx.co (xxx.xxx.xxx.xxx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: OU=Domain Control Validated; OU=PositiveSSL Multi-Domain; CN=sni220244.cloudflaressl.com
*  start date: Jan 11 00:00:00 2019 GMT
*  expire date: Jul 20 23:59:59 2019 GMT
*  subjectAltName: host "test-api.xxx.co" matched cert's "*.xxx.co"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=COMODO ECC Domain Validation Secure Server CA 2
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fe349808400)
> GET /.well-known/assetlinks.json HTTP/2
> Host: test-api.xxx.co
> User-Agent: curl/7.54.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 403
HTTP/2 403
< date: Fri, 08 Feb 2019 01:27:37 GMT
date: Fri, 08 Feb 2019 01:27:37 GMT
< content-type: application/xml; charset=UTF-8
content-type: application/xml; charset=UTF-8
< content-length: 225
content-length: 225
< set-cookie: __cfduid=df44a4d1371a4e9486e23d7ddcbdd1c931549589256; expires=Sat, 08-Feb-20 01:27:36 GMT; path=/; domain=.xxx.co; HttpOnly
set-cookie: __cfduid=df44a4d1371a4e9486e23d7ddcbdd1c931549589256; expires=Sat, 08-Feb-20 01:27:36 GMT; path=/; domain=.xxx.co; HttpOnly
< x-guploader-uploadid: AEnB2UoMpk-6P6Hlucgzt97vgUBEVdOJb-IxE_q2u_P6NkpyurLzullrWCWgOBg4dQy7vDWQoQSGFE3IpJ6wH9Hzm1ZiQpxWlQ
x-guploader-uploadid: AEnB2UoMpk-6P6Hlucgzt97vgUBEVdOJb-IxE_q2u_P6NkpyurLzullrWCWgOBg4dQy7vDWQoQSGFE3IpJ6wH9Hzm1ZiQpxWlQ
< expires: Fri, 08 Feb 2019 01:27:37 GMT
expires: Fri, 08 Feb 2019 01:27:37 GMT
< cache-control: private, max-age=0
cache-control: private, max-age=0
< alt-svc: quic=":443"; ma=2592000; v="44,43,39"
alt-svc: quic=":443"; ma=2592000; v="44,43,39"
< strict-transport-security: max-age=15724800; includeSubDomains
strict-transport-security: max-age=15724800; includeSubDomains
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< server: cloudflare
server: cloudflare
< cf-ray: 4a5a50901cd6a53c-NRT
cf-ray: 4a5a50901cd6a53c-NRT

<
* Connection #0 to host test-api.xxx.co left intact
<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>Anonymous caller does not have storage.objects.get access to test-api.xxx.co/.well-known.</Details></Error>%

aledbf · 2019-02-08T01:49:06Z

AccessDeniedAccess denied.
Anonymous caller does not have storage.objects.get access to test-api.xxx.co/.well-known.
%

Just in case, the bucket must be public

day0ops · 2019-02-08T01:54:15Z

@aledbf Yup bucket is publicly accessible.
Incase if you are interested.

> curl -iv https://storage.googleapis.com/test-api.xxx.co/.well-known/assetlinks.json

*   Trying 172.217.167.80...
* TCP_NODELAY set
* Connected to storage.googleapis.com (172.217.167.80) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=*.storage.googleapis.com
*  start date: Jan 23 09:15:00 2019 GMT
*  expire date: Apr 17 09:15:00 2019 GMT
*  subjectAltName: host "storage.googleapis.com" matched cert's "*.googleapis.com"
*  issuer: C=US; O=Google Trust Services; CN=Google Internet Authority G3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fbea1009600)
> GET /test-api.xxx.co/.well-known/assetlinks.json HTTP/2
> Host: storage.googleapis.com
> User-Agent: curl/7.54.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
HTTP/2 200
< x-guploader-uploadid: AEnB2Upwlj_ShLRO1JhvEdmnqD6w6M1WOTpee2mtAvqz9d-4cYcPtQHqh0m-tTc6U_OpnIftHr86PQjMsYVUxuevhr4a0YRVEg
x-guploader-uploadid: AEnB2Upwlj_ShLRO1JhvEdmnqD6w6M1WOTpee2mtAvqz9d-4cYcPtQHqh0m-tTc6U_OpnIftHr86PQjMsYVUxuevhr4a0YRVEg
< expires: Fri, 08 Feb 2019 02:54:47 GMT
expires: Fri, 08 Feb 2019 02:54:47 GMT
< date: Fri, 08 Feb 2019 01:54:47 GMT
date: Fri, 08 Feb 2019 01:54:47 GMT
< cache-control: public, max-age=3600
cache-control: public, max-age=3600
< last-modified: Tue, 05 Feb 2019 21:53:55 GMT
last-modified: Tue, 05 Feb 2019 21:53:55 GMT
< etag: "0c7595ff08b5333b780dd5385fd03264"
etag: "0c7595ff08b5333b780dd5385fd03264"
< x-goog-generation: 1549403635072975
x-goog-generation: 1549403635072975
< x-goog-metageneration: 2
x-goog-metageneration: 2
< x-goog-stored-content-encoding: identity
x-goog-stored-content-encoding: identity
< x-goog-stored-content-length: 296
x-goog-stored-content-length: 296
< content-type: application/json
content-type: application/json
< x-goog-hash: crc32c=hAO0Iw==
x-goog-hash: crc32c=hAO0Iw==
< x-goog-hash: md5=DHWV/wi1Mzt4DdU4X9AyZA==
x-goog-hash: md5=DHWV/wi1Mzt4DdU4X9AyZA==
< x-goog-storage-class: REGIONAL
x-goog-storage-class: REGIONAL
< accept-ranges: bytes
accept-ranges: bytes
< content-length: 296
content-length: 296
< server: UploadServer
server: UploadServer
< alt-svc: quic=":443"; ma=2592000; v="44,43,39"
alt-svc: quic=":443"; ma=2592000; v="44,43,39"

<
[{
  "relation": ["delegate_permission/common.handle_all_urls"]
* Connection #0 to host storage.googleapis.com left intact
}]%

day0ops · 2019-02-08T01:58:19Z

Interestingly, I dont see the x-goog headers being passed when going via Nginx.

aledbf · 2019-02-08T10:50:50Z

@nixgadget I forgot to mention that you need to change the path to /.well-known(.*) and the rewrite annotation to /test-api.xxx.co/.well-known$1
Please check #3174 (comment)

day0ops · 2019-02-08T11:25:54Z

@aledbf Thanks for pointing that out ... Don't think i came across it in the changelog docs.

day0ops closed this as completed Feb 8, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to get GCS backend working #3741

Unable to get GCS backend working #3741

day0ops commented Feb 8, 2019 •

edited

aledbf commented Feb 8, 2019

day0ops commented Feb 8, 2019

aledbf commented Feb 8, 2019

day0ops commented Feb 8, 2019 •

edited

day0ops commented Feb 8, 2019

aledbf commented Feb 8, 2019

day0ops commented Feb 8, 2019

Unable to get GCS backend working #3741

Unable to get GCS backend working #3741

Comments

day0ops commented Feb 8, 2019 • edited

aledbf commented Feb 8, 2019

day0ops commented Feb 8, 2019

aledbf commented Feb 8, 2019

day0ops commented Feb 8, 2019 • edited

day0ops commented Feb 8, 2019

aledbf commented Feb 8, 2019

day0ops commented Feb 8, 2019

day0ops commented Feb 8, 2019 •

edited

day0ops commented Feb 8, 2019 •

edited