FDS is not discovering AWS functions on 1.2.1

[rickducott]

I downloaded glooctl version 1.2.1 and started minikube. Repro steps:

• glooctl install gateway
• glooctl create secret aws --name aws-creds --access-key ACCESS_KEY --secret-key SECRET_KEY
• Copy this:

apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  name: aws
  namespace: gloo-system
spec:
  aws:
    region: us-east-1
    secretRef:
      name: aws-creds
      namespace: gloo-system

• Run pbpaste | kubectl apply -f - to apply what you copied.

At this point, we'd expect FDS to add all of the lambdas from AWS into the upstream. It fails to do so. The upstream is accepted by gloo but never updated by FDS:

apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gloo.solo.io/v1","kind":"Upstream","metadata":{"annotations":{},"labels":{"valet":"12189196047233956321"},"name":"aws","namespace":"gloo-system"},"spec":{"aws":{"region":"us-east-1","secretRef":{"name":"aws-creds","namespace":"gloo-system"}}}}
  creationTimestamp: "2019-12-09T01:02:24Z"
  generation: 3
  labels:
    valet: "12189196047233956321"
  name: aws
  namespace: gloo-system
  resourceVersion: "15967"
  selfLink: /apis/gloo.solo.io/v1/namespaces/gloo-system/upstreams/aws
  uid: 8fae2097-1a1f-11ea-87b6-765718fe99d9
spec:
  aws:
    region: us-east-1
    secretRef:
      name: aws-creds
      namespace: gloo-system
status:
  reported_by: gloo
  state: 1

FDS is clearly running:

{"level":"info","ts":1575853130.3983667,"logger":"fds.v1.event_loop.fds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:42","msg":"end sync 6782986292515375955","version":"1.2.1"}
{"level":"info","ts":1575853131.3980196,"logger":"fds.v1.event_loop.fds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:31","msg":"begin sync 9919713992579984983 (14 upstreams)","version":"1.2.1"}
{"level":"info","ts":1575853131.4005253,"logger":"fds.v1.event_loop.fds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:42","msg":"end sync 9919713992579984983","version":"1.2.1"}
{"level":"info","ts":1575853345.3945534,"logger":"fds.v1.event_loop.fds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:31","msg":"begin sync 6098242817547083 (15 upstreams)","version":"1.2.1"}
{"level":"info","ts":1575853345.3997355,"logger":"fds.v1.event_loop.fds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:42","msg":"end sync 6098242817547083","version":"1.2.1"}
{"level":"info","ts":1575853559.3946452,"logger":"fds.v1.event_loop.fds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:31","msg":"begin sync 14125428243375918208 (15 upstreams)","version":"1.2.1"}
{"level":"info","ts":1575853559.3972645,"logger":"fds.v1.event_loop.fds.v1.event_loop.syncer","caller":"syncer/discovery_syncer.go:42","msg":"end sync 14125428243375918208","version":"1.2.1"}

---

This means our standard petclinic demo is broken. Full steps for petclinic setup:

• kubectl apply -f https://raw.githubusercontent.com/sololabs/demos/b523571c66057a5591bce22ad896729f1fee662b/petclinic_demo/petclinic.yaml
• kubectl apply -f https://raw.githubusercontent.com/sololabs/demos/b523571c66057a5591bce22ad896729f1fee662b/petclinic_demo/petclinic-vets.yaml
• kubectl apply -f https://raw.githubusercontent.com/sololabs/demos/b523571c66057a5591bce22ad896729f1fee662b/petclinic_demo/petclinic-db.yaml
• glooctl install gateway
• glooctl create virtualservice --name petclinic
• glooctl add route --path-prefix / --dest-name default-petclinic-8080

At this point, the petclinic app should load. Run glooctl proxy url to get the url and then navigate to that in the browser. Petclinic UI should load.

Now we should be able to create a route on the virtual service to the lambda from the aws upstream in order to fix the contact page -- this is blocked.

[ilackarms]

this may be a result of fds needing to be enabled on the upstream/its namespace (and the doc needing an update). did you try enabling fds?

kubectl label namespace <ns> discovery.solo.io/function_discovery=enabled

[rickducott]

I applied that label to gloo-system where my upstreams live, and bounced discovery just to be sure. Didn't fix the issue, going to sync with @ilackarms once i am ready to debug discovery

[rickducott]

Oops, accidentally clicked close

[rickducott]

We found a bug, pushing a quick fix but leaving this open to address several follow ups:

• AWS upstreams are functionless if FDS isn't run on them, so I don't think we should have to whitelist their namespace in order to run FDS. In other words, FDS namespace whitelist/blacklist should only apply to kube upstreams.
• Currently you can't do petclinic in the enterprise UI without manually labeling the namespace you'll create AWS upstream into. If we always run FDS for AWS, then this will be resolved. Otherwise, we need to create this label somehow via the user workflow in the UI

[rickducott]

Actually going to close this and then file a follow up. Also filed #1846