New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Enable scale to zero without an invocation #23

Merged
merged 1 commit into from Dec 1, 2018

Conversation

Projects
None yet
2 participants
@rgee0
Member

rgee0 commented Dec 1, 2018

Still investigating some behaviour observed while testing

Signed-off-by: Richard Gee richard@technologee.co.uk

Description

The existing scale to zero code was such that Prom stats needed to be available in order for faas-idler to scale a function to zero. This meant that on deploy a function would need to be invoked in order to generate Prom stats and consequently scale the function to zero.

This change will remove the need for an invocation to occur in order to scale to zero by allowing entry to the metric generating code when there is no Prom data but the invocation count is zero. The code to create a default value of zero in the metric map has been pulled up out of the data raging loop to enable it to be set in situations where there is no data.

Everything downstream from the metrics generation remains the same.

How Has This Been Tested?

On Kubernetes:

Deploy a clean instance:

$ docker run --rm -it -v $(pwd):/ansible/playbooks -v ~/.ssh/id_rsa_DO:/root/.ssh/id_rsa -v ~/.ssh/id_rsa_DO.pub:/root/.ssh/id_rsa.pub rgee0/ansible-playbook:2.7.0 site.yml -e"orchestrator=k8s"

Scale from zero is enabled by default.

Change the image faas-idler is using to the test image rgee0/faas-idler:zeroOnDeploy also turn dry run to false:

$ vim ~/faas-netes/chart/openfaas/values.yaml

Recreate the pods:

$ helm upgrade --recreate-pods openfaas .

Check currently deployed pods

$ kubectl get pods --all-namespaces 
NAMESPACE     NAME                                               READY   STATUS             RESTARTS   AGE
kube-system   coredns-576cbf47c7-258sx                           1/1     Running            0          119m
kube-system   coredns-576cbf47c7-wnkpw                           1/1     Running            0          119m
kube-system   etcd-openfaas-20181201t133232                      1/1     Running            0          119m
kube-system   kube-apiserver-openfaas-20181201t133232            1/1     Running            0          118m
kube-system   kube-controller-manager-openfaas-20181201t133232   1/1     Running            0          118m
kube-system   kube-proxy-gnzx8                                   1/1     Running            0          119m
kube-system   kube-scheduler-openfaas-20181201t133232            1/1     Running            0          118m
kube-system   tiller-deploy-6f6fd74b68-pc7x7                     1/1     Running            0          119m
kube-system   weave-net-qtn6s                                    2/2     Running            0          119m
openfaas      alertmanager-77fdc8dcfd-c8hmr                      1/1     Running            0          33s
openfaas      faas-idler-7d687685fd-gg8lc                        1/1     Running            0          33s
openfaas      gateway-74f5885846-bglk5                           1/2     CrashLoopBackOff   1          33s
openfaas      nats-6c9c6f8c5d-sbhpp                              1/1     Running            0          33s
openfaas      prometheus-647845457f-8w4fw                        1/1     Running            0          33s
openfaas      queue-worker-6b8986c5f7-4569f                      1/1     Running            2          33s

Confirm that faas-idler is using the test image rgee0/faas-idler:zeroOnDeploy

$ $ kubectl describe pods -n openfaas faas-idler-7d687685fd-gg8lc 
Name:               faas-idler-7d687685fd-gg8lc
Namespace:          openfaas
Priority:           0
PriorityClassName:  <none>
Node:               openfaas-20181201t133232/159.65.86.165
Start Time:         Sat, 01 Dec 2018 15:35:13 +0000
Labels:             app=faas-idler
                    pod-template-hash=7d687685fd
Annotations:        <none>
Status:             Running
IP:                 10.32.0.6
Controlled By:      ReplicaSet/faas-idler-7d687685fd
Containers:
  faas-idler:
    Container ID:  docker://20da209f4f76621c9530689d506d2bcb97a02f6b2801ec99bf2b8f460240ee8b
    Image:         rgee0/faas-idler:zeroOnDeploy
    Image ID:      docker-pullable://rgee0/faas-idler@sha256:a6e30e0c00ac05fdd7feb0b3ea244195e223023c4983b1b8be06d5bf2936d3eb
    Port:          <none>
    Host Port:     <none>
    Command:
      /home/app/faas-idler
      -dry-run=false
    State:          Running
      Started:      Sat, 01 Dec 2018 15:36:33 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Sat, 01 Dec 2018 15:35:49 +0000
      Finished:     Sat, 01 Dec 2018 15:36:19 +0000
    Ready:          True
    Restart Count:  2
    Environment:
      gateway_url:          http://gateway.openfaas:8080/
      prometheus_host:      prometheus.openfaas
      prometheus_port:      9090
      inactivity_duration:  5m
      reconcile_interval:   30s
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-8xwhg (ro)
      /var/secrets/ from auth (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  auth:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  basic-auth
    Optional:    false
  default-token-8xwhg:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-8xwhg
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age               From                               Message
  ----     ------     ----              ----                               -------
  Normal   Scheduled  80s               default-scheduler                  Successfully assigned openfaas/faas-idler-7d687685fd-gg8lc to openfaas-20181201t133232
  Warning  BackOff    13s               kubelet, openfaas-20181201t133232  Back-off restarting failed container
  Normal   Pulling    2s (x3 over 77s)  kubelet, openfaas-20181201t133232  pulling image "rgee0/faas-idler:zeroOnDeploy"
  Normal   Pulled     1s (x3 over 75s)  kubelet, openfaas-20181201t133232  Successfully pulled image "rgee0/faas-idler:zeroOnDeploy"
  Normal   Created    1s (x3 over 75s)  kubelet, openfaas-20181201t133232  Created container
  Normal   Started    0s (x3 over 75s)  kubelet, openfaas-20181201t133232  Started container

Deploy a scale to zer enabled function:

$ faas-cli store deploy figlet --label "com.openfaas.scale.zero=true" -g  http://159.65.86.165:31112

Deployed. 202 Accepted.
URL: http://159.65.86.165:31112/function/figlet

Check the faas-idler logs:

$ kubectl logs -f faas-idler-7d687685fd-gg8lc -n openfaas
2018/12/01 15:36:33 Gateway version: 0.9.5, SHA: 3598da2e513a09a6b8ce6e17c6baa3558a4c8d11
dry_run: false
gateway_url: http://gateway.openfaas:8080/
inactivity_duration: 5m0s 
figlet	idle
2018/12/01 15:36:33 Scale figlet 200 0

figlet	idle

figlet	idle

Check the pods for scaled to zero function

$ kubectl get pods --all-namespaces 
NAMESPACE     NAME                                               READY   STATUS    RESTARTS   AGE
kube-system   coredns-576cbf47c7-258sx                           1/1     Running   0          123m
kube-system   coredns-576cbf47c7-wnkpw                           1/1     Running   0          123m
kube-system   etcd-openfaas-20181201t133232                      1/1     Running   0          122m
kube-system   kube-apiserver-openfaas-20181201t133232            1/1     Running   0          122m
kube-system   kube-controller-manager-openfaas-20181201t133232   1/1     Running   0          121m
kube-system   kube-proxy-gnzx8                                   1/1     Running   0          123m
kube-system   kube-scheduler-openfaas-20181201t133232            1/1     Running   0          122m
kube-system   tiller-deploy-6f6fd74b68-pc7x7                     1/1     Running   0          122m
kube-system   weave-net-qtn6s                                    2/2     Running   0          123m
openfaas      alertmanager-77fdc8dcfd-c8hmr                      1/1     Running   0          3m38s
openfaas      faas-idler-7d687685fd-gg8lc                        1/1     Running   2          3m38s
openfaas      gateway-74f5885846-bglk5                           2/2     Running   2          3m38s
openfaas      nats-6c9c6f8c5d-sbhpp                              1/1     Running   0          3m38s
openfaas      prometheus-647845457f-8w4fw                        1/1     Running   0          3m38s
openfaas      queue-worker-6b8986c5f7-4569f                      1/1     Running   2          3m38s

Deploy non-scale to zero enabled function:

$ faas-cli store deploy figlet -g  http://159.65.86.165:31112 -name figletnoscale

Deployed. 202 Accepted.
URL: http://159.65.86.165:31112/function/figletnoscale

Check faas-idler logs to ensure non-scale to zero function doesn't appear:

$ kubectl logs -f faas-idler-7d687685fd-gg8lc -n openfaas
2018/12/01 15:36:33 Gateway version: 0.9.5, SHA: 3598da2e513a09a6b8ce6e17c6baa3558a4c8d11
dry_run: false
gateway_url: http://gateway.openfaas:8080/
inactivity_duration: 5m0s figlet	idle
2018/12/01 15:36:33 Scale figlet 200 0

figlet	idle

figlet	idle

figlet	idle

figlet	idle

figlet	idle

figlet	idle

figlet	idle

figlet	idle

figlet	idle

figlet	idle

figlet	idle

Check pods for non-scale to zero enabled function:

$ kubectl get pods --all-namespaces 
NAMESPACE     NAME                                               READY   STATUS    RESTARTS   AGE
kube-system   coredns-576cbf47c7-258sx                           1/1     Running   0          127m
kube-system   coredns-576cbf47c7-wnkpw                           1/1     Running   0          127m
kube-system   etcd-openfaas-20181201t133232                      1/1     Running   0          126m
kube-system   kube-apiserver-openfaas-20181201t133232            1/1     Running   0          126m
kube-system   kube-controller-manager-openfaas-20181201t133232   1/1     Running   0          125m
kube-system   kube-proxy-gnzx8                                   1/1     Running   0          127m
kube-system   kube-scheduler-openfaas-20181201t133232            1/1     Running   0          126m
kube-system   tiller-deploy-6f6fd74b68-pc7x7                     1/1     Running   0          126m
kube-system   weave-net-qtn6s                                    2/2     Running   0          127m
openfaas-fn   figletnoscale-6c9c6d688c-m7chb                     1/1     Running   0          2m21s
openfaas      alertmanager-77fdc8dcfd-c8hmr                      1/1     Running   0          7m38s
openfaas      faas-idler-7d687685fd-gg8lc                        1/1     Running   2          7m38s
openfaas      gateway-74f5885846-bglk5                           2/2     Running   2          7m38s
openfaas      nats-6c9c6f8c5d-sbhpp                              1/1     Running   0          7m38s
openfaas      prometheus-647845457f-8w4fw                        1/1     Running   0          7m38s
openfaas      queue-worker-6b8986c5f7-4569f                      1/1     Running   2          7m38s

Try to invoke the zero-scaled function:

$ echo -n 'Scale Up' | faas-cli invoke figlet -g http://159.65.86.165:31112
 ____            _        _   _       
/ ___|  ___ __ _| | ___  | | | |_ __  
\___ \ / __/ _` | |/ _ \ | | | | '_ \ 
 ___) | (_| (_| | |  __/ | |_| | |_) |
|____/ \___\__,_|_|\___|  \___/| .__/ 
                               |_|   
                               

Check logs:

$ kubectl logs -f faas-idler-7d687685fd-gg8lc -n openfaas
2018/12/01 15:36:33 Gateway version: 0.9.5, SHA: 3598da2e513a09a6b8ce6e17c6baa3558a4c8d11
dry_run: false
gateway_url: http://gateway.openfaas:8080/
inactivity_duration: 5m0s figlet	idle
2018/12/01 15:36:33 Scale figlet 200 0

figlet	idle

figlet	idle

figlet	idle

figlet	idle

figlet	idle

figlet	idle

figlet	idle

figlet	idle

figlet	idle

figlet	idle

figlet	idle

figlet	idle

figlet	idle

figlet	idle


figlet	idle
2018/12/01 15:44:33 Scale figlet 200 0

figlet	idle

Check pods:

$ kubectl get pods --all-namespaces 
NAMESPACE     NAME                                               READY   STATUS        RESTARTS   AGE
kube-system   coredns-576cbf47c7-258sx                           1/1     Running       0          128m
kube-system   coredns-576cbf47c7-wnkpw                           1/1     Running       0          128m
kube-system   etcd-openfaas-20181201t133232                      1/1     Running       0          128m
kube-system   kube-apiserver-openfaas-20181201t133232            1/1     Running       0          127m
kube-system   kube-controller-manager-openfaas-20181201t133232   1/1     Running       0          127m
kube-system   kube-proxy-gnzx8                                   1/1     Running       0          128m
kube-system   kube-scheduler-openfaas-20181201t133232            1/1     Running       0          127m
kube-system   tiller-deploy-6f6fd74b68-pc7x7                     1/1     Running       0          128m
kube-system   weave-net-qtn6s                                    2/2     Running       0          128m
openfaas-fn   figlet-8db8f946d-nwl46                             0/1     Terminating   0          50s
openfaas-fn   figletnoscale-6c9c6d688c-m7chb                     1/1     Running       0          4m16s
openfaas      alertmanager-77fdc8dcfd-c8hmr                      1/1     Running       0          9m33s
openfaas      faas-idler-7d687685fd-gg8lc                        1/1     Running       2          9m33s
openfaas      gateway-74f5885846-bglk5                           2/2     Running       2          9m33s
openfaas      nats-6c9c6f8c5d-sbhpp                              1/1     Running       0          9m33s
openfaas      prometheus-647845457f-8w4fw                        1/1     Running       0          9m33s
openfaas      queue-worker-6b8986c5f7-4569f                      1/1     Running       2          9m33s

On Swarm:

Deploy a clean instance:

$ docker run --rm -it -v $(pwd):/ansible/playbooks -v ~/.ssh/id_rsa_DO:/root/.ssh/id_rsa -v ~/.ssh/id_rsa_DO.pub:/root/.ssh/id_rsa.pub rgee0/ansible-playbook:2.7.0 site.yml -e"orchestrator=swarm"

Check deployed services:

$ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                         PORTS
6il4jetxgchj        func_alertmanager   replicated          1/1                 prom/alertmanager:v0.15.0     
reovec882kkj        func_faas-swarm     replicated          1/1                 openfaas/faas-swarm:0.4.4     
j2o6m8bqf1k5        func_gateway        replicated          1/1                 openfaas/gateway:0.9.10       *:8080->8080/tcp
vd29d88c6w0n        func_nats           replicated          1/1                 nats-streaming:0.11.2         
pug7yt8yex8z        func_prometheus     replicated          1/1                 prom/prometheus:v2.3.1        *:9090->9090/tcp
olul0pjljorm        func_queue-worker   replicated          1/1                 openfaas/queue-worker:0.5.4   

Enable scale from zero:

$ cd ~/faas

$ vim docker-compose.yml
Change to `scale_from_zero: "true"`

$ docker stack deploy func -c docker-compose.yml

Updating service func_faas-swarm (id: reovec882kkjeb611l37a55o5)
Updating service func_nats (id: vd29d88c6w0nw57rs5rt4q805)
Updating service func_queue-worker (id: olul0pjljormlyz8lpu6pd5hf)
Updating service func_prometheus (id: pug7yt8yex8z6mdyla7xuf69m)
Updating service func_alertmanager (id: 6il4jetxgchj66rc6tfmojxs4)
Updating service func_gateway (id: j2o6m8bqf1k5drw20198mq5l8)

Clone faas-idler:

$ git clone https://github.com/openfaas-incubator/faas-idler.git  && cd faas-idler

Amend stack to use test image @ rgee0/faas-idler:zeroOnDeploy

$ vim docker-compose.yml

Change the image to `rgee0/faas-idler:zeroOnDeploy`

$ docker stack deploy func -c docker-compose.yml
Creating service func_faas-idler

Check faas-idler is deployed:

$ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
6il4jetxgchj        func_alertmanager   replicated          1/1                 prom/alertmanager:v0.15.0       
s7w2ovn051o2        func_faas-idler     replicated          1/1                 rgee0/faas-idler:zeroOnDeploy   
reovec882kkj        func_faas-swarm     replicated          1/1                 openfaas/faas-swarm:0.4.4       
j2o6m8bqf1k5        func_gateway        replicated          1/1                 openfaas/gateway:0.9.10         *:8080->8080/tcp
vd29d88c6w0n        func_nats           replicated          1/1                 nats-streaming:0.11.2           
pug7yt8yex8z        func_prometheus     replicated          1/1                 prom/prometheus:v2.3.1          *:9090->9090/tcp
olul0pjljorm        func_queue-worker   replicated          1/1                 openfaas/queue-worker:0.5.4    

Deploy a zero scale enabled function:

$ faas-cli store deploy figlet --label "com.openfaas.scale.zero=true" -g  http://206.189.25.109:8080

Deployed. 202 Accepted.
URL: http://206.189.25.109:8080/function/figlet

Check logs for scale event:

$ docker service logs -f func_faas-idler 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 2018/12/01 13:50:22 Gateway version: 0.9.10, SHA: b4c12f824bcea6b3038f5c878001f72e6a57de1e
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | dry_run: false
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | gateway_url: http://gateway:8080/
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | inactivity_duration: 5m0s 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | figlet	idle
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 2018/12/01 13:54:52 Scale figlet 202 0

Check function has scaled:

$ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
llqn1yy29v9w        figlet              replicated          0/0                 functions/figlet:0.9.6          
6il4jetxgchj        func_alertmanager   replicated          1/1                 prom/alertmanager:v0.15.0       
s7w2ovn051o2        func_faas-idler     replicated          1/1                 rgee0/faas-idler:zeroOnDeploy   
reovec882kkj        func_faas-swarm     replicated          1/1                 openfaas/faas-swarm:0.4.4       
j2o6m8bqf1k5        func_gateway        replicated          1/1                 openfaas/gateway:0.9.10         *:8080->8080/tcp
vd29d88c6w0n        func_nats           replicated          1/1                 nats-streaming:0.11.2           
pug7yt8yex8z        func_prometheus     replicated          1/1                 prom/prometheus:v2.3.1          *:9090->9090/tcp
olul0pjljorm        func_queue-worker   replicated          1/1                 openfaas/queue-worker:0.5.4     

Deploy a non-zero scale enabled function:

$ faas-cli store deploy figlet -g  http://206.189.25.109:8080 -name figletNoScale

Deployed. 202 Accepted.
URL: http://206.189.25.109:8080/function/figletNoScale

Check faas-idler logs to ensure no scale to zero occurred for figletNoScale:

$ docker service logs -f func_faas-idler 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 2018/12/01 13:50:22 Gateway version: 0.9.10, SHA: b4c12f824bcea6b3038f5c878001f72e6a57de1e
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | dry_run: false
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | gateway_url: http://gateway:8080/
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | inactivity_duration: 5m0s 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | figlet	idle
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 2018/12/01 13:54:52 Scale figlet 202 0
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | figlet	idle
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | figlet	idle
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | figlet	idle
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | figlet	idle
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | figlet	idle

Check function has not scaled:

$ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
llqn1yy29v9w        figlet              replicated          0/0                 functions/figlet:0.9.6          
pcpsyeffw6j9        figletNoScale       replicated          1/1                 functions/figlet:0.9.6          
6il4jetxgchj        func_alertmanager   replicated          1/1                 prom/alertmanager:v0.15.0       
s7w2ovn051o2        func_faas-idler     replicated          1/1                 rgee0/faas-idler:zeroOnDeploy   
reovec882kkj        func_faas-swarm     replicated          1/1                 openfaas/faas-swarm:0.4.4       
j2o6m8bqf1k5        func_gateway        replicated          1/1                 openfaas/gateway:0.9.10         *:8080->8080/tcp
vd29d88c6w0n        func_nats           replicated          1/1                 nats-streaming:0.11.2           
pug7yt8yex8z        func_prometheus     replicated          1/1                 prom/prometheus:v2.3.1          *:9090->9090/tcp
olul0pjljorm        func_queue-worker   replicated          1/1                 openfaas/queue-worker:0.5.4 

Test scaled down function scales up when required:

$ echo -n 'Scale Up' | faas-cli invoke figlet -g http://206.189.25.109:8080
 ____            _        _   _       
/ ___|  ___ __ _| | ___  | | | |_ __  
\___ \ / __/ _` | |/ _ \ | | | | '_ \ 
 ___) | (_| (_| | |  __/ | |_| | |_) |
|____/ \___\__,_|_|\___|  \___/| .__/ 
                               |_|    

Check that faas-idler logs correspond:

$ docker service logs -f func_faas-idler 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 2018/12/01 13:50:22 Gateway version: 0.9.10, SHA: b4c12f824bcea6b3038f5c878001f72e6a57de1e
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | dry_run: false
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | gateway_url: http://gateway:8080/
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | inactivity_duration: 5m0s 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | figlet	idle
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 2018/12/01 13:54:52 Scale figlet 202 0
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | figlet	idle
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | figlet	idle
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | figlet	idle
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | figlet	idle
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | figlet	idle
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 
func_faas-idler.1.asxa989lm3vk@openfaas-20181201T133905    | 

Check service shows 1 replica:

$ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
llqn1yy29v9w        figlet              replicated          1/1                 functions/figlet:0.9.6          
pcpsyeffw6j9        figletNoScale       replicated          1/1                 functions/figlet:0.9.6          
6il4jetxgchj        func_alertmanager   replicated          1/1                 prom/alertmanager:v0.15.0       
s7w2ovn051o2        func_faas-idler     replicated          1/1                 rgee0/faas-idler:zeroOnDeploy   
reovec882kkj        func_faas-swarm     replicated          1/1                 openfaas/faas-swarm:0.4.4       
j2o6m8bqf1k5        func_gateway        replicated          1/1                 openfaas/gateway:0.9.10         *:8080->8080/tcp
vd29d88c6w0n        func_nats           replicated          1/1                 nats-streaming:0.11.2           
pug7yt8yex8z        func_prometheus     replicated          1/1                 prom/prometheus:v2.3.1          *:9090->9090/tcp
olul0pjljorm        func_queue-worker   replicated          1/1                 openfaas/queue-worker:0.5.4     

Checklist:

I have:

  • updated the documentation if required
  • read the CONTRIBUTION guide
  • signed-off my commits with git commit -s
  • added unit tests
Enable scale to zero without an invocation
The existing scale to zero code was such that Prom stats needed to be available in order for faas-idler to scale a function to zero.  This meant that on deploy a function would need to be invoked in order to generate Prom stats and consequently scale the function to zero.

This change will remove the need for an invocation to occur in order to scale to zero by allowing entry to the metric generating code when there is no Prom data but the invocation count is zero.  The code to create a default value of zero in the metric map has been pulled up out of the data raging loop to enable it to be set in situations where there is no data.

Signed-off-by: Richard Gee <richard@technologee.co.uk>

@rgee0 rgee0 changed the title from Enable scale to zero without an invocation to [WIP] Enable scale to zero without an invocation Dec 1, 2018

@alexellis

Very succinct and brilliant to see how it was tested with repeatable instructions.

res, err := query.Fetch(querySt)
if err != nil {
log.Println(err)
continue
}
if len(res.Data.Result) > 0 {
if len(res.Data.Result) > 0 || function.InvocationCount == 0 {

This comment has been minimized.

@alexellis

alexellis Dec 1, 2018

Member

Could we "and" the right hand side with whether there are any replicas?

This comment has been minimized.

@rgee0

rgee0 Dec 1, 2018

Member

I did have that in to start with but took it out for consistency with the left hand side.

I think idled functions with an function.InvocationCount > 0 also have a res.Data.Result > 0 and a Replicas ==0 value. This means in the idler logs you get a print out of functions considered idle:

func_faas-idler.1.erz2vz7i8hp2@linuxkit-025000000001    | burty	idle
func_faas-idler.1.erz2vz7i8hp2@linuxkit-025000000001    | burtoli	idle
func_faas-idler.1.erz2vz7i8hp2@linuxkit-025000000001    | trent	idle
func_faas-idler.1.erz2vz7i8hp2@linuxkit-025000000001    | burt	idle
func_faas-idler.1.erz2vz7i8hp2@linuxkit-025000000001    | burton	idle
func_faas-idler.1.erz2vz7i8hp2@linuxkit-025000000001    | nodeinfo	idle

If we add a replicas check in the right hand side this log will only show idle functions that have Prom stats.

This comment has been minimized.

@alexellis

alexellis Dec 1, 2018

Member

Thanks

@alexellis alexellis merged commit 22cff29 into openfaas-incubator:master Dec 1, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@rgee0

This comment has been minimized.

Member

rgee0 commented Dec 1, 2018

On the remaining issue of the function scaling down a second time after the first invocation due to the delay in metrics propagating. I have been through all the points that data is extracted from the API and there doesn't appear to be a point where metrics can reliably be used to prevent an additional scale down straight after the first invocation. A couple of choices would be:

  1. add in a delay to allow the metrics to propagate
  2. hold a bool somewhere that tracks whether an initial scale down has occurred and use this in conjunction with the metrics to prevent a second scale down.
  3. open to ideas
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment