Proxy check request splays do not spread out check executions

[amdprophet]

I tried applying a proxy check request to a check that was being run by two agents. Both agents were executing the check at the same time when I believe they should have been spread out.

{
  "command": "/sbin/ping -c 1 {{ .URL | default \"google.com\" }}",
  "environment": "default",
  "handlers": [],
  "high_flap_threshold": 0,
  "interval": 100,
  "low_flap_threshold": 0,
  "name": "ping",
  "organization": "default",
  "publish": true,
  "runtime_assets": [],
  "subscriptions": [
    "all"
  ],
  "proxy_entity_id": "",
  "check_hooks": null,
  "subdue": null,
  "ttl": 0,
  "timeout": 0,
  "proxy_requests": {
    "entity_attributes": [
      "entity.Class == 'agent'"
    ],
    "splay": true,
    "splay_coverage": 90
  },
  "metric_handlers": []
}

{
    "entity_attributes": ["entity.Class == 'agent'"],
    "splay": true,
    "splay_coverage": 90
}

[grepory]

@portertech we need some clarification around splay and what splay is meant to do.

We don't execute every check at the same time (check-level splay), but should there also be client level splay (i.e. should we introduce jitter into the check executions at the client level)? It seems like that's going to make aggregates / matching up executions into a batch harder down the road.

[nikkictl]

The following check configuration creates 2 events (on 2 separate entities) that have been executed 27 seconds apart (60 second interval * .9 / 2 entities = 27s). If that is the expected behavior, then I believe this can be closed. I'll wait for more input!

spec:
  check_hooks: null
  command: ping -c 1 google.com
  handlers: []
  high_flap_threshold: 0
  interval: 60
  low_flap_threshold: 0
  metadata:
    annotations: null
    labels: null
    name: ping-proxy
    namespace: default
  output_metric_format: ""
  output_metric_handlers: []
  proxy_entity_name: "proxy"
  proxy_requests:
    entity_attributes:
    - entity.entity_class == "agent"
    splay: true
    splay_coverage: 90
  publish: true
  round_robin: false
  runtime_assets: []
  stdin: false
  subdue: null
  subscriptions:
  - proxy
  timeout: 0
  ttl: 0
type: CheckConfig

staging-agent-0       ping-proxy                 PING google.com (216.58.217.46) 56(84) bytes of data.                                          0   false      2018-11-28 00:45:17 +0000 UTC  
                                                   64 bytes from den03s10-in-f46.1e100.net (216.58.217.46): icmp_seq=1 ttl=44 time=6.11 ms                                                      
                                                                                                                                                                                                
                                                   --- google.com ping statistics ---                                                                                                           
                                                   1 packets transmitted, 1 received, 0% packet loss, time 0ms                                                                                  
                                                   rtt min/avg/max/mdev = 6.113/6.113/6.113/0.000 ms
staging-agent-1       ping-proxy                 PING google.com (216.58.217.46) 56(84) bytes of data.                                          0   false      2018-11-28 00:45:44 +0000 UTC  
                                                   64 bytes from sea15s08-in-f14.1e100.net (216.58.217.46): icmp_seq=1 ttl=44 time=6.12 ms                                                      
                                                                                                                                                                                                
                                                   --- google.com ping statistics ---                                                                                                           
                                                   1 packets transmitted, 1 received, 0% packet loss, time 0ms                                                                                  
                                                   rtt min/avg/max/mdev = 6.120/6.120/6.120/0.000 ms

[grepory]

That is the desired behavior.