Loki: Fix query-frontend ready handler

[slim-bean]

Fixes #2608

[codecov-commenter]

Codecov Report

Merging #2610 into master will decrease coverage by 0.12%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #2610      +/-   ##
==========================================
- Coverage   63.13%   63.00%   -0.13%     
==========================================
  Files         169      169              
  Lines       15018    15023       +5     
==========================================
- Hits         9481     9465      -16     
- Misses       4784     4798      +14     
- Partials      753      760       +7     

Impacted Files	Coverage Δ
pkg/loki/loki.go	0.00% <0.00%> (ø)
pkg/loki/modules.go	4.15% <0.00%> (-0.02%)	⬇️
pkg/promtail/targets/file/tailer.go	70.83% <0.00%> (-8.34%)	⬇️
pkg/promtail/targets/file/filetarget.go	66.85% <0.00%> (-2.29%)	⬇️
pkg/querier/queryrange/downstreamer.go	95.87% <0.00%> (-2.07%)	⬇️
pkg/logql/evaluator.go	92.47% <0.00%> (-0.41%)	⬇️

[owen-d]

I think we're also going to need some jsonnet changes for this as well.

1. Ensure the headless service for the frontend publishes pods that aren't yet ready (the queriers need to connect to this service instead so that the frontend will eventually read as ready once it has downstream queriers)

Something like (note withPublishNotReaddyAddresses)

  query_frontend_discovery_service:
    $.util.serviceFor($.query_frontend_deployment) +
    // Make sure that query frontend worker, running in the querier, do resolve
    // each query-frontend pod IP and NOT the service IP. To make it, we do NOT
    // use the service cluster IP so that when the service DNS is resolved it
    // returns the set of query-frontend IPs.
    service.mixin.spec.withPublishNotReadyAddresses(true) +
    service.mixin.spec.withClusterIp('None') +
    service.mixin.metadata.withName('query-frontend-discovery'),

[owen-d]

lgtm