Is it possible: self-routing sharded Varnish Cache cluster with libvmod-dynamic ?

[abresson]

Hello,

I usually (without libvmod-dynamic) use cache sharding on varnish, based on what is described here : https://info.varnish-software.com/blog/creating-self-routing-varnish-cluster

With libvmod-dynamic I'm not able to determine the varnish backend identity from which the traffic is coming from because req.backend_hint doesn't return the name of a specific backend, but instead the general name of the dynamic backend.

Here are some vcl configuration to try to make myself clear:

Without dynamic

backend node1 {
  .host = "node1.example.com";
  .port = "80";
}

backend node2 {
  .host = "node2.example.com";
  .port = "80";
}

backend content {
  .host = "content-origin.example.com";
  .port = "80";
}

sub vcl_init
{
  new cluster = directors.hash();
  cluster.add_backend(node1, 1);
  cluster.add_backend(node2, 1);
}

sub vcl_recv
{
  set req.backend_hint = cluster.backend();
  set req.http.X-shard = req.backend_hint;

  if (req.http.X-shard == server.identity) {
    set req.backend_hint = content;
  } else {
    return(pass);
  }
}

In the above code, there is no problem, req.backend_hint will be set to either node1.example.com or node2.example.com

With dynamic

Let's imagine node.example.com is a multi A DNS record with 2 varnish servers,
node.example.com 300 IN CNAME varnish1
node.example.com 300 IN CNAME varnish2

probe node_probe {
  .interval = 5s;
  .timeout = 1s;
  .window = 5;
  .threshold = 3;
  .initial = 3;
  .request =
          "HEAD / HTTP/1.1"
          "Host: node.example.com"
          "Connection: close"
          "User-Agent: Varnish node Health Probe";

}

probe content_probe {
  .interval = 5s;
  .timeout = 1s;
  .window = 5;
  .threshold = 3;
  .initial = 3;
  .request =
          "HEAD / HTTP/1.1"
          "Host: content-origin.example.com"
          "Connection: close"
          "User-Agent: Varnish content Health Probe";
}

sub vcl_init
{
  new d_node = dynamic.director(
    probe = node_probe,
    host_header = "Host",
    port = 80,
    ttl = 30s
  );

  new d_content= dynamic.director(
    probe = content_probe,
    host_header = "Host",
    port = 80,
    ttl = 30s
  );

  new v_cluster = directors.shard();
  varnish_nodes.add_backend(d_node.backend("node.example.com","80"));
  varnish_nodes.reconfigure();

  new c_cluster = directors.round_robin();
  front_nodes.add_backend(d_content.backend("content-origin.example.com","80"));
}

sub vcl_recv
{
  set req.backend_hint = v_cluster.backend();
  set req.http.X-shard = req.backend_hint;

  if (req.http.X-shard == server.identity) {
    set req.backend_hint = c_cluster;
  } else {
    return(pass);
  }
}

The above code doesn't work because req.backend_hint = v_cluster.backend() will be set to v_cluster(node.example.com:80) and not to the actual varnish backend hosts.
I mean req.http.X-shard will be set to either varnish2 or varnish1 whereas req.backend_hint wil always be set to v_cluster(node.example.com:80)

Is there any way to put hostname of the varnish wich actually catched the orign request in
req.backend_hint ?

I'm sorry if my explanation is confused, let me know if you need me to add more information to clarify.

Regards,

[nigoroll]

In general, for clustering, I would recommend to have a look at https://code.uplex.de/uplex-varnish/libvmod-cluster and in particular https://code.uplex.de/uplex-varnish/libvmod-cluster/blob/master/vcl/vshard.inc.vcl . I would conisder the clustering howto by varnish-software suboptimal, in particular because it uses the hash director (rather than shard).

That said, you try to layer shard/round-robin over dynamic. Being able to do so is a long standing request, but I do not think we currently have a safe way of achieving this.

You could try to use req.backend_hint.resolve() (introduced in 97e8a9f58ac4270215f2c2529d9d0897f7d9b168) when you assign X-Shard, but that would only add a backend and never remove it again for the lifetime of a VCL. Because we do not have director reference counting in varnish (yet, see varnishcache/varnish-cache#2725, I actually still have unfinished work to implement it differently), we can not delete dynamic backends before a VCL reload.

Also, let me remind you to call .reconfigure() after changing shard director backends for the change to have any effect.

The basic idea for a correct solution to this problem would, in my mind, be an API for directors to explicitly support layering, that is, for example, allowing one director to dynamically use backends created by another director. This is even more in the future than director reference counting.

[abresson]

Hello @nigoroll ,

Thank you for your answer,

I wasn't aware that dynamic will not be able remove backends without reloading the VLC.
From your answer I understand I won't be able to get dynamic and shard working in a reliable way for the moment.
I will take a look at the libvmod-cluster as It may help me to manage my cluser in an easier way.

Should we close the issue ?

Regards,

[pnbecker]

@nigoroll thank you for providing libvmod-dynamic. It is great. This issue is now 4 years old. Is there any update in combining a cluster and libvmod-dynamic?

[nigoroll]

@pnbecker Yes, there is news, we do fully support layering now (you will find some mentions of layering/layered in CHANGES.rst).

But besides this, to get a proper sharded configuration, we would ideally need a way for one director to add all the backends of another director, and then also update its configuration periodically. Or, as a bit of a hack-ish, the dynamic director could directly integrate the shard director logic, but that would really not be good from a design perspective.

I have dropped an RFC...

[pnbecker]

@nigoroll that is awesome, thank you!

If anyone stumble on this, I was able to solve it with the following data (following the information from CHANGES.rst):

# specify the VCL syntax version to use
vcl 4.1;

import dynamic;
import directors;

backend default none;

sub vcl_init {
  # initiate a directory for vmod dynamic
  new d = dynamic.director(port = "80", ttl=3s);
  new ui_dir = directors.round_robin();
  ui_dir.add_backend(d.backend("server1.example.org", 80));
  ui_dir.add_backend(d.backend("server2.example.org", 80));
}

sub vcl_recv {
  if (req.http.host ~ "^(www\.)?example\.org$") {
      set req.backend_hint = ui_dir.backend();
  }
}