Usage clarification

[absolutejam]

Hey!

First of all, I want to apologise for posting this as I think it may be a bit outside the realm of what you expect in an issue, but I was hoping you could help clarify usage of the this director. Secondly, thanks for providing this, as all of my searches for similar functionality have been dead ends (Varnish 3, varnish-plus). I'm pretty new to Varnish as a whole, and looking to use it in a dynamic Kubernetes environment.

Basically, I'm just trying to understand at what I need to facilitate the usage of this director and the limitations. As an overview, my current setup that I'm toying with is a layer of Varnish servers using the shard director to load-balance caching, and a backend 'content' layer (Nginx) for the actual content to be cached.

From my understanding of the man page..

• I set up the director in sub vcl_init
• Then in sub vcl_recv I basically add any kind of routing logic (eg. based on req.http.host) nothing out of the ordinary so far
• And then ultimately define set req.backend_hint = content.backend("mybackend"); which declares my (dns)resolved backend.

Does this mean that I cannot now leverage a load-balanced director (as the dynamic director must resolve a hostname), and in turn must use another load-balancing service for my content nodes (ideally. with a hostname that can be resolved by this director). This obviously contrasts from the current setup of my director declaring all 4 or so backends and load-balancing, vs. the dynamic director only being able to target a single, unless there's something like DNS RR in place, which would mean another service for my setup.

Thanks for your time!

[nigoroll]

it's fine. While, as a https://github.com/varnishcache/varnish-cache/ developer in the main project I strictly endorse use of github issues for actual bugs only, for this vmod (as for most of our others found on https://gitlab.com/uplex/varnish ) we do not provide the usual community support (mailing-list/forum/...), so using issues for questions is fine for me in this case.
I am not exactly sure if I understand your precise question (maybe you can try to clarify), but the dynamic director just gives you backends to "DNS names" while other directors (like shard) provide load balancing. Speaking of which, shard has been explicitly designed to work with dynamic backends: you can add and remove backends outside vcl_init, so myshard.add_backend(dynamic.backend(....)) should work, see http://varnish-cache.org/docs/trunk/reference/vmod_generated.html#func-shard-add-backend . Just don't forget the shard .reconfigure call. For use with dynamic backends, I recommend to use the ident argument to be able to identify a backend to be removed (if required).

Regarding k8s, I'd like to place an ad here: we are still looking for additional sponsors for https://gitlab.com/uplex/varnish/k8s-ingress

[nigoroll]

one final word: in varnish-cache we still have not solved the director reference issue, see varnishcache/varnish-cache#2072.
The last attempt I made in varnishcache/varnish-cache#2725 was rejected by fellow developers for a killer reason (use of an additional lock), but I got another idea which can avoid this (use the existing VCL Ref). So I'll likely get back to this.
Until then, make sure that thedomain_usage_timeout parameter is longer than the lifetime of any task.

[absolutejam]

Honestly, I was trying to find a way to leverage the dynamic nature of the vmod to allow me to add and remove backends to a live config (to react to servers spinning up & down). But I would need to add them to a pool, which is currently either the shard director or the round-robin director.

I've been reading more into it and it looks like external creation & validation of VCL and then loading with varnishadm might be a better way to tackle my needs.

Just as an aside, will I need to call .add_backend(), .remove_backend(), .reconfigure() etc. when I load a different config, or will the newly loaded config cause the original one's resources to be released (As per the 'temperature' docs)?

[nigoroll]

add/remove backend are intended for use with a running vcl.
add_backend is what you also use for the per-vcl config in vcl_init

[absolutejam]

Sorry to keep pulling this thread, but how would I use them with a running vcl? Can they be called interactively, or is this just something I would utilise in the logic of my vcl (eg. based on a custom PURGE request)?

Is there perhaps another medium for these kind of discussions (As I realise I've really derailed the issue now)?

[nigoroll]

I'm being terse because I need to prep and pack for my holidays, sorry.

You should be able to add/remove backends from anywhere in vcl, yes, not just vcl_init.

As long as this a core varnish question (which it is), the options are in http://varnish-cache.org/support/index.html

[absolutejam]

Thanks for your time. I didn't realise there was an IRC channel, and I'll continue this in there.

All the best with your holidays!