Use caddy file-server --browse for backend-mode=web

[mholt]

Hi team,

Just a thought! I was playing with --backend-mode=web, and was wondering if there'd be any interest in using that as kind of a wrapper for caddy file-server --browse which has a nicely-formatted file browser (with image previews in grid mode). You can see screenshots of the file browser here:

caddyserver/caddy#5427

The file browser appears whenever there isn't an index.html file. Basically it's a production-ready file server but with a really nice file listing when an index is missing.

[michaelquigley]

Absolutely. That seems like pretty low-hanging fruit!

I may end up with a few questions about the best way to "pre-configure" an embedded Caddy when we use it for things like --backend-mode web. I believe the right approach might be to generate a JSON configuration structure and pass it to caddy.Run?

But if theres' a better way that I should be thinking about this, let me know!

[mholt]

Yep, that's the basic idea. Here's how we do it for the caddy file-server command: https://github.com/caddyserver/caddy/blob/master/modules/caddyhttp/fileserver/command.go

You'll notice we skip the JSON part and go directly to building the struct.

[michaelquigley]

Makes perfect sense! I had assumed there would be a way to go directly to structs skipping the JSON, but I hadn't dug enough for a good example yet... so that's very helpful.

I'll probably work on getting this slotted into the next zrok release also. Like I said, seems like pretty low-hanging fruit.

One of the remaining issues for a TUI-based Caddy integration is still capturing the logging from Caddy. We're using logrus for all of our logging, and I just basically change the destination of the logrus output so that I can capture it and display it in the TUI. I haven't looked deeply into Caddy's logging yet... is there a way I could capture Caddy's logging output to handle it differently in the TUI?

[mholt]

Yeah; we use zap -- and we expose a way to change the output of our logs in Caddy config. You will want to set a writer for the log called default: https://caddyserver.com/docs/json/logging/logs/writer/ (again, you should be able to just build this config with structs without needing JSON, as a more direct approach, if you don't need JSON).

[michaelquigley]

Perfect! I think that's all the silly questions I have for the moment! You just saved me a bunch of time at least knowing that the things I'm trying to do are possible and linking me to an example. Thank you!

👍

p.s. It's on my roadmap to look at other logging frameworks... but I built a custom appender for logrus that captures the function name from the runtime (not super fast, I know... but we try to avoid logging in performance-critical sections). Having the function name in the logging output in a nice way makes log comprehension a lot easier. I just need to take a look at some of the newer frameworks and see if there's a reasonable way to approximate the same thing.

[mholt]

Sounds good!

I don't think there's necessarily anything wrong with logrus 🤷‍♂️ We use zap for its zero-allocation benefits, so we can still log in hot paths. There's probably a way to do what you're asking (Go has functions for getting the calling function name you could add to the log yourself, even.)

Quick off-topic question while we're here, if that's OK... the blog post has these diagrams in it:

I understand a central server is needed to establish trust between the two nodes so they can start communicating; does the second diagram imply that the data has to flow through that central node as well, or is it true P2P? (If it has to flow through the central node, how do you manage all that bandwidth of many users?)

I did that demo myself and it worked btw :)

Question 2: If I want to run my own OpenZiti network, would that be this command here?

[michaelquigley]

I'd bet there's a way to capture the logging stacks in newer, potentially more efficient logging frameworks like zap. I'm just due for a survey of the logging landscape at some point. logrus has been pretty good to us, though.

Yes, OpenZiti exists as a mesh network. The OpenZiti SDK clients make an "edge connection" to a router on that mesh, then the mesh is responsible for using its own routing methods to determine the best path across the mesh (using various smart routing options and self-healing options) to another SDK client. So your connection does traverse networks and compute resources that make up the deployment. We're not doing STUN/TURN style rendezvous, and then allowing traffic to transit outside of OpenZiti.

In reality it's not all that different than horizontally scaling anything else... You can configure and design traffic paths that work whatever way makes sense, and then add resources where they're necessary. The big, fast links still go across the big fast parts of the internet, so it's not actually that much additional overhead in practice. We think of the internet and LAN segments and such as "underlay" and the OpenZiti mesh and clients as the "overlay". OpenZiti supports massive amounts of mesh network infrastructure. And you can also run it with a single controller and a single router if that's what you're looking for.

Future changes to OpenZiti will see things like multi-tenancy, so that multiple control planes can share a single mesh... allowing for economies of scale. Really big deployments with lots of compute and bandwtdth can then be shared by a lot of network control planes.

The bulk of OpenZiti ships as a single executable. Our "router", "controller", "tunnelers", and a lot of administration tooling are all bound into a big cobra CLI namespace. That's just the root of that cobra structure. If you download a copy of the OpenZiti binary you can take a look at the whole CLI tree. There's a handy OpenZiti quickstart that you can use to explore the underlying OpenZiti network and self-host:

https://openziti.io/docs/learn/quickstarts/

The OpenZiti quickstart is used by the zrok self-hosting guide to provide an OpenZiti network for a local zrok deployment:

https://docs.zrok.io/docs/guides/self-hosting/self_hosting_guide/

Glad you got something working! Assuming you're talking about the zrok SDK examples?

Anything I can help with or answer, don't hesitate to ask!

[michaelquigley]

Some early progress on embedded Caddy for --backend-mode web:

[michaelquigley]

@mholt What's the best way for me to update the template used to render these pages so that I can also include a zrok logo? Was thinking about something along the lines of "Powered by zrok Served with Caddy`...

Also have the TUI integration working:

[mholt]

That's looking good! 💯

So your connection does traverse networks and compute resources that make up the deployment.

Gotcha, I think -- so if I wanted to have kind of a self-hosted, independent network of connected clients, I would host a router and then all the traffic would flow through that router?

Just trying to plan infrastructure, like it sounds like I may want a host that does unmetered transfer since it could be quite a lot.

[mholt]

Ope, just saw your other reply.

You can use a custom browse template by specifying template_file: https://caddyserver.com/docs/modules/http.handlers.file_server#browse/template_file

I'd recommend just getting the source code from ours: https://github.com/caddyserver/caddy/blob/master/modules/caddyhttp/fileserver/browse.html -- and then modify it to your liking. (Of course, we do update ours occasionally. So you might want to merge in patches here and there.)

Also have the TUI integration working:

Beautiful!

[michaelquigley]

Thanks! It'll definitely be going out in the next zrok release that includes the Caddy integration. I think it's a really nice improvement in the --backend-mode web functionality.

Gotcha, I think -- so if I wanted to have kind of a self-hosted, independent network of connected clients, I would host a router and then all the traffic would flow through that router?

Yes... you could host a single router along with a single controller. And those would likely only consume very minimal resources. We have lots of people who run that kind of configuration on things like Raspberry Pis.

[michaelquigley]

You can use a custom browse template by specifying template_file: https://caddyserver.com/docs/modules/http.handlers.file_server#browse/template_file

I'd recommend just getting the source code from ours: https://github.com/caddyserver/caddy/blob/master/modules/caddyhttp/fileserver/browse.html -- and then modify it to your liking. (Of course, we do update ours occasionally. So you might want to merge in patches here and there.)

Got it! I'll knock that out next. Then I just need to capture the logging and it's basically done.

[mholt]

Thanks! It'll definitely be going out in the next zrok release that includes the Caddy integration. I think it's a really nice improvement in the --backend-mode web functionality.

Hurray! I think so too 😅

Yes... you could host a single router along with a single controller. And those would likely only consume very minimal resources. We have lots of people who run that kind of configuration on things like Rasberry Pis.

Cool, alright. I'm not too worried about hardware resources so much as paying for excess data transfer through a web hosting provider. DigitalOcean has hard caps on bandwidth (before a pricey 1 cent per GB), but I think services like Scaleway limit speed instead of size. Ramnode also hard-caps bandwidth but has a more affordable .4 cents per GB after that. Anyway, just brainstorming. Thanks!

[michaelquigley]

Cool, alright. I'm not too worried about hardware resources so much as paying for excess data transfer through a web hosting provider. DigitalOcean has hard caps on bandwidth (before a pricey 1 cent per GB), but I think services like Scaleway limit speed instead of size. Ramnode also hard-caps bandwidth but has a more affordable .4 cents per GB after that. Anyway, just brainstorming. Thanks!

You run into anything at all that you need an assist with you just let me know. 👍

Also, for what it's worth the public zrok instance at zrok.io has a pretty generous 10gb/day limit currently. And I'm totally a "I'd rather host it myself" person... but I can vouch for the fact that there are real non-nefarious humans involved with running that service.

[michaelquigley]

I'm not sure that there's going to be a way for us to use our own embedded browse.html:

	if fsrv.Browse.TemplateFile != "" {
		tpl = tplCtx.NewTemplate(path.Base(fsrv.Browse.TemplateFile))
		tpl, err = tpl.ParseFiles(fsrv.Browse.TemplateFile)
		if err != nil {
			return nil, fmt.Errorf("parsing browse template file: %v", err)
		}

By default Caddy is bootstrapping a //go:embed-ed tempalte using tpl.Parse:

		tpl = tplCtx.NewTemplate("default_listing")
		tpl, err = tpl.Parse(defaultBrowseTemplate)
		if err != nil {
			return nil, fmt.Errorf("parsing default browse template: %v", err)
		}

If we were referencing an external file, the TemplateFile config would work, but I don't think there's a way to provide a string path to the go runtime that will reference an embedded file.

[mholt]

Oh, are you trying to embed a custom template into the binary? Hmm 🤔 I'm not sure we've had that request before.

[michaelquigley]

Exactly. We only ship the single binary to end users and we want to co-brand the template, so it would need to be embedded.

[mholt]

Ah... well, for now my suggestion would be to change the browse.html file and compile caddy with that.

I wonder if we made it possible to set the template as a struct field directly -- like as a string, that could probably work?

Still, huge yikes for such a big string in your JSON (you won't see the JSON, but other users will be able to do this, which 🤦‍♂️ is not the most elegant IMO).

I will think on this but if you have any other ideas, let me know!

[mholt]

Or what if there was just a string you could set in the JSON/config that simply added a line to the footer next to "Served with Caddy" -- I've seen a couple people change the whole template to do just this, actually. Maybe that's all we need?

[michaelquigley]

I wonder if we made it possible to set the template as a struct field directly -- like as a string, that could probably work?

That would honestly be the best possible option. We can just //go:embed our own template and then pass it to Caddy. It would really only be useful to people who want to compile their own Caddy builds and override the default template, but that would for sure work for us.

And it also wouldn't muddy up the Caddy configuration when used conventionally.

[mholt]

@michaelquigley How would you feel about my follow-up suggestion:

what if there was just a string you could set in the JSON/config that simply added a line to the footer next to "Served with Caddy"

So you could do, maybe this in your config:

"footer": "Powered by zrok"

and it would appear next to "Served with Caddy"

Would that work?

Embedding the whole thing as a string is a little unwieldy for the reasons I mentioned, so it's not my top priority right now, but if you still want to do that and do it sooner rather than later, let's get NetFoundry set up on a Business sponsorship and I will be able to tackle it right away 👍

[michaelquigley]

@michaelquigley How would you feel about my follow-up suggestion:
Would that work?

No, not really. I suspect that we're probably also going to want to potentially end up making changes to that template for other (functionality-related) reasons as we move forward. And we're also going to want to put our SVG graphic in there, too.

Being able to //go:embed our own template is the solution that makes the most sense.

Embedding the whole thing as a string is a little unwieldy for the reasons I mentioned, so it's not my top priority right now, but if you still want to do that and do it sooner rather than later, let's get NetFoundry set up on a Business sponsorship and I will be able to tackle it right away 👍

I don't really want to expose this as a JSON configuration setting... that doesn't seem like the right approach. I just want to change the default embedded template in the binary.

Business sponsorhips are something we can look into in the medium-term, but it's not likely to happen in the short term. We're just barely establishing a foothold for Caddy in zrok at this point.

We may just have to maintain a private fork of Caddy in the short term to make the changes we need. That's not ideal and I would prefer not to do that... but all we need to do is provide a function in the fileserver package, like this:

//go:embed browse.html
var defaultBrowseTemplate string

func SetDefaultBrowseTemplate(tpl string) {
   defaultBrowseTemplate = tpl
}

And we should be good to go. That would allow us to call fileserver.SetDefaultBrowseTemplate before instantiating the fileserver components, and we're off to the races.

[michaelquigley]

Even simpler would be to change:

var defaultBrowseTemplate string

To:

var DefaultBrowseTemplate string

And also instead of forking all of Caddy, I might be able to just derive our own fileserver module from Caddy's. I'll keep chewing on it...

[mholt]

Ah ok. I can probably do that. Let me look into that tomorrow 👍

[mholt]

Alrighty, caddyserver/caddy@ed8bb13 should be what you're looking for :)

[michaelquigley]

Awesome! Thank you so much! Saves me a bunch of time.

Hoping to get v0.4.6 out early next week with the Caddy integration. Working on a demonstration video this week.