[FEATURE] Use Nomad Workload Identity and Task API

[gulducat]

Heyaheya! Making an issue for ya as promised @protochron :)

A couple of associated Nomad features might help with folks running Nomad with ACLs enabled, and improve the scalability of Netreap's usage of Nomad API.

• Workload Identity
• Task API
• both via the identity{} block

I don't think this requires any Go application code changes, except to make sure the nomad/api package is up to date to be able to use a unix:// NOMAD_ADDR, and could update your example job with an identity{}, and maybe some additional documentation showing how to set up Nomad policy for the job.

Workload identity provides Nomad tokens dynamically to tasks, rather than needing some static secret (or provisioned by Vault or such). This could also allow the task to read/write Nomad Variables if you wanted to, say, put Cilium policy in a Nomad var instead of Consul k/v, or do leader election with variable locks 😉

The Task API is basically "just" a unix domain socket for the Nomad API through the Nomad client process, which re-uses its open RPC connection to its server, which can reduce open network connections. It also removes the complexity of TLS, which sometimes can be unwieldy. The main caveat is that if the client agent is restarted, or otherwise goes away, the API won't be available until it comes back up.

These features can be used separately, but especially together they're quite nifty. Hopefully the linked documentation helps decide how you might like to implement it, and share with your community as you'd like.

Happy to answer questions or help if you'd like! <3