Using environment variables for setting Networks and Volumes

[ravenpride]

At the moment I'm working on a project that builds a docker image providing an auto-scaling gitlab-runner connected to docker-machine and your driver for Hetzner Cloud. The docker container pulls its settings from environment variables, so I set the variables HETZNER_VOLUMES and HETZNER_NETWORKS to configure bound volumes and connected private networks at Hetzner, but I could not get it working. The variables seem to be ignored.

Is there anything I can do to track the issue down?

Thank you in advance!

[JonasProgrammer]

A good point to start would be to simply put bogus values into these variables. The names or ids of the corresponding components are simply put through to the corresponding retrieval functions of hcloud-go. If they return an error, we propagate it.
So if you can put anything there and it does not error out pre-creation, something is up.

If this turns out to be the case, we have to dig deeper; using a custom-build version with some debug output is the easiest way to go then. I can help you with that, if you want.

Aside from that, can you give us a little more insight on your environment? Judging from your Dockerfile it's alpine:3.10? I have never tried running this under alpine, though there shouldn't be any issues with that.
Also, please run sha256sum or something on the binary and paste the output here.

[ravenpride]

Ok, it seems there is something strange going on. I've put bogus input into HETZNER_VOLUMES and HETZNER_NETWORKS, but no error pops up. I'm afraid the sha256 hash will not enlighten the issue as the image builds docker-machine and the Hetzner driver from source :-/

[ravenpride]

I've downloaded the prebuilt binaries for docker-machine and the Hetzner driver. My platform is Ubuntu 18.04 LTS (amd64) now. The following script...

#!/bin/bash

export HETZNER_API_TOKEN='[[ REDACTED ]]'
export HETZNER_IMAGE='ubuntu-18.04'
export HETZNER_LOCATION='fsn1'
export HETZNER_TYPE='cx11'
export HETZNER_VOLUMES='test12345'
export HETZNER_NETWORKS='test1235'

docker-machine create \
  --driver hetzner \
  some-machine

produces the following output:

Creating CA: /home/sascha/.docker/machine/certs/ca.pem
Creating client certificate: /home/sascha/.docker/machine/certs/cert.pem
Running pre-create checks...
Creating machine...
(some-machine) Creating SSH key...
(some-machine) Creating Hetzner server...
(some-machine)  -> Creating server some-machine[3022235] in create_server[26268622]
(some-machine)  -> Server some-machine[3022235]: Waiting to come up...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with ubuntu(systemd)...
Installing Docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env some-machine

Although the specified volume and the network do not exist, no error is signaled.

[JonasProgrammer]

Mmh, that's strange.

Given that you build it from source anyways, could you please try and add a log message before https://github.com/JonasProgrammer/docker-machine-driver-hetzner/blob/master/driver.go#L255 (and do the same before line 266) and print nameOrID? If there is no output for this, then the options are indeed not getting passed.

Btw, have you tried if it works when you pass in the option via command line?

[ravenpride]

Ok, I added log.Infof(">>> NETWORKS: %s", networks) before line 255 and log.Infof(">>> VOLUMES: %s", volumes) before line 266.

Here is the result:

...
(some-machine) >>> NETWORKS: []
(some-machine) >>> VOLUMES: []
...

The arguments seem to be stuck somewhere.

---

Then I've tried the following:

export HETZNER_API_TOKEN='[[ redacted ]]'
export HETZNER_IMAGE='ubuntu-18.04'
export HETZNER_LOCATION='fsn1'
export HETZNER_TYPE='cx11'

docker-machine create \
  --driver hetzner \
  --hetzner-networks='volume-12345' \
  --hetzner-volumes='test12345' \
  some-machine

Result:

Running pre-create checks...
Creating machine...
(some-machine) Creating SSH key...
(some-machine) Creating Hetzner server...
(some-machine) >>> NETWORKS: []
(some-machine) >>> VOLUMES: []
Error creating machine: Error in driver during machine creation: Panic in the driver: runtime error: invalid memory address or nil pointer dereference
goroutine 27 [running]:
runtime/debug.Stack(0x8d3a80, 0x0, 0x0)
        /usr/lib/go-1.10/src/runtime/debug/stack.go:24 +0xa7
github.com/docker/machine/libmachine/drivers/rpc.(*StandardStack).Stack(0xaf6680, 0x7fea40, 0xac89e0, 0x8d3a80)
        /home/sascha/go/src/github.com/docker/machine/libmachine/drivers/rpc/server_driver.go:23 +0x22
github.com/docker/machine/libmachine/drivers/rpc.trapPanic(0xc420225b60)
        /home/sascha/go/src/github.com/docker/machine/libmachine/drivers/rpc/server_driver.go:129 +0x96
panic(0x7fea40, 0xac89e0)
        /usr/lib/go-1.10/src/runtime/panic.go:502 +0x229
github.com/hetznercloud/hcloud-go/hcloud.(*ServerClient).Create(0xc42042aeb8, 0x8d6ca0, 0xc4200b4010, 0xc42019a530, 0xc, 0xc4201a8000, 0xc4201b4270, 0xc42019e158, 0x1, 0x1, ...)
        /home/sascha/go/src/github.com/hetznercloud/hcloud-go/hcloud/server.go:295 +0x2d2
main.(*Driver).Create(0xc4201ac000, 0x0, 0x0)
        /home/sascha/go/src/github.com/jonasprogrammer/docker-machine-driver-hetzner/driver.go:292 +0x897
github.com/docker/machine/libmachine/drivers/rpc.(*RPCServerDriver).Create(0xc42013d720, 0xaf6900, 0xaf6900, 0x0, 0x0)
        /home/sascha/go/src/github.com/docker/machine/libmachine/drivers/rpc/server_driver.go:140 +0x65
reflect.Value.call(0xc4200e21e0, 0xc4200b6740, 0x13, 0x880262, 0x4, 0xc420069f18, 0x3, 0x3, 0xc4201a4380, 0x7bc6e0, ...)
        /usr/lib/go-1.10/src/reflect/value.go:447 +0x969
reflect.Value.Call(0xc4200e21e0, 0xc4200b6740, 0x13, 0xc420249718, 0x3, 0x3, 0xc4204012b0, 0x1, 0x1)
        /usr/lib/go-1.10/src/reflect/value.go:308 +0xa4
net/rpc.(*service).call(0xc4200bb900, 0xc4200b8500, 0xc42019a050, 0xc42019a060, 0xc4201a8d80, 0xc4201a0400, 0x7ca4e0, 0xaf6900, 0x16, 0x7ca4e0, ...)
        /usr/lib/go-1.10/src/net/rpc/server.go:384 +0x14e
created by net/rpc.(*Server).ServeCodec
        /usr/lib/go-1.10/src/net/rpc/server.go:480 +0x43a

[JonasProgrammer]

I did some debugging and this actually looks like a bug somewhere downstream in libmachine. Even in SetConfigFromFlags the raw names are already empty, when using environment variables.
When using options, they're properly populated (although invalid values crash the driver binary, which I opened #38 for).

Given that it works for non-slice flags, I for now can just assume this is not our field. Do you have, by any chance, any out-of-tree machine driver that uses string slice flags as well? Would be interesting to see if the behaviour is the same there.

[ravenpride]

Hmm, there is no driver I'm aware of.

To be honest, I'm not a GO programmer, so tracking the issue down could become a bit tedious.

[JonasProgrammer]

Sorry to coming back to you only now. I just want to let you know I'm still on this, but don't hold your breath please.

I still think this is an issue in libmachine or one of its dependencies, but even if I can figure it out, I'm not sure how to go about this. Let's just pretend I were able to figure out the root cause and fix it, getting it back into libmachine would take quite some time, so this is not really a solution, if you need a quick fix.
Working around this issue in the driver code (i.e. manually parsing the env var -- if it is even passed to the driver binary), on the other hand, is basically onion code and not really in the driver's scope.

I'll try and look into this in the weekend.

[ravenpride]

Thank you very much for your engagement, Jonas!

I think I can work around the issue by mapping the environment variables to docker-machine command line parameters. This circumvents the problem entirely and enables me to proceed.

BTW: Your driver is a great piece of work and Hetzner Cloud is also my favorite :-)

[JonasProgrammer]

https://github.com/docker/machine/blob/a555e4f7a8f518a8b1b174824c377e46cbfc4fe2/commands/create.go#L371
This in turn calls this, which does not even consider env vars (also it is fixed in the current version, but docker-machine has locked it's package to the former commit).

That's your problem right there. The flag is not even considered, if it's not set by the command line. Kinda useless to then even provide EnvVar in the interface, but anyways.

Basically, this confirms what I already guessed: It's a problem of docker-machine in general, which cannot be fixed here. Feel free to create a downstream issue there. For now, you have to stick to your workaround, I guess :/

EDIT: I'll still leave this open, just in case machine fixes this some time, so we can update our dependencies in turn.

BTW: Your driver is a great piece of work and Hetzner Cloud is also my favorite :-)

Thanks, but aside from building the initial implementation in the beta times I don't do as much; @mxschmitt is the person carrying this. Heck, this guy fixed the #38 before I had finished typing the answer...

[ravenpride]

Then the laurels go to you too, @mxschmitt. Good work and many thanks for the driver :-)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[sergey-kudriavtsev]

Hi guys! @JonasProgrammer @ravenpride ...
Thanks for your work, it's important!

I want to make a suggestion for an additional parameter and functionality.

HETZNER_TYPE=test_net # - network selection
HETZNER_TYPE_MASK='10.0.3.0/24' # - subnet selection by mask.

Explanations:
When running clusters through a Rancher, we cannot create two clusters on the same network. This is because when using CNI as iptables, the network provider drivers (from the first and second cluster) try to kill the previously deployed routes. And it doesn't stop. One cluster sees that the table does not match and changes it, the second also sees that the table does not match itself and changes it again.

Therefore, either you need to first create a dedicated network, a balancer, and dance with a tambourine ...

Or we can determine exactly which subnet the new node should be patched to.

I tried it differently, took out the created clusters in the subnet, but as it turned out, hetzner randomly issues IP and subnet wounds for one cluster, conditionally for each call to get a node, a new IP from a random subnet.

I hope we will can use your plugin for Rancher with new feature ((manual or selecting subnetwork).