Option to use private ip address with EC2

[joamaki]

Often one wants to block SSH on the public IP address of the EC2 instance and only
expose SSH on the internal IP. It would be great if NixOps could fallback to, or prefer, the private IP
of the target when deploying.

[edolstra]

That would only work if you're running NixOps from inside EC2, right?

[joamaki]

Yes, or proxying through a jumphost into the same security group.

[copumpkin]

👍

[cwp]

If this isn't implemented, why does the deployment.ec2.associatePublicIpAddress option exist? If it's set to false (which is the default), nixops can't connect to the instance to configure it.

[mbbx6spp]

@cwp I can be on a VPN that is connected to a igw subnet and want to provision and deploy to instances in there. You shouldn't need to have a public IP address to deploy to those instances if you can connect to it. HTH.

[mbbx6spp]

If this is still a work in progress, I would like to attempt it as I have this very need right now. Let me know if anyone else is actively working on it. Thanks.

[mbbx6spp]

Sorry @cwp I think you were asking a slightly different question. You should use the deployment.ec2.associatePublicIpAddress option if the instances in your VPC subnet that you are deploying needed a public IP to be associated with it in a vgw subnet. Hope that makes sense as to the difference in request.

[mbbx6spp]

I just looked into doing this and it seems we might need a slight change to the backend API to support this. Honestly, it seems like it would be a useful API change for a number of things anyway. But I wanted to run it by people here to see what they think.

I added a deployment.ec2.usePrivateIpAddress option in Nix and read into the Python EC2Definition class and then in the EC2State we have the method get_ssh_name which currently only takes self as an argument. It would need the definition passed into it to be able to do what we need above. This requires a change to the backend API as a whole (across all backends). I didn't want to go in and just do this without gathering thoughts from others on this.

Cheers.

[cwp]

@mbbx6spp The problem is that nixops will only connect to the public IP address via SSH. If deployment.ec2.associatePublicIpAddress is false, then it will create an instance and then sit there endless printing out "waiting for IP address...". It's waiting for the instance metadata to have a public IP address, which will never happen.

AFAICT, deployment.ec2.associatePublicIpAddress must be set to true, or deployment will always fail.

[mbbx6spp]

@cwp yes, I totally understand this current limitation. The request was to remove that limitation though. I don't believe this is a valid limitation that should be imposed if NixOps is to be used for internal deployments. Please see my above comment about the API issues with the backend stuff in NixOps. There is not enough context. We should not impose this model. NixOps should be able to deploy on internal IPs. Maybe I am missing something, but deploying on public IPs seems like a major limitation to me for larger deployments.

[mbbx6spp]

The one horrible hack that I can think of that gets around the issue of changing the backend API for MachineState#get_ssh_name is to just return self.private_ipv4 if it is not None and self.public_ipv4 is None, but this seems like a horrible horrible hack and highlights the API design issue of not passing in the definition to the MachineState methods across the board.

[cwp]

@mbbx6spp I thought about doing that as well, but I think it's a little more complicated.

I'm currently running a patched version of nixops that always uses the private address. I need that because I'm running nixops on an EC2 instance. Even though my instances do have public IP addresses, I can't connect to them from within the VPC; I have to use the private address.

And that's the tricky bit. The decision about which address to use isn't really a property of the deployment specification. It depends on the state of the routers between the nixops host and the target instance. If deployment.ec2.usePrivateIpAddress was used, then the same nix expression would work when evaluated on machine and fail when evaluated on another.

I think the proper thing is to have that flag be in the state database and set command line flag on "nixops create" (and nixops modify).

[mbbx6spp]

@cwp I shouldn't have to expose my internal instances with public IP addresses just to use a deployment tool though. That is not acceptable to me. And I am not sure why that is deemed appropriate.

[mbbx6spp]

@cwp ooooh I will investigate the nixops create avenue, though the logic is inside EC2MachineState#get_ssh_name itself :(

[mbbx6spp]

Also I might want to SSH using private for one machine definition in the same deployment and public for another, but maybe that is being overly flexible for little gain at that point. I could live with public or private flagged whole deployments personally, which would be a huge improvement over the current limitation.

[mbbx6spp]

My current workaround is to update the sqlite database publicIPv4 property of the resource with the private IP, which is far from ideal as I do not want to use a patched version of nixops to avoid maintaining upstream changes with it.

[aszlig]

Okay, in theory this could be implemented in set_common_state(), but that won't suffice, because it only propagates the definitions from set_common_state() when doing create(), so it will not work if you do nixops ssh before the first deploy.

So, IIUC the pragmatic approach would be to have a list of addresses which ssh_util tries in order (something like [private_ipv4, public_ipv4]), right?

That would basically allow the use the same unmodified deployment state, but going through different routes (one using the private address, one using the public address).

Also, is it necessary to restrict this, like for example allowing only private addresses but under no circumstances try to connect using the public address?

[cwp]

@mbbx6spp Agreed, nixops shouldn't require public IP addresses.

Another question: Do we need to use the raw IP address? Why not use the DNS name?

That would simplify things, since Amazon will resolve the public DNS name to the private IP address for queries originating from within EC2. Thus, if deployment.ec2.associatePublicIpAddress is false, we always use the private DNS name, but if it's true, we use the public DNS name.

I've tried just making get_ssh_name always return self.public_dns_name, but that didn't work, and I didn't have time to investigate.

[edolstra]

@mbbx6spp Adding a usePrivateIpAddress option sound good to me. And it should be enabled by default if associatePublicIpAddress is false.

Regarding the get_ssh_name issue, we can do one of:

• Add a ssh_name field to the database to contain the IP address or DNS name to use to connect to the machine. This could be a generic thing in MachineState.
• Store the value of usePrivateIpAddress (in EC2State) to allow EC2State.get_ssh_name to return the right IP address.

[mbbx6spp]

@cwp @edolstra excellent suggestions. I will work on it this evening after work. Thanks for the advise. :) :) :)

[andrewlmurray]

I ran into this issue this evening and have submitted a PR which fixes the problem.

@edolstra My solution is pretty similar to what you've suggested here but simply adding a ssh_name wasn't sufficient.

Some of the intent described in the defn needs to be persisted in the machine state to do this without major refactoring so that _wait_for_ip knows what to wait for and how to update keys. (this is called during creation and starting)

It seems this is already being done for other intents such as the spot price so this seems consistent at least.

[andrewlmurray]

@rbvermaa FYI This ticket was addressed by #287

[joamaki]

Thanks Andrew!

[ip1981]

@mbbx6spp Adding a usePrivateIpAddress option sound good to me. And it should be enabled by default if associatePublicIpAddress is false.

or elasticIPv4 is not set :-)