Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow GCE VMs to be reached by private IP address if specified #1124

wants to merge 2 commits into from


Copy link

@mwilsoninsight mwilsoninsight commented Apr 4, 2019

Related issue: #829

…s from gsutil on a provisioning machine, and links to nixos-cloud-imgages github
Copy link

@AmineChikhaoui AmineChikhaoui commented Apr 10, 2019

Can you share the network configuration needed to get this working. I would like to see if it's possible to add a test for this or at least add a sentence or two in the option description about what needs to be done.

Copy link
Contributor Author

@mwilsoninsight mwilsoninsight commented Apr 10, 2019

I plan on setting up such an infrastructure shortly and I'll share the details here.

Currently I just tested for the expected failure (can provision, but not reach GCE vm at private IP address when running nixops from my laptop).

My proposed setup for testing for success is to build a VPN endpoint in GCE, connect it to a VPN endpoint on-prem (I spoofed a small on-prem in Azure for the time being), and then provision a GCE vm with no public IP address- if all goes well, connection to the GCE vm should succeed using a private IP from an "on-prem" network VPN'd to GCE

@mwilsoninsight mwilsoninsight force-pushed the mwilsoninsight:gcePrivateIP branch from 0c0d605 to ceed6ef Apr 15, 2019
Copy link
Contributor Author

@mwilsoninsight mwilsoninsight commented Apr 15, 2019

Okay- I changed around the git history for clarity's sake.

Had to correct known_hosts accommodations in a number of places and get the correct order of operations in the many 'if' statements under GCEState::create_node(), but I think this works.

My test infrastructure (built out by-hand, prior to testing if private-IP-space works):

 _______________________           ___________________
| on prem ( |         | GCE ( |
|  ______________       |         |  ______________   |
| | vpn endpoint <------------------> vpn endpoint |  |
|  --------------       |         |  --------------   |
|  __________________   |          -------------------
| | nixops mgmt srvr |  |
|  ------------------   |

From here, you can specify options as you see fit and get expected behavior

One cautionary note:
GCE will not allow machines with private IPs only to reach the outside internet, so a workaround for the time being is to provision them publicly first and then modify the deployment so your private VMs have no public IP association (I mention this because on first deployment, nix tries to curl out to the public cache, but fails even though firewall permissions allow the communication- likely because no public IP exists in this case). Following that up with a final nixops deploy should get the job done

@mwilsoninsight mwilsoninsight force-pushed the mwilsoninsight:gcePrivateIP branch from ceed6ef to d41e042 Apr 15, 2019
Copy link

@grahamc grahamc commented Mar 26, 2020


Thank you for this PR.

In the past several months, some major changes have taken place in

  1. Backends have been removed, preferring a plugin-based architecture.
    Here are some of them:

  2. NixOps Core has been updated to be Python 3 only, and at the
    same time, MyPy type hints have been added and are now strictly
    required during CI.

This is all accumulating in to what I hope will be a NixOps 2.0
. There is a tracking issue for that:
#1242 . It is possible that
more core changes will be made to NixOps for this release, with a
focus on simplifying NixOps core and making it easier to use and work

My hope is that by adding types and more thorough automated testing,
it will be easier for contributors to make improvements, and for
contributions like this one to merge in the future.

However, because of the major changes, it has become likely that this
PR cannot merge right now as it is. The backlog of now-unmergable PRs
makes it hard to see which ones are being kept up to date.

If you would like to see this merge, please bring it up to date with
master and reopen it
. If the or mypy type checking fails, please
correct any issues and then reopen it. I will be looking primarily at
open PRs whose tests are all green.

Thank you again for the work you've done here, I am sorry to be
closing it now.


@grahamc grahamc closed this Mar 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.