Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Commits on Nov 24, 2015
  1. @rbvermaa

    Merge pull request #359 from ninegua/master

    rbvermaa committed
    Change sbin/mount.vboxsf to bin/mount.vboxsf
Commits on Nov 17, 2015
  1. @ninegua
  2. @rbvermaa

    Print error message next to the error code for EC2 retries. Code is n…

    rbvermaa committed
    …ot enough information, most useful information is in the message.
Commits on Nov 16, 2015
  1. @aszlig

    scripts/nixops: Properly encode command in op_ssh.

    aszlig committed
    Regression introduced by ffbbe13.
    Using "ssh some_machine 'echo foo > /dev/null'" works, but in our case
    using "nixops ssh some_machine 'echo foo > /dev/null'" bails out with an
    The reason for this is that without allow_ssh_args being set, the
    command will end up getting escaped and passed as a single joined
    argument to OpenSSH.
    However, we don't need to do this and using allow_ssh_args will prevent
    that, especially because we already have splitted off SSH arguments
    using split_openssh_args().
    Signed-off-by: aszlig <>
    Reported-by: Eelco Dolstra <>
Commits on Nov 11, 2015
  1. @edolstra
  2. @edolstra

    Merge pull request #358 from mayflower/fix/ssh_flags_none_backend

    edolstra committed
    Pass deployment.targetPort to ssh for none backend
  3. @fadenb

    Make get_ssh_flags return flags from super class

    fadenb committed
    Adopted from fix for issue #277
    Otherwise the deployment.targetPort is not passed to ssh
Commits on Oct 31, 2015
  1. @aszlig

    backends/hetzner: Set state_version on install.

    aszlig committed
    This reverts commit f2bf137.
    Because we now have config['nixosRelease'], we can properly set the
    state version before we even switch to the configuration even in rescue.
    So we no longer need the workaround of omitting the state version check
    while in rescue, because at that time the machine should already have a
    Signed-off-by: aszlig <>
  2. @aszlig

    backends/hetzner: Create /mnt/run if not existing.

    aszlig committed
    We need to create this in _bootstrap_rescue_for_existing_system() to
    make sure the bootstrap even works for NixOS 14.12, because when
    bootstrapping, /run isn't yet available.
    Signed-off-by: aszlig <>
  3. @aszlig

    eval-machine-info: Switch to system.nixosRelease.

    aszlig committed
    "system.nixosVersion" contains a version suffix, so for our purposes
    it can lead to bugs if we want to compare it against the stateVersion.
    In order to avoid confusion, the attribute exported by eval-machine-
    info is now called nixosRelease rather than nixosVersion, because it
    corresponds to system.nixosRelease introduced in NixOS/nixpkgs@d166c85.
    If there is no system.nixosRelease is available, we're falling back to
    using nixosVersion and stripping off the nixosVersionSuffix so if we use
    an older release we still have the same value, because
    system.nixosVersion is just nixosRelease + nixosVersionSuffix.
    Tested against release-15.09 and release-14.12 using the VM tests.
    Signed-off-by: aszlig <>
Commits on Oct 30, 2015
  1. @aszlig

    tests/none: Fix use of "nixops ssh".

    aszlig committed
    We no longer need to use "--" to pass SSH options (in this case "-v").
    Signed-off-by: aszlig <>
  2. @aszlig

    doc: Fix note about OpenSSH options.

    aszlig committed
    We no longer need -- to pass flags to OpenSSH options.
    Signed-off-by: aszlig <>
  3. @aszlig

    scripts/nixos: Fix handling SSH args in do_ssh.

    aszlig committed
    Regression introduced by da1a81a.
    This really doesn't pass the username to the SSH master process but just
    the ssh call that's using the master, which then will obviously fail.
    The last two commits were a preparation for improving this, so on one
    side we have the user keyword argument, which passes the user name
    directly as user@machine to the SSH master process.
    On the other side we now have split_openssh_args(), which we can use to
    distinguish between a SSH command and a SSH flag. The reason this is
    necessary is that we don't want to add the command to the SSH master
    process but of course we want to have flags appended to the master
    While this certainly increases complexity, but if we really want to pass
    SSH options verbatim, we don't have any other option except adding them
    like this:
    nixops ssh --ssh-flags="-i /foo/bar" my_shiny_command
    However, this would introduce a backwards-incompatibility, because SSH
    flags passed verbatim will then throw an error.
    And I've stated the reason why we can't avoid starting an SSH master
    process in da1a81a already.
    On the upside we gain one more thing (except fixing the functionality)
    and it should be mostly backwards-compatible:
    We can finally do the following: nixops ssh shiny_machine ls -l
    The reason for this is that we now use argparse.REMAINDER, so we don't
    get nasty quirks and escaping goofs if "--" is used.
    And similar to argparse.REMAINDER is the actual OpenSSH argument parsing
    where after the last non-flag the target and the command are used
    without trying to parse additional arguments.
    The reason why it's only "mostly" backwards-compatible is that something
    like this:
    nixops ssh foo -- -v ls -l
    Has now to be called like this:
    nixops ssh foo -v ls -l
    Signed-off-by: aszlig <>
  4. @aszlig

    ssh_util: Add new split_openssh_args() method.

    aszlig committed
    This is useful if we want to distinguish from a bunch of SSH flags
    whether the argument is a normal OpenSSH flag or whether it's an actual
    command that's passed to OpenSSH.
    We do this by simply recursing through the list of flags and eat up all
    flags and non-flags until we either get a command or a "--" in which
    case we treat everything afterwards as a command.
    Note that this is specifically for use in NixOps, because OpenSSH not
    only has a command argument but also a target argument which we don't
    need so we don't handle it.
    Signed-off-by: aszlig <>
  5. @aszlig

    ssh_util: Add 'user' keyword arg to run_command().

    aszlig committed
    So far we're using -l in op_ssh(), but unfortunately that's not working
    for the SSHMaster.
    This keyword arg now ensures that the master gets a proper target with
    the specified username, so we no longer need to pass -l.
    Signed-off-by: aszlig <>
  6. @rbvermaa
  7. @aszlig

    hetzner-bootstrap: Allow to easily enter chroot.

    aszlig committed
    So far entering the chroot always has been a bit awkward:
    chroot /mnt /nix/var/nix/profiles/system/sw/bin/bash
    And this worked only in a very limited way, because we also would need
    to set a proper $PATH like /nix/var/nix/profiles/system/sw/bin.
    But even that will still only enter the BASH shell and doesn't respect
    the preference of the root user of the target system.
    So we now create a wrapper called nixos-enter that essentially uses su
    to change to the root user using the right login shell.
    But in order to use the real user's environment we also need to make
    sure that we have things such as /run/current-system, for which the
    method _bootstrap_rescue_for_existing_system() has just been introduced.
    It also adds a small banner to /etc/motd so that the user will be aware
    of the existence of the nixos-enter command.
    Signed-off-by: aszlig <>
  8. @aszlig

    backends/hetzner: Fix PEP8 compliance.

    aszlig committed
    Fixes the following errors from the PEP8 checking tool:
    160:11: E121 continuation line under-indented for hanging indent
    230:80: E501 line too long (80 > 79 characters)
    702:80: E501 line too long (80 > 79 characters)
    Signed-off-by: aszlig <>
  9. @aszlig

    hetzner-bootstrap: Replace ensureDir with mkdir.

    aszlig committed
    Using ensureDir is deprecated since NixOS/nixpkgs@b3b9c51.
    Signed-off-by: aszlig <>
  10. @aszlig

    hetzner-bootstrap: Fix wrapping Nix inside chroot.

    aszlig committed
    So far we've made wrappers for the store paths used for building the
    bootstrap installer. But those paths aren't necessarily available in the
    store of the live system after the first deploy (especially after a GC).
    We now check whether the "true" command can be executed within the
    system path of the chrooted target system and if that really returns
    true, we use the system profile link to get to the correct store paths.
    That way it's still possible to deploy and/or rollback while the machine
    is in rescue state no matter if there was a GC in-between.
    We also now directly execve() the wrapped file so we won't do more forks
    than necessary (we already have one additional fork for the execution of
    Signed-off-by: aszlig <>
  11. @aszlig

    tests: Drop <nixos> from NIX_PATH.

    aszlig committed
    This is obsolete since a long time now and no longer used in the NixOps
    code either, so let's remove it from the tests as well.
    Signed-off-by: aszlig <>
Commits on Oct 29, 2015
  1. @aszlig

    ssh_util: Move host key ignore to Hetzner backend.

    aszlig committed
    While the Hetzner backend is the only backend which needs to have
    password authentication right now, ignoring the host key really should
    be done in the backend implementation rather than unconditionally in
    ssh_util once a password is set.
    This should make the implementation more generic as we can still pass
    additional options from backends without fiddling around with removing
    options from that end.
    Signed-off-by: aszlig <>
  2. @aszlig

    ssh_util: Revert adding invoke_shell().

    aszlig committed
    This reverts commit 6781c0a.
    Bypassing the SSH master isn't going to work for reasons stated in
    So we can safely drop this implementation to prevent anyone from even
    attempting to rely on it.
    Signed-off-by: aszlig <>
  3. @aszlig

    scripts/nixops: Use m.ssh.run_command in op_ssh().

    aszlig committed
    We already had that before e57686c.
    The reason this has been changed was to be able to supply a user name,
    which I had accidentally dropped much earlier in 9cca2d8.
    So the actual problem here is that we can't ssh into a machine using
    passwords and _not_ have a SSH master. This is because the SSH_ASKPASS
    helper is only called when there is no TTY available, which is kind of a
    contradiction to what the "nixops ssh" command should actually do.
    Of course, we could even manage to do it without a master socket, but
    that would involve creating a PTY and pipe everything to SSH forth and
    back (and we also would need a bit of signal handling). Which pretty
    much boils down to increasing the surface for bugs.
    Instead of just reverting back to the old version, we actually now pass
    the username along using OpenSSH's -l flag.
    Signed-off-by: aszlig <>
  4. @aszlig

    deployment: Fix state_version warning for RESCUE.

    aszlig committed
    Hetzner machines are deployed using *only* an (almost) empty store
    directly from the rescue system. With *only* I mean there is no /etc (or
    to be more exact: only a dummy /etc), so getting the value of
    /etc/os-release will not only fail but also won't work at all because
    there is no pre-existing deployment on the system.
    As 877158c introduced a warning if NixOps is unable to determine the
    state_version, we simply shut up the warning whenever we're in rescue
    An alternative would be to not even try to get the version when in
    RESCUE, but in cases where the deployment is done to a pre-existing
    rescue system with a custom kickstart script it's actually legit to
    fetch the version even in RESCUE state.
    Signed-off-by: aszlig <>
  5. @aszlig

    tests/hetzner: Improve instrumentation for rescue.

    aszlig committed
    The <nixpkgs/nixos/modules/testing/test-instrumentation.nix> is used as
    a reference here, and we want to get as closely as possible to that so
    we have the full testing API at hand even for the rescue system.
    This also gets rid of the annoying getty at the serial console.
    Signed-off-by: aszlig <>
  6. @aszlig

    tests/hetzner: Switch to kernel 4.2.3.

    aszlig committed
    We want our live system to resemble the real Hetzner rescue system as
    closely as possible. And the Hetzner rescue system is a Debian Wheezy
    system at the moment with a more recent 4.x kernel but will very soon be
    a Debian Jessie with a 4.x kernel as well.
    So switching to 4.2.3 is very reasonable to be as close as possible to
    the real system.
    On the implementation side however, I could have used the
    debClosureGenerator for the backports. Unfortunately, packages on
    backports do frequently change and so does Packages.xz, so for every
    change of a backport package we need to update the hash of the
    Packages.xz file.
    Because of that I opted for using the packages directly because we don't
    have a whole lot of dependencies to cope with.
    Signed-off-by: aszlig <>
  7. @aszlig

    tests/hetzner: Test if SSH works in rescue system.

    aszlig committed
    This actually tests the implementation done in 6781c0a and e856016.
    We want to make sure that even when we have to rely on a askpass helper,
    the ssh subcommand still works.
    So getting this to work with our current tests it also means that we
    need to generalize the setupAndStartRescue function to wrap a function
    that's solely for the post-boot commands executed on the rescue system,
    which we need to reboot into the rescue system *after* the machine has
    had its first deployment.
    Signed-off-by: aszlig <>
  8. @aszlig

    scripts/nixops: Switch to using SSH.invoke_shell.

    aszlig committed
    We now should have a ssh subcommand which works the same way as the
    ssh-for-each subcommand, except that it doesn't rely on a SSH master
    socket to be present.
    This not only has the benefits of making the codebase more DRY but also
    things like ssh into a machine that's in rescue mode now work properly.
    Signed-off-by: aszlig <>
  9. @aszlig

    ssh_util: Add invoke_shell() to bypass SSHMaster.

    aszlig committed
    Or rather a spawn() method within the SSHMaster class to do direct
    connections to the host.
    The reason for this is that the current implementation of the ssh
    subcommand doesn't use all options that can be available for a
    particular machine.
    One example would be password authentication while having a machine in
    the Hetzner rescue system. Currently we do not correctly pass the
    askpass helper to SSH and the connection just fails while the system is
    in rescue mode.
    Signed-off-by: aszlig <>
  10. @aszlig

    ssh_util: Reconnect on dead SSH master socket.

    aszlig committed
    This took me hours to debug, because I was originally hunting for the
    issue behind the test failure of the Hetzner backend:
    It did not fail on my own Hydra instance or my build machines, because
    the build of the target machines was fast enough so the gap between the
    last connection to the master and the next connection when copying the
    closure to the target machines was less than 600 seconds.
    Those 600 seconds is the ControlPersist timeout for the SSH master
    connection and was introduced in bbe469a. On a slow or highly loaded
    system however, this gap is too long, the master socket dies in-between
    and a SSH client connecting to the non-existing sockets waits forever.
    Fortunately, OpenSSH cleans up the socket, so we can easily test whether
    the socket is still available.
    Also, OpenSSH only creates the socket once the connection is
    established, so while we usually start the SSHMaster when we want to
    connect anyway, it's not necessarily the case.
    So we wait 60 seconds until the master socket is created, so that
    is_alive() doesn't return False in-between. Which in turn would lead to
    the startup of another master process while the first one is still
    starting up.
    Signed-off-by: aszlig <>
  11. @aszlig

    tests: Provide a name for the VM tests.

    aszlig committed
    Having "vm-test-run-unnamed" sounds a bit ugly, so let's give them
    proper names.
    Signed-off-by: aszlig <>
  12. @aszlig

    tests/hetzner: Print output of debootstrap.log.

    aszlig committed
    Very useful to debug failing builds of the rescue image like this one:
    Signed-off-by: aszlig <>
Commits on Oct 27, 2015
  1. @edolstra

    Fix determining system.stateVersion

    edolstra committed
    We didn't handle "14.12pre-git", causing the stateVersion to be set to
    15.09 on subsequent deployments.
Commits on Oct 26, 2015
  1. @edolstra

    Merge pull request #201 from aszlig/none-improvements

    edolstra committed
    Use dedicated SSH keypair for "none" backend.
Something went wrong with that request. Please try again.