NixOS Azure image

[rbvermaa]

I see that NixOS has images available for some of the popular clouds, but not for Azure. I’d love to see a NixOS image on VM Depot. VM Depot is a community managed repository of virtual machines for Azure. We have something like 8 different Linux distros and around 1700 images built on those distros (ranging from developer stacks through to end user applications).

The first step to getting folks to publish NixOS based images is to have a base distribution of NixOS available and, preferably, updated every time there is an official release of NixOS. Is anyone here interested in creating and upload an image to VM Depot? I’m happy to help guide the process.

Some common questions for the curious:

1.  Does it cost anything to store an image on VM Depot? No – all storage costs are paid by Microsoft Open Technologies, Inc (my employer)
   

2.  Does it cost anything to publish an image on VM Depot? Probably not - You will need an Azure subscription to temporarily store the image and there will be bandwidth charges for the initial copy. However, there are mechanisms by which we can ensure open source projects have sufficient Azure credits to do this without receiving a bill. Create a free Windows Azure trial subscription to get started straight away (one month, $200 credit)
   

3.  Are there an restrictions on what can be uploaded to VM Depot? – Short answer – if its open source then no there are no restrictions. Long answer is in the Terms of Use http://vmdepot.msopentech.com/ToU.htm
   

4.  Why would I want to upload an image to VM Depot? It is easy for people to deploy a VM from VM Depot to Azure. This means it is easy for people to experiment with your project. More people experimenting means more users, more users means more potential contributors to the project and more potential customers for those employing contributors.
   

5.  How do I get started creating a new VM based on an existing distribution? See http://msopentech.com/blog/2014/05/14/deploy-customize-freebsd-virtual-machine-image-microsoft-azure/ for a description of the general process (need not be FreeBSD as the starting image, the process is the same for any of the other images available)
   

6.  How do I get started creating a new base distribution VM? http://azure.microsoft.com/en-us/documentation/articles/virtual-machines-linux-tutorial/
   

7.  Who can I contact for assistance? Ross Gardler – ross.gardler@microsoft.com
   

[wmertens]

I'm interested in working on this later this week

[rbvermaa]

Azure usually works with an agent: https://github.com/Azure/WALinuxAgent

[rbvermaa]

The agent gets the password (that is entered in the console) from a file on the filesystem:

# cat /var/lib/waagent/ovf-env.xml                                                                                                                                                                                                                                             
<?xml version="1.0" encoding="utf-8"?>                                                                                                                                                                                                                                                    
<Environment xmlns="http://schemas.dmtf.org/ovf/environment/1" xmlns:oe="http://schemas.dmtf.org/ovf/environment/1" xmlns:wa="http://schemas.microsoft.com/windowsazure" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">                                                           

    <wa:ProvisioningSection><wa:Version>1.0</wa:Version><LinuxProvisioningConfigurationSet xmlns="http://schemas.microsoft.com/windowsazure" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><ConfigurationSetType>LinuxProvisioningConfiguration</ConfigurationSetType><HostName>nixos-test</HostName><UserName>azureuser</UserName><UserPassword>somepassword</UserPassword><DisableSshPasswordAuthentication>false</DisableSshPasswordAuthentication></LinuxProvisioningConfigurationSet></wa:ProvisioningSection>                                                           

  <wa:PlatformSettingsSection><wa:Version>1.0</wa:Version><PlatformSettings xmlns="http://schemas.microsoft.com/windowsazure" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><KmsServerHostname>kms.core.windows.net</KmsServerHostname><ProvisionGuestAgent>false</ProvisionGuestAgent><GuestAgentPackageName i:nil="true" /></PlatformSettings></wa:PlatformSettingsSection>                                                                                                                                                                                                 
</Environment>       

I think it is being written by the hypervisor on the root filesystem.

Personally, I don't like agents like this which create users and change system configuration.

[rbvermaa]

@wmertens do you mean next week, or this weekend?

[wmertens]

@rbvermaa from Tuesday onwards

I don't like agents like that either but I assume they can be configured to only do what's needed to allow access to NixOS... (eg ssh key for root, which gets reset on the next run?

[rbvermaa]

@rgardler Trying to run a generated NixOS image on Azure. However, it seems to be stuck in provisioning stage. Any way to e.g. get console output or more logging?

$ azure vm show nixos-test1                                            
info:    Executing command vm show                                                                                                         
+ Getting virtual machines                                                                                                                 
data:    DNSName "nixos-test1.cloudapp.net"                                                                                                
data:    Location "West Europe"                                                                                                            
data:    VMName "nixos-test1"                                                                                                              
data:    IPAddress "100.73.32.39"                                                                                                          
data:    InstanceStatus "Provisioning"                                                                                                     
data:    InstanceSize "Large"                                                                                                              
data:    Image "nixos-test"                                                                                                                
data:    OSDisk hostCaching "ReadWrite"                                                                                                    
data:    OSDisk name "nixos-test1-nixos-test1-2014-09-06"                                                                                  
data:    OSDisk mediaLink "https://portalvhdsrhwstp9614rfb.blob.core.windows.net/vm-images/kewyuimo.ve4201409061819350700.vhd"             
data:    OSDisk sourceImageName "nixos-test"                                                                                               
data:    OSDisk operatingSystem "Linux"                                                                                                    
data:    Network Endpoints 0 localPort 22                                                                                                  
data:    Network Endpoints 0 name "SSH"                                                                                                    
data:    Network Endpoints 0 port 22                                                                                                       
data:    Network Endpoints 0 protocol "tcp"                                                                                                 
data:    Network Endpoints 0 virtualIPAddress "23.97.227.143"                                                                              
data:    Network Endpoints 0 enableDirectServerReturn false                                                                                
info:    vm show command OK                                   

[rbvermaa]

@rgardler No technical support for 'Free Trial' accounts? Seriously?

[rbvermaa]

Progressing... I can SSH into a NixOS machine that is in provisioning status, but only from another machine that has been provisioned on Azure.

[SorraTheOrc]

@rbvermaa I'm here what more do you need in the way of technical support ;-) (probably someone with more technical skills than me, but at least I know who to ask)

You can use a working Linux machine in Azure (for example standard Ubuntu machine based on azure gallery image) and attach the OS disk of failed NixOS as a data disk to this Ubuntu VM. Once you checked the log, you can detach and delete the data disk.

[wmertens]

@rbvermaa what is the state of this?

[rbvermaa]

@wmertens See my latest comment. Also, the image build function currently needs to have a ssh-pub-key in the NIX_PATH, which hardcodes the SSH public key for root.

The metadata 'cdrom' is mounted at /metadata. Next step should be to parse the info there and process SSH public key that is passed to the instance, and make it appear in /root/.ssh/authorized_keys. This would eliminate the need to hardcode the SSH public key in the image.

Then we need to figure out why the machine does not get into 'running' state, which seems to prevent the Azure firewall from applying the firewall rules. They seem to be passing some API endpoint address via 'option 245' in DHCP (https://github.com/Azure/WALinuxAgent/blob/2.0/waagent#L4094), which is then used to pass some status information to the API. This is done in the agent somewhere I think.

[Phreedom]

The current status is: if you provide the right custom data, the image boots, obtains IPs and otherwise works. Agent is still a todo item and without it, the vm is going to be permanently "provisioning" and will fail load balancer health checks. Unfortunately, the agent is a mess. It insists on rewriting files, making substantial config changes. It isn't that hard to stub/comment out the unwanted things, but it's a very flaky solution because there are many of them and an agent update can arrive at any time. Reimplementing the relevant functionality also doesn't look very feasible either :( Luckily, we still have basic functionality working without it, so at least the chance of bricking hundreds of vms is low.

[domenkozar]

Couldn't we just package https://github.com/Azure/WALinuxAgent ourselves?

[Phreedom]

The problem is not about packaging it, but that it needs heavy patching in so many places. Thus, not only it's a headache, but everytime there's a new release, there's quite some work to do review the patch. I have opted for now for a dumb solution which at least ensures the image boots and is reachable. Of course this isn't the final decision or anything. This is a bare minimum which doesn't look too ugly and unblocks the development.

[dcht00]

+1, was just offered free Azure and thought of playing with NixOS first

[ericbmerritt]

@Phreedom whats the dumb solution?

[Phreedom]

no agent, ssh keys are provided via metadata and fetched by a simple systemd job

[ericbmerritt]

@Phreedom any chance you could walk through that in a bit more detail?

[lizzha]

Hi all, I'm from the team that owns the WALinuxAgent, we would like to enable the agent to support NixOS and enable NixOS running in Azure. I was learning NixOS but not much familiar yet, so it's great to see the discussion here.
Would anyone be willing to help and work with us to come out a solution?

[Phreedom]

@lizzha : I will be doing some azure-related work in
a day or two and will contact you to discuss potential
solutions and patches.

@*ericbmerritt[1] Sorry for not replying sooner. I
have been too busy lately. *

---

[1] https://github.com/ericbmerritt

[Phreedom]

On Thursday, November 26, 2015 21:59:31 lizzha wrote:

Hi all, I'm from the team that owns the WALinuxAgent, we would like to
enable the agent to support NixOS and enable NixOS running in Azure. I was
learning NixOS but not much familiar yet, so it's great to see the
discussion here. Would anyone be willing to help and work with us to come
out a solution?

---

Reply to this email directly or view it on GitHub:
#3986 (comment)

I have patched WALinuxAgent and added support for it in NixOS azure image. See
6db6718 and
c16f90f

The image can now be correctly provisioned by azure-cli and its status ends up being
"provisioned" instead of "provisioning timed out".

Feedback and testing are welcome. This is a sample script to build the image:
https://github.com/NixOS/nixpkgs/blob/master/nixos/maintainers/scripts/azure/create-azure.sh

Unfortunately, at this moment you are likely to encounter bugs
https://bugs.launchpad.net/qemu/+bug/1490611 and https://github.com/Azure/azure-xplat-cli/issues/2168

To work around them, you may have to revert qemu to 2.2.0, and in azure-image.nix add
"-o subformat=fixed" option to "${pkgs.vmTools.qemu}/bin/qemu-img convert -f raw -O
vpc $diskImage $out/disk.vhd".

[ericbmerritt]

@Phreedom I will attempt to give this a test in the next few days.

[LiliDeng]

@Phreedom
Hi,
I have a few questions to you about build azure nixos vhd/image, thanks!

1. I must run create-azure.sh against a nixos VM to build it, right?
2. How do you install azure-cli in nixos? I install it by source code.
   $ nix-env -iA nixos.iojs
   $ nix-env -iA nixos.python
   $ nix-env --install wget-1.16.3
   $ nix-env --install make-1.0
   $ wget https://github.com/Azure/azure-xplat-cli/releases/download/v0.9.13-December2015/azure-cli.0.9.13.tar.gz
   $ mkdir azure-cli
   $ tar -xvf azure-cli.0.9.13.tar.gz -C ~/azure-cli
   $ cd ~/azure-cli
   $ npm install

If I install it by this command
$ npm install azure-cli -g, firstly, will hit /nix/store is read-only issue, then I run mount -o remount,rw /nix/store, this issue resolved.
Secondly, I hit build error though make is installed, I add make location into $PATH, didn't work, seems it ran which command without output.
gyp ERR! build error
gyp ERR! stack Error: not found: make
gyp ERR! stack at F (/nix/store/ihnlcv6i3q0yfm0kmipaq9svkl4w9kz0-iojs-3.0.0/lib/node_modules/npm/node_modules/which/which.js:72:28)
gyp ERR! stack at E (/nix/store/ihnlcv6i3q0yfm0kmipaq9svkl4w9kz0-iojs-3.0.0/lib/node_modules/npm/node_modules/which/which.js:75:29)
gyp ERR! stack at /nix/store/ihnlcv6i3q0yfm0kmipaq9svkl4w9kz0-iojs-3.0.0/lib/node_modules/npm/node_modules/which/which.js:83:16
gyp ERR! stack at FSReqWrap.oncomplete (fs.js:82:15)
gyp ERR! System Linux 3.18.24
gyp ERR! command "/nix/store/ihnlcv6i3q0yfm0kmipaq9svkl4w9kz0-iojs-3.0.0/bin/iojs" "/nix/store/ihnlcv6i3q0yfm0kmipaq9svkl4w9kz0-iojs-3.0.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild" "--release"
gyp ERR! cwd /nix/store/ihnlcv6i3q0yfm0kmipaq9svkl4w9kz0-iojs-3.0.0/lib/node_modules/azure-cli/node_modules/streamline/node_modules/fibers
gyp ERR! node -v v3.0.0
gyp ERR! node-gyp -v v2.0.2
gyp ERR! not ok

[Phreedom]

@LiliDeng

Sorry for the delay. Packaging azure-cli took some effort.

You can get a shell with azure-cli by running nix-shell -p azure-cli.
You need a very fresh nixpkgs checkout for this.

Also, I have pushed a temporary branch with the qemu patches reverts and other workarounds: https://github.com/Phreedom/nixpkgs/commits/revert-qemu

This is how you could build the image(tested on a nixos machine):

cd $TEMP
git clone https://github.com/Phreedom/nixpkgs.git
cd nixpkgs
#checkout the workaround branch
git checkout revert-qemu

#build the image
NIXOS_CONFIG=$TEMP/nixpkgs/nixos/modules/virtualisation/azure-image.nix  NIX_PATH=$TEMP nix-build '<nixpkgs/nixos>' -A config.system.build.azureImage --argstr system x86_64-linux -o azure --option extra-binary-caches https://hydra.nixos.org -j 4
# the image hopefully ends up in $TEMP/nixpkgs/azure//disk.vhd

# get azure-cli running
NIX_PATH=$TEMP nix-shell -p azure-cli
# azure-cli becomes available in this shell only
azure login

An image built using this branch is available at https://nixos.blob.core.windows.net/images/nixos-unstable-standalone.vhd

[Phreedom]

I have removed the distinction between nixops and "standalone" images, packaged azure-vhd-tools-for-go, updated image generation and upload scripts for maintainers, pushed this into nixos master, regenerated the default image and tested it. The base commit for the current image is 158d723 .

[Phreedom]

cherry-picked all the relevant commits into 16.03 and tested