"Unable to find an etc directory with fstab" upon launching NixOS on EC2

[Zachaccino]

Brief Description:

I built a NixOS image locally and launched it using AWS EC2. An error "ClientError: Unable to find an etc directory with fstab." was raised during launching the instance.

Might not be a bug on NixOS but I barely able to find anyone talking about this issue.

Steps to reproduce the behavior:

1. Inside the nixpkgs directory. Build an image using this command: nix-build -A amazonImage.x86_64-linux nixos/release.nix

2. The image built is stored in ./result directory named nixos-amazon-image-20.03pre130979.gfedcba-x86_64-linux.vhd

3. Upload this image to AWS S3 storage.

4. Use awscli to import and launch the image from S3.

5. Use awscli to check the status of the launch process and the error was produced here.

Expected behavior

The NixOS image is successfully launched on EC2 and running normally.

Screenshots

This is the output when I was checking the launch process. The status message indicates the error.

$ aws ec2 describe-import-image-tasks --import-task-ids import-ami-******************
{
    "ImportImageTasks": [
        {
            "Status": "deleted",
            "SnapshotDetails": [
                {
                    "Status": "completed",
                    "UserBucket": {
                        "S3Bucket": "my-amis",
                        "S3Key": "nixos-amazon-image-20.03pre130979.gfedcba-x86_64-linux.vhd"
                    },
                    "DiskImageSize": 1153721856.0,
                    "Description": "Test Server Image",
                    "Format": "VHD"
                }
            ],
            "Description": "My Server",
            "StatusMessage": "ClientError: Unable to find an etc directory with fstab.",
            "ImportTaskId": "import-ami-*****************"
        }
    ]
}

Additional context

NixOS Version: 19.09
Nixpkgs Version: Just updated within a week.

Failed Attempts

1. Used a clean install of Virtual Box NixOS to build the image, same error.
2. Used ova format instead of amazonImage like this: nix-build -A ova.x86_64-linux nixos/release.nix, same error.
3. Used an older version (19.03) to build the image, same error.
4. I opened the image using Virtual Box and I can confirm that the /etc/fstab is there.
5. The devs talked about this issue back in May 2019 and didn't seem to have a solution: https://logs.nix.samueldr.com/nixos-dev/2019-05-10

Metadata

Something went wrong when retrieving Metadata..

nix: src/libexpr/attr-set.hh:54: void nix::Bindings::push_back(const nix::Attr&): Assertion `size_ < capacity_' failed.
zsh: abort (core dumped)  nix run nixpkgs.nix-info -c nix-info -m

Thanks everyone.

[Zachaccino]

The problem was resolved by:

1. Build the script using nix-build -A amazonImage.x86_64-linux nixos/release.nix in nixpkgs directory.

2. Use create-amis.sh BUILT_IMAGE_PATH with minor modifications to create-amis.sh can successfully upload the image to EC2.

[ghost]

The problem was resolved by:

1. Build the script using `nix-build -A amazonImage.x86_64-linux nixos/release.nix` in nixpkgs directory.

2. Use `create-amis.sh BUILT_IMAGE_PATH` with minor modifications to `create-amis.sh` can successfully upload the image to EC2.

Would you mind sharing the modification you made to upload ? I'm facing the exact same issue.

[Zachaccino]

So after you build the image using the command:

nix-build -A amazonImage.x86_64-linux nixos/release.nix

You can use the script here to upload it to your AWS.
create-amis.sh

The modifications in the script should be made here:

state_dir=$HOME/amis/ec2-images
home_region=eu-west-1
bucket=nixos-amis

regions=(eu-west-1 eu-west-2 eu-west-3 eu-central-1 eu-north-1
         us-east-1 us-east-2 us-west-1 us-west-2
         ca-central-1
         ap-southeast-1 ap-southeast-2 ap-northeast-1 ap-northeast-2
         ap-south-1 ap-east-1
         sa-east-1)

You can probably leave the state_dir untouched. It's just used to save the upload progress, task_id, snapshot_id etc. So it can be anywhere that is accessible.

home_region should be the AWS home region of your choice.

bucket is the name of the S3 bucket you want to upload to. Make sure you created a bucket for that.

regions is a bunch of other regions you want your AMI to be uploaded to that is not the home region. I just left it the same with my home region tho and it worked...

There is another line at line 38:

store_path=$1

Note that this is using a command-line argument, so you can pass the store_path like this: create-amis.sh BUILT_IMAGE_PATH. This is the folder where your image is stored. To be clear, as long as your store path allow the script to access the image-info.json file in the following way, then it should be fine.

store_path/nix-support/image-info.json

After that, you should be able to execute the script and the custom AMI can be found is AMIs tabs in your EC2 management page. You can choose to launch it on AWS Console or using AWS CLI.

If you decided to do it manually. Then here is the summary of what needs to be done:

1. Upload the Image to S3
2. Import as Snapshot from S3
3. Register the Snapshot.
4. Launch the Snapshot.

I was having that error coz I forgot to register the snapshot. And register the snapshot gives it an EBS volume, and this created the fstab.

[ghost]

So after you build the image using the command:

nix-build -A amazonImage.x86_64-linux nixos/release.nix

You can use the script here to upload it to your AWS.
create-amis.sh

The modifications in the script should be made here:

state_dir=$HOME/amis/ec2-images
home_region=eu-west-1
bucket=nixos-amis

regions=(eu-west-1 eu-west-2 eu-west-3 eu-central-1 eu-north-1
         us-east-1 us-east-2 us-west-1 us-west-2
         ca-central-1
         ap-southeast-1 ap-southeast-2 ap-northeast-1 ap-northeast-2
         ap-south-1 ap-east-1
         sa-east-1)

You can probably leave the state_dir untouched. It's just used to save the upload progress, task_id, snapshot_id etc. So it can be anywhere that is accessible.

home_region should be the AWS home region of your choice.

bucket is the name of the S3 bucket you want to upload to. Make sure you created a bucket for that.

regions is a bunch of other regions you want your AMI to be uploaded to that is not the home region. I just left it the same with my home region tho and it worked...

There is another line at line 38:

store_path=$1

Note that this is using a command-line argument, so you can pass the store_path like this: create-amis.sh BUILT_IMAGE_PATH. This is the folder where your image is stored. To be clear, as long as your store path allow the script to access the image-info.json file in the following way, then it should be fine.

store_path/nix-support/image-info.json

After that, you should be able to execute the script and the custom AMI can be found is AMIs tabs in your EC2 management page. You can choose to launch it on AWS Console or using AWS CLI.

If you decided to do it manually. Then here is the summary of what needs to be done:

1. Upload the Image to S3

2. Import as Snapshot from S3

3. Register the Snapshot.

4. Launch the Snapshot.

I was having that error coz I forgot to register the snapshot. And register the snapshot gives it an EBS volume, and this created the fstab.

Thank you very much, it has indeed worked :).

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/fstab-error-while-creating-a-custom-amazon-ec2-ami/6697/2

[bouk]

For those who come across this: you should use aws ec2 import-snapshot, not aws ec2 import-image

[baloo]

If you wanted to use import-image and not rely on ebs (as with import-snapshot).

You may get aws to stop complaining about the fstab error with:

environment.etc.fstab.mode = "0644";

By default the fstab will be a symlink to the nix store but aws when it imports images will break the abstraction and make various changes to the image, including changes to the fstab.

But even if you were to fix the fstab issue, you'd get stuck further down the pipe. (it get stuck and operation will never complete).
This is because of https://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-image-import.html#import-modify-vm

AWS also wants to install drivers and change the initrd. Which obviously won't work.

[baloo]

Raised the issue with support, not much they could do, but they gave me an "official" image checker:
https://gist.github.com/giuliocalzolari/8ad1ab693e82f77646d6857cef732de6

I didn't take time to run it on our generated images yet.