[Fix GCP plugin] Introduce public GCE Bootstrap Image #1

tewfik-ghariani · 2020-03-18T23:44:39Z

Background

After reading some docs related to running NixOs on GCE, I've seen that we currently have to bootstrap our own image from an object publicly shared in GS according to this : https://nixos.wiki/wiki/Install_NixOS_on_GCE

And at this time, the nixops-gcp plugin is deemed broken specifically due to this particular 'bootstrap-image' resource

Proposed Solution

However, according to the GCP docs, we can actually just bake some images in a given GCP account, and then make the images public.
https://cloud.google.com/compute/docs/images/managing-access-custom-images#share-images-publicly

Even better, we may create image families to allow better flexibility to manage single images. No need to specify the machine name per say but just the family. And under the hood, the maintainer may deprecate old ones and keep the 'latest' up-to-date.

Validation

I started by testing the whole scenario using gcloud commands and it worked like a charm!

Building an image from source

$ gcloud compute images create nixos-18091228a4c4cbb613c-x86-64-linux  \
              --source-uri gs://nixos-cloud-images/nixos-image-18.09.1228.a4c4cbb613c-x86_64-linux.raw.tar.gz \
              --family=nixos-1809

Making the image public

$ gcloud compute images  add-iam-policy-binding nixos-18091228a4c4cbb613c-x86-64-linux \
             --member='allAuthenticatedUsers'                                                                              \
             --role='roles/compute.imageUser'

And then that image may be used publicly so that all nixops users won't have the need to provision their own 'bootstrap-image' resource for every deployment.

$ gcloud compute instances create test-nixos-18  \
            --image-family=nixos-1809                        \
            --zone=europe-west1-c                             \
            --image-project=predictix-operations

Implementation

Implemented the change described above in this codebase and the gce plugins should be working fine now.

The way it works

Extract the public image based on family name and project
Create a volume out of that image [ No need to copy it in every deployment]
Proceed naturally and use that volume as bootDisk

A summary of what has been achieved overall

Deprecated the automatic provisioning of gce-image resource.
Introduced bootstrap images as public image families.
Added 'publicImageProject' option for blockDeviceMapping and gceDisk to specify the project from which publicly available image families are retrieved from.
Controlled the exceptions and error messages

Small Additions

Display GCE volume size in 'nixops-info'
Remove Uniqueness condition on fileSystemsOption
Update some GCP documentation links

ToDo

Update the 'image-family' to point to the 'gce-images.nix' file under nixpkgs

  image-family = import ./gce-images.nix;
  # To be changed to
  # image-family = import <nixpkgs/nixos/modules/virtualisation/gce-images.nix>;

Prepare the images in the nixOs GCP account : Prepare public images in the NixOS GCP account NixOS/infra#110
Update the 'project' variable in the 'gce-images.nix'

Please let me know what do you think about this. If you have any suggestions or recommendations, feel free to share them.
cc @PsyanticY @AmineChikhaoui

cc @rbvermaa Can we consider this as part of NixOS/nixpkgs#6991

… and using it to retrieve images from publicly available image families. Extracting the public image from the target project first then create a volume out of that image [ No need to copy it in every deployment]

…condition on fileSystemsOptions

…ethod

PsyanticY · 2020-03-19T11:44:55Z

nix/gce.nix

@@ -193,7 +200,7 @@ let
    options = {
      gce = mkOption {
        default = null;
-        type = with types; uniq (nullOr (submodule gceDiskOptions));


why remove uniq ?

So that we don't find ourselves in a situation where we have to re-define all options
Example
gce.nix

fileSystems."/data" = { fsType = "xfs"; options = [ "noatime" "nodiratime" ]; autoFormat = true; formatOptions = "-f"; gce.disk = null; gce.disk_name = null; gce.size = lib.mkDefault 500; gce.diskType = "ssd"; gce.encrypt = encrypt; };

partitions-gce.nix

{ fileSystems."/data".gce.disk = lib.mkForce resources.gceDisks."${machine}-data"; fileSystems."/data".gce.disk_name = lib.mkForce "data"; }

This error is raised

error: The unique option `fileSystems./data.gce' is defined multiple times, in `<unknown-file>' and `/nixops/partitions-gce.nix'. (use '--show-trace' to show detailed location information)

Solution

fileSystems."/data".gce = lib.mkOverride 10 { disk = resources.gceDisks."${machine}-data"; disk_name = "data"; size = volumeSize; diskType = "ssd"; encrypt = encrypt; };

Removal of itertems and replacing it with items/values Removal of 'filter' and replacing it with list comprehension Updating relative imports

…rgetEnv to gce

… into poetry2nixify

This reverts commit b7fbd63.

adisbladis · 2020-05-01T16:37:40Z

nix/gce-images.nix

+  "18.09" = "nixos-1809";
+
+  latest = self."18.09";
+  project = "predictix-operations";


This should be updated to a "trusted" upstream like nixos/nix-community.

Definitely, added the steps to prepare the images in the description :

Building an image from source

$ gcloud compute images create nixos-18091228a4c4cbb613c-x86-64-linux \ --source-uri gs://nixos-cloud-images/nixos-image-18.09.1228.a4c4cbb613c-x86_64-linux.raw.tar.gz \ --family=nixos-1809

Making the image public

$ gcloud compute images add-iam-policy-binding nixos-18091228a4c4cbb613c-x86-64-linux \ --member='allAuthenticatedUsers' \ --role='roles/compute.imageUser'

@AmineChikhaoui should be familiar with these operations

I think we should update the create-gce.sh script in nixpkgs to add those steps. Thanks for researching this !
@adisbladis yeah we'll have to host this under a NixOS account. I'm looking into that. Also we're missing a couple of releases in GCP images, so will start first with 20.03 and see if it works.

adisbladis · 2020-05-01T16:39:42Z

nixopsgce/backends/gce.py

@@ -66,7 +66,8 @@ def parse_block_device(xml):
                'disk': self.get_option_value(xml, 'disk', 'resource', optional = True),
                'disk_name': opt_disk_name(self.get_option_value(xml, 'disk_name', str, optional = True)),
                'snapshot': self.get_option_value(xml, 'snapshot', str, optional = True),
-                'image': self.get_option_value(xml, 'image', 'resource', optional = True),
+                'image': self.get_option_value(xml, 'image', str, optional = True),
+                'publicImageProject' : self.get_option_value(xml, 'publicImageProject', str, optional = True),


What happens if this is None(null)?

This must be optional because in case it has value of None, the old method would be used consisting in creating the disk from our own bootstrap image

if v['publicImageProject']: try: img = self.connect().ex_get_image_from_family( image_family=v['image'], ex_project_list=[v['publicImageProject']], ex_standard_projects=False, ) ....................... ......................... else: img = v['image']

grahamc · 2020-05-01T16:40:37Z

nix/gce.nix

@@ -211,6 +218,12 @@ let
    };
  };

+  nixosVersion = builtins.substring 0 5 (config.system.nixos.version or config.system.nixosVersion);


Just noting this is very similar to how nixops-aws works: https://github.com/NixOS/nixops-aws/blob/master/nix/ec2.nix#L166

adisbladis · 2020-05-01T16:44:17Z

@AmineChikhaoui Could you check this PR?

AmineChikhaoui · 2020-05-03T03:15:01Z

nix/gce-images.nix

@@ -0,0 +1,7 @@
+let self = {


I think we can have this file in nixpkgs somhow like ec2-amis.nix and export an attrs:

{ imageFamily = "nixos-20-03"; project = "nixos-org"; }

So that updates of images are automatic and without changes to nixops-gce.

AmineChikhaoui · 2020-05-03T03:16:02Z

nix/gce.nix

+
+  image-family = import ./gce-images.nix;
+  # To be changed to
+  # image-family = import <nixpkgs/nixos/modules/virtualisation/gce-images.nix>;


oh you made a comment about it already :)
nitpick: image-family -> imageFamily

Done fd88dea , 3322656

The imageOptions contains the following : - image : either image-resource or the name of an existing image - family : the image family to be used - project : the parent project containing the image/imageFamily Note that we have to specify either name or family. Default = null Then adjusted the image option to inherit from the imageOptions submodule in - gceDiskOptions in gce.nix - gce-disk.nix Finally, updated the bootstrapImage gce option accordingly

- Adding config as GCEMachinesOptions in GCEDefinition - Updating GCEDefinition attributes by specifying gce as parent attribute - adding the scheduling options - Correcting the instanceServiceAccount attrs ( email and scopes ) - nixosRelease as attr - Using disk name while attaching a GCE Disk - fileSystems backend option made optional - Finally, adding a temporary check over gcp_common options

Add mypy types & compat with latest nixops master

…ency

A somehow trivial example of frontend & backend machines alongside their respectively attached frontend-volume and backend-volume GCE disks + updating poetry.lock

The imageOptions contains the following : - image : either image-resource or the name of an existing image - family : the image family to be used - project : the parent project containing the image/imageFamily Note that we have to specify either name or family. Default = null Then adjusted the image option to inherit from the imageOptions submodule in - gceDiskOptions in gce.nix - gce-disk.nix Finally, updated the bootstrapImage gce option accordingly

…ency

…to gcp-plugin

Moving out the method to retrieve image in gcp_common Enhancing it and adding a couple of exceptions Making it available for both gce.py and gce_disk.py Adding some assertions in gce.nix and gce-disk.nix Removal of publicImageProject attribute Usage of ImageOptions class Upgrading libcloud ignoring libcloud mypy annotations

tewfik-ghariani · 2020-06-16T15:16:24Z

Hello again :))

As a result of the meeting with @AmineChikhaoui and based on his remarks, updated the code in a way to make it more intuitive to create root disks or separate disks from a public image.

The usage syntax shall be as simple as the following :

machine = {
  deployment.gce = {
    bootstrapImage = {    
      name = "base-image-bootstrap";
      family = null;
      project = "nixos-org";
    };
  };
}

resources.gceDisks.main-volume = {
  image = {
    name = null;
    family = "super-family";
    project = "another-project";
  };
};

I've completed my work based on the new changes related to types and options per #7

Change log

Added an example of Machines + GCE Disks.
Introduced imageOptions and defined it as submodule.

The imageOptions contains the following :

image : either image-resource or the name of an existing image

family : the image family to be used

project : the parent project containing the image/imageFamily

Note that we have to specify either name or family.
Default value = {}
Then adjusted the image option to inherit from the imageOptions submodule in

gceDiskOptions in gce.nix

gce-disk.nix
Finally, updated the bootstrapImage gce option accordingly

Created a new common retrieve_gce_image method to fetch the GCENodeImage object.
Removed the publicImageProject attribute
Upgraded libcloud
Ignored libcloud mypy annotations

Full list of changes : master...tewfik-ghariani:gcp-plugin

cc @AmineChikhaoui , @adisbladis can you review please? Or do you prefer if I create a new PR?

tewfik-ghariani · 2020-06-17T13:22:43Z

For some reason, the changes are no longer visible and the commit log has gotten confusing. Maybe it's because the second PR containing the original changes was already merged and this branch was re-based of master

In any case, I will be closing this PR and creating a new one

Cheers :))

nixos-discourse · 2022-10-10T18:09:43Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixops-flake-gce/22355/2

tewfik-ghariani and others added 8 commits March 17, 2020 21:39

First fix : removing pkgs

d2e51c9

First fix --

c437ae3

Introducing bootstrap-images

ddcde62

Displaying GCE disk size in nixops-info

3b2d1fe

Fixing evaluation by addressing some syntax issue

fbf9265

Fixing public_image_project GCE Disk attribute and remove Uniqueness …

cf26f47

…condition on fileSystemsOptions

Granularly controlled the exceptions for 'ex_get_image_from_family' m…

41c6f8d

…ethod

PsyanticY reviewed Mar 19, 2020

View reviewed changes

tewfik-ghariani and others added 10 commits April 12, 2020 01:15

Starting python3 migration.

10b3326

Removal of itertems and replacing it with items/values Removal of 'filter' and replacing it with list comprehension Updating relative imports

Updating release.nix

a71af0f

Starting the migration to poetry

4b4b6dd

Fixing libcloud dependency

12feddc

Renaming to nixops_gcp and updating README

fdeeca6

removing release.nix

6f9f268

Installing cryptography as a dependency for libcloud and reverting ta…

bc6d259

…rgetEnv to gce

Adding .gitignore

42eb8da

blackify

b7fbd63

Reverting gcp targetEnv in the examples as well

22b22cb

tewfik-ghariani mentioned this pull request Apr 18, 2020

Poetry2nixify nixops_gcp + Python3 migration #4

Merged

tewfik-ghariani added 3 commits April 19, 2020 11:12

Reverting blackify and updating targetEnv

4f5e3da

Merge branch 'poetry2nixify' of github.com:tewfik-ghariani/nixops-gce…

ab659d5

… into poetry2nixify

Revert "blackify"

51732aa

This reverts commit b7fbd63.

adisbladis reviewed May 1, 2020

View reviewed changes

grahamc reviewed May 1, 2020

View reviewed changes

adisbladis approved these changes May 1, 2020

View reviewed changes

AmineChikhaoui reviewed May 3, 2020

View reviewed changes

adisbladis and others added 23 commits June 1, 2020 13:31

Add types to gce_forwarding_rule.py

e1b1221

Add types to gce_http_health_check.py

3197bdc

Add types to gce_image.py

a17b76b

Add types to gce_network.py

1012b80

Add types to gce_target_pool.py

5c0b7ac

Add types to gse_bucket.py

670f7ec

Add types to gce-route.py

84643b3

Add types to GCE machine

d070cbd

Remove legacy option copy functions

e048a94

Update to latest nixops master

4387792

gcp_common

0b3455c

Fixing type of GCE disk

54a0c61

gce_routes -> gce_route and removing GceImagesOptions

bcbcdd4

gce.py: Fix black

6b9eb21

Merge pull request nix-community#7 from nix-community/mypy-types

d817855

Add mypy types & compat with latest nixops master

Fixing GCE disk default value and adding nixos-modules-contrib depend…

f2b263e

…ency

Example of Machines + GCE Disks.

9456375

A somehow trivial example of frontend & backend machines alongside their respectively attached frontend-volume and backend-volume GCE disks + updating poetry.lock

Fixing GCE disk default value and adding nixos-modules-contrib depend…

27c45a0

…ency

Merge branch 'gcp-plugin' of github.com:tewfik-ghariani/nixops-gce in…

9b709e3

…to gcp-plugin

tewfik-ghariani marked this pull request as ready for review June 16, 2020 14:59

tewfik-ghariani closed this Jun 17, 2020

tewfik-ghariani mentioned this pull request Jun 17, 2020

Enhance the bootstrap image object #9

Merged

tewfik-ghariani mentioned this pull request Jan 13, 2021

Network and image fixes #12

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fix GCP plugin] Introduce public GCE Bootstrap Image #1

[Fix GCP plugin] Introduce public GCE Bootstrap Image #1

tewfik-ghariani commented Mar 18, 2020 •

edited

Loading

PsyanticY Mar 19, 2020

tewfik-ghariani Mar 19, 2020

adisbladis May 1, 2020

tewfik-ghariani May 2, 2020

AmineChikhaoui May 3, 2020

adisbladis May 1, 2020

tewfik-ghariani May 2, 2020

grahamc May 1, 2020

adisbladis commented May 1, 2020

AmineChikhaoui May 3, 2020

AmineChikhaoui May 3, 2020

tewfik-ghariani May 4, 2020

tewfik-ghariani commented Jun 16, 2020

tewfik-ghariani commented Jun 17, 2020

nixos-discourse commented Oct 10, 2022

[Fix GCP plugin] Introduce public GCE Bootstrap Image #1

[Fix GCP plugin] Introduce public GCE Bootstrap Image #1

Conversation

tewfik-ghariani commented Mar 18, 2020 • edited Loading

Background

Proposed Solution

Validation

Implementation

The way it works

A summary of what has been achieved overall

Small Additions

ToDo

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adisbladis commented May 1, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tewfik-ghariani commented Jun 16, 2020

tewfik-ghariani commented Jun 17, 2020

nixos-discourse commented Oct 10, 2022

tewfik-ghariani commented Mar 18, 2020 •

edited

Loading