oci-image: init scripts to build and upload image #119856
Conversation
ilian
force-pushed
the
oci-image
branch
2 times, most recently
from
fbae55a
to
0b413d2
Compare
| @@ -0,0 +1,98 @@ | |||
| #! /usr/bin/env bash | |||
There was a problem hiding this comment.
For what it is worth, after working with the AWS uploader for some time, I have needed to reimplement it with Terraform to keep track of the details. I wonder if using Terraform for this instead of a script would work too?
There was a problem hiding this comment.
I think Terraform would work for this too. The oci terraform provider seems to have the functionality needed to import a custom image: https://registry.terraform.io/providers/hashicorp/oci/latest/docs/resources/core_image
Would you mind sharing what details needed to be tracked in your use-case for AWS? I can't currently write/test Terraform code as my free trial has expired which unfortunately prevents me from importing custom images.
| --operating-system NixOS \ | ||
| --source-image-type QCOW2 \ | ||
| --launch-mode PARAVIRTUALIZED \ | ||
| --display-name NixOS \ |
There was a problem hiding this comment.
This is not very much metadata. Can we add more, or am I missing it? For example:
- exact version of the channel used to build it
- any relevant filesystem information
- who built it and when
- perhaps a version number for the metadata itself, so we could add more types of images another time and not break automated integrations
There was a problem hiding this comment.
Yes, I agree. I think this can be achieved similar to how EC2 images are built by creating a JSON manifest when buliding the image and passing this data to OCI:
nixpkgs/nixos/maintainers/scripts/ec2/amazon-image.nix
Lines 86 to 92 in 6933d06
I think we should add some basic metadata such as the channel version without any metadata version number to the display name to avoid clutter, and include all verbose metadata to the Custom Image itself with Resource Tags which can be provided in the JSON format with the oci cli tool.
| systemd.services.fetch-ssh-keys = { | ||
| description = "Fetch authorized_keys for root user"; | ||
|
|
||
| wantedBy = [ "sshd.service" ]; | ||
| before = [ "sshd.service" ]; | ||
|
|
||
| after = [ "network-online.target" ]; | ||
| wants = [ "network-online.target" ]; | ||
|
|
||
| path = [ pkgs.coreutils pkgs.curl ]; |
There was a problem hiding this comment.
Should this service be enabled only on the very first profile? I wonder if this should be part of the -user config.
There was a problem hiding this comment.
The idea here is that the service is executed at least once before the initial ssh connection. This service does indeed need to only run once, since the ssh keys can't be modified in OCI after creating the instance AFAIK.
It should indeed be enabled only for the first profile. The user is allowed to change the system configuration to not include the fetch-ssh-keys service, since all ssh keys should be present in /root/.ssh/authorized_keys by then, hence this service is not included in oci-config-user.nix.
https://github.com/NixOS/nixpkgs/pull/119856/files Add image configuration for Oracle Cloud Infrastructure and scripts to build and upload the image as a Custom Image.
|
Hi! Expect to see some activity on your PR, since that announcement about their A1 instances. On my end I worked out how to adapt this for those A1 instances (Look carefully at the commits, the tip adds a "bad" commit to work around an issue with RK3399... That one is not needed.) I don't know if there's anything else that should be done here though. I'm definitely not an OCI expert. I basically started learning about their cloud offering less than 48 hours ago. Some additional notesYou will have to ensure that the image capabilities only includes UEFI as the firmware type.
Otherwise the instances will invalidly use "BIOS", and will not boot. AFAICT it requires to be set on the image. This image will not build on A1 VMs!!! This is because it requires |
That's great news! Thank you for making me aware of their new ARM instances.
Thanks! I'll add your commits to this PR.
Can this manual configuration step be avoided after modifying the Unfortunately, my Free Trial has expired so I am not able to test out any new additions to the build script since the Feel free to add/modify any commits as I have enabled edits by maintainers for this PR. |
|
Just a reminder:
As far as I understand OCI, the "Launch Mode" is distinct from "Firmware". And as the documentation indeed states, "BIOS" seemingly needs to be set for Given the following link: I wonder if it's as simple as adding
It would be nice to see which shapes actually run on UEFI. I am betting that the documentation stating it needs to be legacy boot is plainly wrong.
Oh what??? The custom images are not part of the free offer?? I wasn't finding any information about the pricing for those, like at all... This sucks :| So I guess going with "infect" like systems will be better in the end. Unless the NixOS foundation somehow gets in touch with Oracle to get some kind of useful deal. |
|
Reading the (admittedly thick to navigate) pricing list, coupled with the free tied page, I'm led to believe there is up to 10GB available free for custom images. Expect an edit to this message, I'm chatting with a live agent about this. Update: Custom Images, or sometimes called BYOI (bring your own image) is NOT part of the free tier. The account needs to be a paid account for this to work. Note that during the trial you can upload custom images. It is unclear what happens after the fact, but with the agen't responses I believe they count towards the 10GB available for block storage, but you're simply unable to upload new images. |
|
Hey folks, I just replied to @samueldr (I assume, based on the avatar) over on the Oracle Cloud forum. I'm a member of the Oracle Cloud Developer Adoption team as well as one of Oracle's GitHub admins. I'm more than happy to be your official tour guide to OCI and make sure you have the right tools and contacts to get NixOS published to our Marketplace. The first thing I'll mention is that upgrading your free trial account to pay-as-you-go is (ironically) one of the requirements to activate the Always Free resources, which will then increase your custom image count back up to 25. You will not be charged for any resources that qualify for Always Free, and those that do will have an "Always Free" tag visible in the Console. I was also an Oracle Linux product manager for over a decade prior to joining the OCI developer adoption team, so I know how the Oracle Linux images are built (hint: we use Packer. A lot) and published. I can also work as the interface between the NixOS Foundation and our Linux engineers, particularly if you're interested in merging some of the GCC optimizations for the Ampere A1/Neoverse-N1 platform that have still not been merged upstream. |
|
Hi Avi, thanks for reaching out! Is it possible and useful for the NixOS
foundation to publish official images for Oracle Cloud? Ideally
individual users would not need to maintain their own base images out of
the gate.
On May 29, 2021, GitHub ***@***.***> wrote:
Hey folks, I just replied to @samueldr <https://github.com/samueldr>
(I assume, based on the avatar) over on the Oracle Cloud forum. I'm a
member of the Oracle Cloud Developer Adoption team as well as one of
Oracle's GitHub admins. I'm more than happy to be your official tour
guide to OCI and make sure you have the right tools and contacts to
get NixOS published to our Marketplace.
The first thing I'll mention is that upgrading your free trial account
to pay-as-you-go is (ironically) one of the requirements to activate
the Always Free resources, which will then increase your custom image
count back up to 25. You will not be charged for any resources that
qualify for Always Free, and those that do will have an "Always Free"
tag visible in the Console.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#119856 (comment)>,
or unsubscribe <https://github.com/notifications/unsubscribe-
auth/AAASXLEEQZQ2CO6MVFRVWRTTQGHW3ANCNFSM43FSFLTA>.
|
|
It's both useful and possible to publish NixOS images to the Oracle Cloud Marketplace. This will make it available to all Oracle Cloud users via the Create Instance dialog on the Console, as well as via the API or SDK. Edited to add that publishing to the Marketplace allows the image to take advantage of our metadata service to allow for deeper cloud-init integration and other tooling. |
|
Using kexec via https://github.com/cleverca22/nix-tests/blob/master/kexec/session.md worked for me to get into a live system. Unfortunately it rebooted into the EFI shell after a while. I terminated the previous instance and created a new one and disabled the cloud agent in the instance configuration. I assume it works like a watchdog, so remember to disable it if you try that. |
|
For aarch64, can we use systemd-boot as the bootloader? On boot, GRUB gives me the following error: It boots, but it's not possible to select the system generation through the serial console. systemd-boot seems to work fine from my testing. |
|
@zhaofengli in theory yes. In practice last I tried our systemd-boot tooling failed on AArch64. Though what should be done, instead or in addition, is fix the GRUB issue. It goes a bit more deeply than that specific warning/error. Serial vs. graphical GRUB are not working correctly with the current configuration, and I haven't looked at the actual underlying issue. I really don't recall what the problem was, but I recall it wasn't an issue of that module being found or not. |
I tried and it worked fine, though I ran
I think it's fixable by simply removing |
|
Though I'm 99% sure this is not the reason you don't see the graphical menu with GRUB. So yes, it probably should be removed, but I expect it will change nothing. |
| cfg = config.oci; | ||
| in | ||
| { | ||
| imports = [ ../profiles/qemu-guest.nix ]; |
There was a problem hiding this comment.
what does OCI use from this?
There was a problem hiding this comment.
The profile contains the necessary kernel modules for several QEMU/VirtIO devices. Running lspci on my instance running on OCI confirms that there are several emulated VirtIO devices present:
00:00.0 Host bridge: Red Hat, Inc. QEMU PCIe Host bridge
00:01.0 Display controller: Red Hat, Inc. Virtio GPU (rev 01)
00:02.0 USB controller: Red Hat, Inc. QEMU XHCI Host Controller (rev 01)
00:03.0 Ethernet controller: Red Hat, Inc. Virtio network device
00:10.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
...
18:00.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI (rev 01)
|
Maybe we could rewrite the scripts to Python instead of bash? Python has more API access and better controls than the CLI. |
|
In case someone wonders, this PR still works quite well (14 months after the latest contribution).
Are you guys still interested in merging it? EDIT: Meanwhile, I have to keep the |
|
Will love to see this merged! |
|
I've been using the |
|
I also would love to see this merged. Hi @puffnfresh. By any chance, are you using Oracle's ARM VPS? If yes, didn't you cross-compile the image for "aarch64-linux" on a "x86_64-linux" machine? I'm asking because I tried to copy the modules of this PR and then to cross-build an image, but I get the error: If you did something similar, I would really appreciate some suggestions. Thanks! |
|
@chenlijun99 I probably did, can't remember. I do have access to a few arm64 systems so I might have just compiled the image from there. It looks like you're not actually cross-compiling. You probably want to set system to |
Add image configuration for Oracle Cloud Infrastructure and scripts to build and upload the image as a Custom Image.
thiagokokada
force-pushed
the
oci-image
branch
from
4d109af
to
3cbd7f9
Compare
|
Rebased this PR and did a bunch of fixes:
I think this should fix most of the basic issues in this PR. Tested and it is working fine in my personal account. There is a few outstanding other issues, but I think we could merge this PR as-is and work on then in the future, since this PR already has some value as-is (e.g.: people wanting to play with NixOS in Oracle Cloud, specially because of their generous Always Free tier). |
|
Code looks fine, unfortunately I can't test it because of trade embargo. |
Follows what amazon images does.
A couple notes: --------------- Adding invalid `console=` parameters is not an issue. Any invalid console is unused. The kernel will use the "rightmost" (last) valid `console=` parameter as the default output. Thus the SBBR-mandated AMA0 on A1, and ttyS0 on x86_64 as documented by Oracle. `nvme_core.shutdown_timeout=10` was added as it was written this way in the A1 images. Unclear whether `nvme.shutdown_timeout=10` is wrong. At worst this is a no-op.
thiagokokada
force-pushed
the
oci-image
branch
from
3cbd7f9
to
371ceba
Compare
|
I did 2 tests:
Both took a while to be reachable through SSH (a few minutes VS a few seconds in the official images from OCI), but they work well enough to give most people a start point. While most people are probably more interested in the However, for the |
|
Going to merge this. Future fixes can go after merging it. |
|
Does anyone have a simple step by step tutorial on how to use this? |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: |
Motivation for this change
Add image configuration for Oracle Cloud Infrastructure and scripts to
build and upload the image as a Custom Image.
Things done
sandboxinnix.confon non-NixOS linux)nix-shell -p nixpkgs-review --run "nixpkgs-review wip"./result/bin/)nix path-info -Sbefore and after)