[Feature Request] - Run Amazon Linux 2023 as a virtual machine on premises

[Manouchehri]

Is your feature request related to a problem? Please describe.
Creation of Amazon Linux 2022 virtual machine (VM) images for on-premises development and testing. e.g. the same as: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/amazon-linux-2-virtual-machine.html

Describe the solution you'd like
This URL to be valid:

https://cdn.amazonlinux.com/os-images/2022/kvm/

Describe alternatives you've considered
I can probably figure out how to make my own bootstrapped install using dnf install --installroot=/mnt, but it would be nice to have an official image to start with.

[chadbrewbaker]

I'm working on this right now! Trying to get a UTM image for my M1, here are the instructions for Arch: https://ktprograms.codeberg.page/blog/posts/2022-03-17_1750_utm-arch-arm/

The first steps I think can be refactored into an Alpine linux container for x86/ARM.

• Build container
• Image for wsl2 - upstream to https://github.com/yosukes-dev/AmazonWSL
• Image for UTM - upstream to https://github.com/utmapp/mac.getutm.app/tree/main/_virtual_machines
• Image for kvm

[Manouchehri]

@chadbrewbaker Hmm, not sure I get how that helps with AL2022?

[liesenml]

Hello, thank you for submitting your request! We review all requests on a weekly basis and will reply to this ticket by with our evaluation.

[chadbrewbaker]

Bump. This is nontrivial. I blew another day on it yesterday - need some official AWS help here.

My notes for inspecting Fedora images on macos:

strings Fedora-Minimal-36-1.5.armhfp.raw | grep RSA "PRIVATE"
xxd -a Fedora-Minimal-36-1.5.armhfp.raw | less
fdisk Fedora-Minimal-36-1.5.armhfp.raw
hdiutil attach -imagekey diskimage-class=CRawDiskImage -nomount Fedora-Minimal-36-1.5.armhfp.raw
hdiutil mount /dev/disk5s1

From what I can tell AmazonLinux2 used effectively the same grub2 VM structure as Fedora. https://arm.fedoraproject.org

@iximiuz has project to take the OSI container and stuff a bootloader etc on it: iximiuz/docker-to-linux#25 This might be the most elegant solution.

[bplessis-swi]

The FAQ still references kvm images as to be provided on the GA release:

Q: Can I use AL2023 outside of AWS?

A: AL2023 images can be used outside of AWS, however, these images are not covered by AWS Support Plans when used outside of AWS. When Amazon Linux 2023 becomes Generally Available, it will be provided as a virtual machine image for on-premises use, enabling you to easily develop, test, and certify applications from a local development environment. This option is not available during the preview.

Look like someone forgot this ^^

[nmeyerhans]

Indeed the FAQ did indicate that VM images would be available at GA for use outside of EC2. Let's consider that FAQ entry to be inaccurate, for now.

We are working on VM images and will note their availability in the release notes and resolve this issue when they're available.

[emboss64]

Any updates on this one? Thanks!

[yoonjs2]

Any updates? Thanks!

[r-vw]

Also very interested in any progress wrt this request

[Rosch]

I am missing AL2023 in a vmware instance desperately! Is there anything known when this comes to life?

[rotan-hanrahan]

According to the Wayback Machine, the following text was present on https://aws.amazon.com/linux/amazon-linux-2023/faqs/ in March 2023:

When Amazon Linux 2023 becomes Generally Available, it will be provided as a virtual machine image for on-premises use, enabling you to easily develop, test, and certify applications from a local development environment. This option is not available during the preview.

The June 3 snapshot of the same page no longer contains this text, so the promise made during the preview regarding the VM image once AL2023 reaches GA appears to be withdrawn. (Note: the revision of the FAQ was suggested by @nmeyerhans in early April, almost three months ago.)

I see no evidence of any outreach to the community to explain this, nor any requests for technical assistance (assuming the issue is technical). If the issue is bureaucratic in nature, we might never see the promised VM image.

Some clarification from Amazon is overdue.

[inflatador]

Red Hat's recent announcement opens a window of opportunity for Amazon Linux to greatly increase on-prem adoption. Here's hoping Amazon will take advantage of that opportunity, starting with the VM images requested here.

[tanvp112]

It's becoming more and more difficult to use AL2 now, eg. https://urllib3.readthedocs.io/en/latest/v2-migration-guide.html#common-upgrading-issues

[stewartsmith]

Apologies for the long time radio-silence here, I understand the frustration this must cause, and there's certainly developers like myself who share that frustration.

While this isn't an image, it may be a useful starting point for building your own. Using the pretty neat kiwi-ng with the following config, I can boot to a login prompt. There's a few other things that are missing that turn this into a base image that we'd release (some cloud-init config at least), but it's a starting point.

You'll need this at root/etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0,115200n8 console=tty0 rd.emergency=poweroff rd.shell=0 selinux=1 security=selinux quiet"
GRUB_TIMEOUT=0
GRUB_DISABLE_RECOVERY="true"
GRUB_TERMINAL="ec2-console"
GRUB_ENABLE_BLSCFG="true"
GRUB_X86_USE_32BIT="true"
GRUB_DEFAULT=saved
GRUB_UPDATE_DEFAULT_KERNEL=true

and this at root/etc/cloud/cloud.cfg.d/02-onprem.cfg

datasource_list: [ NoCloud, AltCloud, ConfigDrive, OVF, None ]

and then your appliance.kiwi can look like:

<?xml version="1.0" encoding="utf-8"?>

<image schemaversion="7.5" name="amazonlinux-2023">
    <description type="system">
        <author>Stewart Smith</author>
        <contact>trawets@amazon.com</contact>
        <specification>Example Amazon Linux 2023 kiwi config</specification>
    </description>
    <profiles>
        <profile name="Virtual" description="Virtual image of Amazon Linux 2023"/>
    </profiles>
    <preferences>
        <version>2.0.0</version>
        <packagemanager>dnf</packagemanager>
        <keytable>us</keytable>
        <timezone>UTC</timezone>
        <rpm-check-signatures>true</rpm-check-signatures>
    </preferences>
    <preferences profiles="Virtual">
      <type image="oem" filesystem="xfs" kernelcmdline="console=ttyS0" firmware="efi" format="qcow2">
            <oemconfig>
                <oem-resize>false</oem-resize>
            </oemconfig>
            <bootloader name="grub2" console="serial" timeout="10"/>
        </type>
    </preferences>
    <repository imageinclude="false" sourcetype="mirrorlist">
      <source path="https://cdn.amazonlinux.com/al2023/core/mirrors/latest/x86_64/mirror.list"/>
    </repository>
    <packages type="bootstrap" patternType="onlyRequired">
      <package name="dnf"/>
      <package name="curl-minimal"/>
      <package name="gnupg2-minimal"/>
    </packages>
    <packages type="image" patternType="onlyRequired">
      <namedCollection name="onprem"/>
      <namedCollection name="onprem-minimal"/>
      <package name="kernel"/>
      <package name="grub2-pc"/>
      <package name="grub2-pc-modules"/>
      <package name="grub2-efi-x64-modules"/>
      <package name="grub2-efi-x64-ec2"/>
      <package name="zstd"/>
      <package name="systemd-networkd"/>
    </packages>
</image>

Built on a Fedora 38 instance, and tested with qemu-system-x86_64 -boot c -net nic -net user -m 4G -cpu max -smp 4 -hda testboot -bios /usr/share/edk2/ovmf/OVMF_CODE.fd -nographic just to booting to a Login prompt, and not a moment further.

i.e. this is a starting point for someone to continue what that looks like.

[chadbrewbaker]

Apparently you can do this with AWS CLI? https://docs.aws.amazon.com/vm-import/latest/userguide/vmexport.html

al2023-ami is the keyword apparently?

[cpswan]

I've been keeping an eye on this issue for a while, and it seems AWS have really dropped the ball here.

Given the chaos in the Enterprise Linux (EL) space since RedHat announced changes to RHEL source availability there was a real opportunity for Amazon Linux to become the preferred distro for those wanting big company backing without subscribing to RHEL (or dealing with Oracle).

But that opportunity is being squandered, on this very issue. If people can't run AL23 everywhere that they want EL, then they'll probably choose something else as their baseline.

Furthermore, promising images and then failing to deliver makes it look like the project isn't properly resourced, which is another ding on trust.

Choice of distro has been a topic I've been following for a long time (since well before this post almost 7y ago). It seems almost incomprehensible that AWS isn't grabbing hold of the chance to get customers onto AL wherever they run Linux, which of course smoothes the path to migrating stuff to AWS.

Repo maintainers... this isn't aimed at you. I guess you're struggling to keep everything together with insufficient resources. This is for you to show to the higher ups, so they're making an informed decision about where they're placing their bets, and the consequences of ruined expectations and inaction.

[AaronSwearingen]

What OS does AWS recommend for hybrid cloud solutions? AL2 is less then two years away from EOL. AL2023 isn't available as a virtual machine and also presumably isn't supported for use with https://aws.amazon.com/hybrid/ features like ECS/EKS Anywhere.

I appreciate the repo maintainers trying to help out the community, but AWS needs to be made aware this is more than just a community request. There is a real business need to enable companies to leverage (and pay for) AWS hybrid cloud services. Anyone expecting AL2023 to fulfill that need now finds themselves in a position of uncertainty.

[chadbrewbaker]

What OS does AWS recommend for hybrid cloud solutions? AL2 is less then two years away from EOL. AL2023 isn't available as a virtual machine and also presumably isn't supported for use with https://aws.amazon.com/hybrid/ features like ECS/EKS Anywhere.

I appreciate the repo maintainers trying to help out the community, but AWS needs to be made aware this is more than just a community request. There is a real business need to enable companies to leverage (and pay for) AWS hybrid cloud services. Anyone expecting AL2023 to fulfill that need now finds themselves in a position of uncertainty.

Usual suspects - Ubuntu, Debian, Fedora, Suse, Arch. ML practitioners love Ubuntu, Linus loves Fedora.

I would probably use Fedora Desktop as it is closest to AL2023 for laptops.

This Reddit thread is spot on. AWS lost its way firing passionate maintainers like Tim McNamara that would have knocked out this ticket a year ago to get more community test coverage/adoption.

[AaronSwearingen]

I would probably use Fedora Desktop as it is closest to AL2023 for laptops.

ECS Anywhere supports Fedora 33 according to https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-anywhere.html. That's the latest version of Fedora on this list - EOL since 2021-11-30. The rest of the list isn't looking much healthier.

Getting AL2023 on the list would go a long way towards AWS showing it is committed to supporting hybrid cloud.

[hesaurey]

Amazon please release OVA for Vmware it's good for you, good for us!

[cpswan]

The Register referenced this issue with their piece Amazon Linux 2023 virtual machine images still MIA

It closes with the usual:

Amazon did not respond to a request for comment

But let's hope that it helps get some attention onto fixing the problem.

[Rosch]

Hi Amazon

An on premise version of AL 2023 is truly missing. When will it be available?
We want to fully support AL2023 with our software but developing for it
is only possible when it is available in our intranet. So yet we can only support AL2.

Beside of helping software developers an on premise version of AL 2023
is really a good chance for AL to fill the gap caused by RHELs actions.

Thanks

Roland

[daniejstriata]

I installed a host with AL2023 on-prem using DistroBox. It's adequate for my own limited use case, not the same as a VM running AL2023 though.

[stewartsmith]

Undoubtedly there could be UX improvements here for running cloud like images in UTM; but it does appear to work.

I just realized that the drive-resizing UI element that makes this practical was something I worked on a bit back when: utmapp/UTM#4087 - maybe I'll get bored on a weekend again and try and do something for cloud like images.

[ccassidy-maxar]

Follow up to the issue w/ VMWare Workstation and the disk2 error. The following method at least allowed me to boot up to a prompt. Not sure if it will cause any unintended consequences yet.

1. Extract component files from the ova using tar
2. Delete references to the disk2 file from the .ovf file (3 places)
3. Get the SHA512 sum on the updated ovf file
4. Delete the disk2 references from the mf file, and update the ovf checksum
5. Recreated the ova with the following:
   ovftool al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt.ovf al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt.ova

Also noticed that the al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt-disk2.vmdk file was actually XML, not sure if that is expected or not.

[ozbenh]

Follow up to the issue w/ VMWare Workstation and the disk2 error. The following method at least allowed me to boot up to a prompt. Not sure if it will cause any unintended consequences yet.

1. Extract component files from the ova using tar
2. Delete references to the disk2 file from the .ovf file (3 places)
3. Get the SHA512 sum on the updated ovf file
4. Delete the disk2 references from the mf file, and update the ovf checksum
5. Recreated the ova with the following:
   ovftool al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt.ovf al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt.ova

Also noticed that the al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt-disk2.vmdk file was actually XML, not sure if that is expected or not.

This is very strange.... we built this ova using the same tooling we've always used for AL2 and tested it with both VMware workstation (17.5) and vSphere... I will take a look as soon as I'm not in a plane

[rotan-hanrahan]

I'm bound to a VirtualBox environment, and have used the VB images from Amazon ( such as here https://cdn.amazonlinux.com/os-images/2.0.20231101.0/ ), so I guess I will wait while the bugs are ironed out of the kvm/vmware versions and see what happens. So glad to see movement on this again, though I did lose one client to the "dark side" on account of no offsite pre-deployment options like we had with the previous images. (I might try to pull them back if the VB image appears.)

I'll take a quick peek once the URLs stop returning 404s.

Thanks @stewartsmith.

[sn-tm]

I'm experiencing the same issue as @ccassidy-maxar. VMWare Workstation Pro 17.5.0 on Windows 11. Open the .ova, give the VM a name and path, then see the same error, "Disk (al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt-disk2.vmdk) could not be found."

I retested the same procedure using the AL2 .ova and had no problems.

By the way, thanks for providing this image. This is what we've been waiting for to help us test our software and migrate to AL2023 (once we resolve this issue of course!)

[ozbenh]

So it does look like the al2023 OVA has two "disks" in it, while previously we had only one. Probably due to its size (which incidently will be reduced in the next release as @stewartsmith mentioned).

I wonder if there's a host type related issue here. What are you using (both the good and base cases) ? We do our testing on a Ubuntu 22.04.3 system with VMWare Workstation for Linux, could it be a Windows specific issue ?

[sn-tm]

Hi @ozbenh, I just tried this on Ubuntu 22.04.3. Downloaded and installed VMWare Workstation 17 Pro, opened al2023-vmware_esx-2023.2.20231113.0-kernel-6.1-x86_64.xfs.gpt.ova, entered a name and location, clicked Import.

[ozbenh]

Ok, something's wrong on our end with the testing :-( We'll get to the bottom of it asap

[ozbenh]

We found the cause of the issue and will be updating the images. A sad story involving different versions of open-vmdk having different (incompatible) syntax for mkova.sh ... ugh. It should have been caught, sorry about that, something slipped in the test validation. Stay tuned for updated images. We'll also be slimming them a bit as mentioned earlier by @stewartsmith

[ozbenh]

A fixed (and slimmed) OVA has been uploaded and replaces the previous one. We haven't changed the KVM ones at all for now

[ccassidy-maxar]

A fixed (and slimmed) OVA has been uploaded and replaces the previous one. We haven't changed the KVM ones at all for now

Tested the new OVA and worked flawlessly on VMWare Workstation 17.0.2 / Windows 10, thanks!

[tanvp112]

I'm bound to a VirtualBox environment, and have used the VB images from Amazon ( such as here https://cdn.amazonlinux.com/os-images/2.0.20231101.0/ ), so I guess I will wait while the bugs are ironed out of the kvm/vmware versions and see what happens. So glad to see movement on this again, though I did lose one client to the "dark side" on account of no offsite pre-deployment options like we had with the previous images. (I might try to pull them back if the VB image appears.)

I'll take a quick peek once the URLs stop returning 404s.

Thanks @stewartsmith.

The OVA can be imported into VB 6.1.48, but after change password there seems to be storage driver incompatibility; either the new password is not recognized upon reboot or error with "Authentication token manipulation". Will need to try 7.x and possibly tune the seed files...

[ozbenh]

I'm bound to a VirtualBox environment, and have used the VB images from Amazon ( such as here https://cdn.amazonlinux.com/os-images/2.0.20231101.0/ ), so I guess I will wait while the bugs are ironed out of the kvm/vmware versions and see what happens. So glad to see movement on this again, though I did lose one client to the "dark side" on account of no offsite pre-deployment options like we had with the previous images. (I might try to pull them back if the VB image appears.)
I'll take a quick peek once the URLs stop returning 404s.
Thanks @stewartsmith.

The OVA can be imported into VB 6.1.48, but after change password there seems to be storage driver incompatibility; either the new password is not recognized upon reboot or error with "Authentication token manipulation". Will need to try 7.x and possibly tune the seed files...

VirtualBox has been .... weird ... can you try converting the image away from vmdk to vdi (I suppose there's a way to do that, I haven't researched) first ? I would not be surprised if VB has bugs around vmdk handling...

[tanvp112]

The VMDK formatted hard drive has no issue convert into VDI. Previously for AL2, a VB image is provided by AWS and it works out of the box. Believe it or not, I have run this copy & still running today for >5 years day-in-day-out. AL2 is arguably the best Linux experience I ever had so far.

The release of AL2023 image is a big step to many... kudos for AWS Linux team. Hopefully, there will be a VB image like the former and the superb experience would continue. For the case above, VB 6.1.x is sunsetting already, will need to test with latest VB 7.x instead.

Happy to hear more about others' experience using VB with the new image!

[yoonjs2]

Is possible to convert AL2023 arm64 qcow2 to parallels hdd format? I'm bounded to macOS with apple silicon environment and uses parallels in dev/production.

[ozbenh]

Is possible to convert AL2023 arm64 qcow2 to parallels hdd format? I'm bounded to macOS with apple silicon environment and uses parallels in dev/production.

I don't have a Mac to try (though @stewartsmith does, he might come up with an answer), but Google has a few hits on this. Alternatively there's a qemu for Mac which uses Hypervisor Framework under the hood and there's UTM as a front-end for it all

[ozbenh]

Is possible to convert AL2023 arm64 qcow2 to parallels hdd format? I'm bounded to macOS with apple silicon environment and uses parallels in dev/production.

I don't have a Mac to try (though @stewartsmith does, he might come up with an answer), but Google has a few hits on this. Alternatively there's a qemu for Mac which uses Hypervisor Framework under the hood and there's UTM as a front-end for it all

I did some experimentations and it didn't go particularly well. Not only converting to parallel is tricky and messy (the tools don't work well and parallels itself doesn't seem to be helping), but the end result doesn't boot.

I think I know why (they and vmware fusion both seem to be exposing storage as some kind of ACPI generated platform AHCI that isn't PCI backed, and we don't enable that driver). I'll see if I can get that turned on in a future kernel release. That said, we don't have plans at this stage to produce officially supported Parallels images.

[michailf]

Does anyone have a reference to login information? What user and password would I use for the images downloaded from the links provided?

[azN2]

Does anyone have a reference to login information? What user and password would I use for the images downloaded from the links provided?

Cloud-init

[stewartsmith]

Does anyone have a reference to login information? What user and password would I use for the images downloaded from the links provided?

Be sure to read the instructions on https://docs.aws.amazon.com/linux/al2023/ug/outside-ec2.html for configuring the VM. There isn't a password set, you need to set up something so that cloud-init can configure the instance on first boot.

[michailf]

I spent hours trying to figure out how to login to the provided image. I have read multiple documents, including the reference here. I cannot log in to the image. I was able to set the local hostname via the meta-data (guestinfo.metadata) so I believe I provide the data correctly. What I do not know is why the users are not being setup with a simple password.

Is there a working example of the user-data (guestinfo.userdata) that allows a simple password log in with sudo rights to the image provided?

[michailf]

I was able to log in using an ssh key. I then could create/adjust other users to my needs. It seems the cloud-init in the image does not allow for plain passwords.

[tanvp112]

I think I know why (they and vmware fusion both seem to be exposing storage as some kind of ACPI generated platform AHCI that isn't PCI backed, and we don't enable that driver).

This really rings a bell! I suspect this is required for the image to be compatible for VB as well.

[gebailey]

The VMDK formatted hard drive has no issue convert into VDI. Previously for AL2, a VB image is provided by AWS and it works out of the box.

I've built Vagrant cloud images (virtualbox provider) of Amazon Linux 2 for a few years now, and just uploaded one for Amazon Linux 2023 that was built from the VMware OVA. A few steps were required to convert from OVA->VMDK->VDI but seems to work just fine. https://app.vagrantup.com/gbailey/boxes/al2023

[ozbenh]

I was able to log in using an ssh key. I then could create/adjust other users to my needs. It seems the cloud-init in the image does not allow for plain passwords.

I have used images with passwords a few times but usually by creating a new user, something like:

  - name: benh
    groups: sudo
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    ssh-authorized-keys:
      - <key>
    plain_text_passwd: <password>
    lock_passwd: false

This doesn't seem to work with ec2-user last I tried, but did you check if the old-school chpasswd method works ? From the AL2 on-prem docs:

chpasswd:
  list: |
    ec2-user:<plain_text_password>

[michailf]

I have used images with passwords a few times but usually by creating a new user, something like:

  - name: benh
    groups: sudo
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    ssh-authorized-keys:
      - <key>
    plain_text_passwd: <password>
    lock_passwd: false

I got ssh-authorized-keys to work but not plain_text_passwd independently. I have not tried both of the options for the same user.

This doesn't seem to work with ec2-user last I tried, but did you check if the old-school chpasswd method works ? From the AL2 on-prem docs:

chpasswd:
  list: |
    ec2-user:<plain_text_password>

I also tried the chpasswd method but it did not work for me.

[ozbenh]

I just tested again with

#cloud-config
#vim:syntax=yaml
users:
  - name: test
    groups: sudo
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    plain_text_passwd: passw0rd
    lock_passwd: false

and it worked fine

Are you logging in via the console or serial ? Or are you trying to ssh ? Passsword authentication is disabled in the ssh server by default.

[michailf]

I just tested again with

#cloud-config
#vim:syntax=yaml
users:
  - name: test
    groups: sudo
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    plain_text_passwd: passw0rd
    lock_passwd: false

and it worked fine

I tried your example and it did work. I am not sure what I was doing wrong before.

The following example works to create a new user (al) with sudo access, enable the default user (ec2-user), set the ec2-user's password and force a password change on login.

#cloud-config
#vim:syntax=yaml
users:
  - default
  - name: al
    groups: sudo
    sudo: ALL=(ALL) NOPASSWD:ALL
    plain_text_passwd: Passwor8
    lock_passwd: false
chpasswd:
  list: |
    ec2-user:Passwor8

The following is a minimal example to gain full access to the system as ec2-user has sudo rights:

#cloud-config
#vim:syntax=yaml
users:
  - default
chpasswd:
  list: |
    ec2-user:Passwor8

[denisgabriel5]

Does anyone have some instructions on how to modify the VMware image according to this article?