Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider support for Android VMs #2233

Open
andrewdavidwong opened this issue Aug 7, 2016 · 57 comments
Open

Consider support for Android VMs #2233

andrewdavidwong opened this issue Aug 7, 2016 · 57 comments

Comments

@andrewdavidwong
Copy link
Member

@andrewdavidwong andrewdavidwong commented Aug 7, 2016

Over the years, many Qubes users on the mailing lists have expressed the desire to be able to run some kind of Android VM (e.g., HVM) under Qubes OS.

@andrewdavidwong andrewdavidwong added this to the Far in the future milestone Aug 7, 2016
@JohnnyCalavera
Copy link

@JohnnyCalavera JohnnyCalavera commented Sep 18, 2016

Did someone try to create an android-x86 VM on qubes? I was planning on doing it myself in the near future. (by the way: the stable Marshmallow release is out now)

@grote
Copy link

@grote grote commented Sep 18, 2016

I tried running RemixOS, but without success. There might be a general problem of running Android as a guest in a Xen hypervisor.

@Jeeppler
Copy link

@Jeeppler Jeeppler commented Sep 20, 2016

@andrewdavidwong can you reference to those mailing list entries?

@andrewdavidwong
Copy link
Member Author

@andrewdavidwong andrewdavidwong commented Sep 21, 2016

@Jeeppler: This is based on my memory of the mailing lists over the years, so I don't have an exhaustive list. However, here are some examples:

@Jeeppler
Copy link

@Jeeppler Jeeppler commented Sep 23, 2016

According to the posts, people want to run Android x86 to do application development.

However, it seems the Android emulator uses the CPU virtualization extension. The problem could also be solved by using nested virtualization on Xen.

@andrewdavidwong
Copy link
Member Author

@andrewdavidwong andrewdavidwong commented Nov 8, 2016

Cross-linking new discussion thread:

https://groups.google.com/d/topic/qubes-users/frK8xaBh9pI/discussion

@entr0py
Copy link

@entr0py entr0py commented Nov 8, 2016

@grote I have not had any issues running RemixOS under Qubes. But it suffers from the same Android-x86 mouse issue mentioned in the thread that Andrew just linked.

Qubes is actually a convenient platform for running Android emulators because most of them don't allow static IP addresses. Qubes' mini-DHCP server works just fine.

@grote
Copy link

@grote grote commented Nov 8, 2016

@3n7r0p1 which version did you use? Could you send me the link to the download?

Please also let me know if you find a solution for the mouse issue.

@entr0py
Copy link

@entr0py entr0py commented Nov 9, 2016

I used an older version of RemixOS so I checked again with the latest release and it also installs and boots fine.

version: 3.0.206, 64-bit
filename: Remix_OS_for_PC_Android_M_64bit_B2016101201.zip
download: http://www.jide.com/remixos-for-pc#downloadNow
https://www.fosshub.com/Remix-OS.html

when booting iso, add INSTALL=1 to kernel parameters
create partition, gpt: no, efi: no
if boot hangs, add vga=ask to kernel parameters (may need debug mode on)
allow 10-15 mins for first boot, 10-15 mins for desktop after setup

Confirmed mouse behavior hasn't changed. Not all functionality can be replicated from keyboard. Unusable without mouse fix.

IMO, RemixOS is more resource-heavy than Android-x86 (especially graphics).
I've Googled the heck out of the mouse issue. Don't have the expertise to take it any further. :(

@mfc mfc mentioned this issue Jan 31, 2017
2 tasks
@Wikinaut
Copy link

@Wikinaut Wikinaut commented Jun 26, 2017

👍 yes, please

@micahflee
Copy link

@micahflee micahflee commented Jul 17, 2017

I just tried android-x86 in an HVM, and it actually works great (the only change I had to do was increase the RAM -- I'm using 4GB, but it probably will work with less). It doesn't have the same mouse issues as RemixOS (which, as of today, is a discontinued project).

screenshot_2017-07-17_13-28-13

I downloaded an Android 4.4.4 iso from https://sourceforge.net/projects/android-x86/ -- but the more I look into it, it appears there are later versions of android-x86 available, including 7.1: http://www.android-x86.org/releases/releasenote-7-1-rc1

@micahflee
Copy link

@micahflee micahflee commented Jul 17, 2017

It would be cool to make an Android template. I'm not quite sure how the partitioning could work so that android-x86 AppVMs could have private images though.

@micahflee
Copy link

@micahflee micahflee commented Jul 17, 2017

Update: I tried installing android-x86 7.1 in an HVM, and it doesn't work nearly as well as 4.4.4. I could boot in live mode, but I had the same mouse issues that exist in RemixOS, and when I install to the hard disk and try booting, it never finishes.

@nm8800
Copy link

@nm8800 nm8800 commented Jul 29, 2017

@micahflee How did you get an AndroidVM to work in Qubes? Currently trying to figure out how to use the Android-x86 vm to do the same.

@aseralis
Copy link

@aseralis aseralis commented Feb 15, 2018

I push the topic gently upwards, an android HVM in qubes would be great! Unfortunately it's very painful to get Android x86 running, but I'm going to test a bit by myself and share results.

@kuzega
Copy link

@kuzega kuzega commented Mar 15, 2018

So hows going with android vm ?

My testing results on qubes r4.0-current-testing are that none of android works so far.
Tested iso images : remixOS, android x86 latest cyanogenmod 7.1 release1 and normal 7.1r1, 4.4r5, phoenixOS.
None of them detect the hard drive.
So another test I installed centos7minimal then andro x86 latest cyanogenmod in rpm package so it added grub entry, it does not boot
Most of the tests showed familiar boot screen but did not run, VM killed itself.

But i remember it was working on qubes 3.2, cant remember what android did i run but probably it was remixOS, when i was testing this like a year ago or so. It was install-able but did not boot from installation, had to use iso and all the changes were gone. The performance was really choppy and i gave up, that probably was because of SLAT feature missing on previous laptop.

PS: was not editing grub entries at all, only tested different VESA modes when available

Hint: try to test preinstalled android VM images and just put them in /var/lib/qubes . Personally i couldnt get this method to work, first have to convert given preinstalled images to qcow RAW format

@tonsimple
Copy link

@tonsimple tonsimple commented Mar 24, 2018

@micahflee
Sorry for nagging, but could you please kindly step-by-step creation of android HVMs under Qubes R3.2 for the less talented of us?

can't seem to make it boot/install (tried Android 4.4.4 and later)

@tschakram
Copy link

@tschakram tschakram commented Jul 4, 2018

Hello,
any solution to get a android hvm running on Qubes 4 ? Booting a live iso worked for all Versions of Androidx86,RemixOS and CM. But installtion failed because of the error "No hard drive avalible". Possible to prepare the hvm drive before installing?
thanks

@ptitdoc
Copy link

@ptitdoc ptitdoc commented Aug 7, 2018

Hello,

Here are my limited success to deal with an Android HVM in Qubes R4:
1/ Download Android x86 ISO (tested with android-x86-7.1-r2.iso)
2/ Create an HVM
3/ Give at least 4000MB of RAM to the HVM and disable memory balancing (testing with 2000MB caused the VM to crash during startup)
4/ In the QubesVM advanced settings, select "Boot qubes from CDROM" and use "from file in Qubes" to select the ISO you downloaded
5/ When the VM boots, select Advanced options > Live CD VESA mode - No GPU hardware acceleration. Other options are working but this is the most stable I found. (Press space when android boots up)

Here are the issues I'm also facing:

  • The mouse works strangely because of QubesOS libvirt settings. For this reason, it is needed to use drag and drop all the time inside the Android window in order to use the mouse. To improve this it is possible to edit in dom0 /usr/share/qubes/templates/libvirt/xen.xml and to change the line:
    <input type="tablet" bus="usb" />
    to
    <input type="mouse" bus="usb" />
  • It is still not possible to install the android VM using recent Android version because the hard disk drive is not detected. Apparently Android does not supports disks presented as Xen block devices. Maybe the problem is similar to #3068.

@airelemental
Copy link

@airelemental airelemental commented Aug 7, 2018

@ptitdoc
Copy link

@ptitdoc ptitdoc commented Aug 8, 2018

Thanks for the pointer.

By the way, people are talking about problems with networking, but networking works out of the box in my case.

@mat-rex
Copy link

@mat-rex mat-rex commented Aug 8, 2018

ptitdoc, what version of android did you try? I think I shot myself in the foot with 7.1.2, as might have a bug with ethernet and dhcp.

I did the following 2 changes:

  1. Changed 1 line in "init" inside initrd
    from:
    for device in ${ROOT:-/dev/[hmnsv][dmrv][0-9a-z]}; do
    to:
    for device in ${ROOT:-/dev/[hmnsvx][dmrv][0-9a-z]
    }; do
    That allows the XEN hard drive to be used at initialization, once the kernel has the driver.
    I would appreciate if this change would go into the repo there if you (enyone) are an android-x86 developer.

  2. I configured the kernel

@mat-rex
Copy link

@mat-rex mat-rex commented Sep 21, 2018

A bit of a time passed, and on the sideline I have a config with which the mouse is a mouse, but you have two pointers. If you change the cursor to large, you can get used to it. Network configuration is stil manual and a pain.

Once you build android with that you get a CD (The best if you use the android common build environment... it took me days to get a build on 16G with qubes... but can be done). If that is too much, I can provide you the CD somehow (but why would you trust someone to provide you a CD ... but if in that VM, you have nothing to loose, you do not risk too much :-) ).

Create a new HVM. Boot a recent linux CD/DVD/thumbdrive (I was using the openly available, but unannounced knoppix 8,2).

boot your VM from the linux CD. Create an EXT2 partition to boot (50 or 500 megs? do not remember). install the bootloader from the cd.

Shutdown linux. Boot android.

Install Android to the free space, creating a new partition "ext4". reboot.

Boot linux again. Mount both partitions, copy kernel and inititial ramdisk, into the boot directory. Do NOT use the same name as what the android-x86 install created on the EXT4 partition.

google:
andoid-x86 boot grub

update grub/menu.lst according to the instructions found, modify directory name.

reboot end enjoy if you can without a network.

If you need networking, google android-x86 static ip.Use the IP quebes allocated your vm. Or find instructions on how to setup dhcp from the command line - I did not figure that. I have no clue why it does not autodetect.

If some succeeds, please post detailed instructions and CD. I posted this without the actual system at hand, so steps are as I remember from top of my head from many weeks ago.

Please post if you fail as well, when I'll check here (maybe soon, maybe a few month) if there are failure reports I'll try to post a detailed instructions with the system at hand.

@mat-rex
Copy link

@mat-rex mat-rex commented Oct 28, 2018

The behaviour of the mouse in qubes 4 appears to come from:

/usr/share/qubes/templates/libvirt/xen.xml

Replace the word " tablet " with " mouse ", and mouse will become an emulated mouse in the HVMs. You get two mouse pointers - but if you change the size (I set it to max of 48) of your desktop mouse (Dom0 - system tools - settings manager - Mouse and touchpad - Theme) so the two can be distinguished, it is usable.

This allowed me to run a BSD gui environment in a HWM. I'll see how this changes things for Android.

@thestinger
Copy link

@thestinger thestinger commented Nov 3, 2018

It would be completely reasonable to only have updates of the base images externally with them always read-only within the OS. That's what I mean by handling it another way, i.e. not actually handling updates within the OS, and sharing the entire OS images even after updates rather than copy-on-write since they're always identical across installations. I don't know the best approach, but it would certainly be easier to avoid dealing with all the bootloader complexity for A/B updates, especially dealing with verified boot and rollback protection.

@v6ak
Copy link

@v6ak v6ak commented Nov 3, 2018

@thestinger
Copy link

@thestinger thestinger commented Nov 3, 2018

The existing update systems (A/B updates or the legacy recovery system) require some bootloader support that I'd need to implement so that's why it's worth figuring out the best approach in advance. I think it'd be easier to implement the legacy system, but it's on the way out and has serious robustness disadvantages. It seems the Android-x86 fork doesn't provide updates at all, so there's not an existing implementation of the bootloader support.

@fgvhcw
Copy link

@fgvhcw fgvhcw commented Nov 16, 2018

@mat-rex Can you please upload the working kernel config? The config that you posted in qubes-users got truncated when you copied it between VMs.

@mat-rex

This comment has been hidden.

@mat-rex
Copy link

@mat-rex mat-rex commented Dec 13, 2018

@mat-rex
Copy link

@mat-rex mat-rex commented Dec 13, 2018

@marmarek
Copy link
Member

@marmarek marmarek commented Feb 23, 2019

A post with detailed setup (including build from source) instruction for Android x86 in a Standalone VM on Qubes 4.0:
https://groups.google.com/d/msgid/qubes-users/53bdab0a-8c70-4fc3-9d4b-224b43849946%40googlegroups.com

@adrelanos
Copy link
Member

@adrelanos adrelanos commented Oct 14, 2019

Manged to run Anbox inside Whonix-Workstation.

https://www.whonix.org/wiki/Anbox

It should be very much possible to port these instructions to Debian buster based and possibly other VMs too.

@thestinger
Copy link

@thestinger thestinger commented Oct 14, 2019

I wouldn't recommend Anbox since the security model and mitigations within it are missing / broken.

@Linnxam
Copy link

@Linnxam Linnxam commented Nov 21, 2019

@adrelanos Could you please provide a guide on how you accomplished to run Anbox inside Whonix-WS 15?
I followed the directions in the Whonix Wiki but it doesn't say how to tackle the in-VM kernel issue. Can it be done in PHV mode?
@marmarek That solution doesn't offer clipboard or file sharing between VMs

@adrelanos
Copy link
Member

@adrelanos adrelanos commented Nov 21, 2019

@Linnxam
Copy link

@Linnxam Linnxam commented Nov 21, 2019

@adrelanos The idea is to install the kernel from within the VM and not having to change anything in dom0, similar to how you can do it with a cloned Debian-10 TemplateVM by selecting 'none' as kernel.
Is there any way to make that work for Whonix-ws? Alternatively I could also continue with the cloned Debian TemplateVM but once I installed Anbox and ran it, it started up but the Anbox windows were just blank white. I was able to interact with objects within the window by clicking them, but the weird graphical glitch prevented me from seeing what I was clicking.

@adrelanos
Copy link
Member

@adrelanos adrelanos commented Nov 22, 2019

The idea is to install the kernel from within the VM and not having to change anything in dom0, similar to how you can do it with a cloned Debian-10 TemplateVM by selecting 'none' as kernel.
Is there any way to make that work for Whonix-ws?

Qubes VM kernel works the same in Whonix as it works in Debian.

Alternatively I could also continue with the cloned Debian TemplateVM but once I installed Anbox and ran it, it started up but the Anbox windows were just blank white. I was able to interact with objects within the window by clicking them, but the weird graphical glitch prevented me from seeing what I was clicking.

I have no idea.

@Linnxam
Copy link

@Linnxam Linnxam commented Nov 26, 2019

Can you (or someone) help me figure out the problem?
I just reproduced it with a fresh install (RC 4.0.2 iso).
Steps to reproduce:

  1. Create new qubes (Standalone qube copied from a template -> debian-10, sys-whonix as NetVM)
  2. Go to settings and select Kernel 'none' and Virtualization HVM
  3. Start a terminal in the newly created qube (updated & upgraded)
  4. sudo apt-get install linux-image-amd64 linux-headers-amd64 adb fastboot anbox
  5. Download android_amd64.img into /var/lib/anbox and rename to android.img
  6. Reboot & start Anbox (via GUI)
  7. See startup screen with the Android logo
  8. New window org.anbox.appmgr pops up with a blank/white screen inside the window

When hovering the mouse over the window and clicking in random places, new windows will pop up (e.g. 'Files') but they'll also launch in a blank window where you can't see anything.
Any troubleshooting advice?

The kernels are installed correctly
ls -1 /dev/{ashmem,binder}
/dev/ashmem
/dev/binder
And Anbox services seem to be running fine
sudo systemctl status anbox-container-manager.service

@Linnxam
Copy link

@Linnxam Linnxam commented Mar 1, 2020

@adrelanos
The solution was to enable software rendering because Qubes doesn't support OpenGL within VMs. I'm surprised that you managed to run it, the Whonix guide doesn't instruct to enable software rendering.
snap set anbox software-rendering.enable=true
snap restart anbox.container-manager

I'll tag @marmarek here too
The reason I wanted an Android VM is to be able to use proprietary apps that also function offline, and generally for other testing purposes. So after downloading the apps I cut networking to the Android VM to make sure they don't call home and due to other security concerns, as @thestinger previously mentioned. Now I'd like to make sure the VM itself leaves no traces in QubesOS. Is it possible to set the whole VM to read-only to make sure nothing gets written to disk?

@marmarek
Copy link
Member

@marmarek marmarek commented Mar 1, 2020

You can use Disposable VM out of this VM (on R4.0 you need TemplateVM -> AppVM and then DispVM from this AppVM). Not really "nothing gets written to disk", but "all modifications are discarded on shutdown". See https://www.qubes-os.org/doc/disposablevm-customization/

@Linnxam
Copy link

@Linnxam Linnxam commented Mar 1, 2020

@marmarek it's a standalone Debian-10 VM (HVM)
The goal is to have an offline & read-only Android container that doesn't write/leak anything into OS/disk.
Not really looking for an "all modifications are discarded on shutdown", but making the whole VM read-only.
Can I do this from either outside (dom0) or from the inside (VM)?
If it's not possible, what would be the best approach to not let Anbox/Android apps write anything on disk and only execute in RAM?

@marmarek
Copy link
Member

@marmarek marmarek commented Mar 1, 2020

You can try setting its volumes to read-only using qvm-volume config <vmname>:<volume> rw False for each volume. Haven't tried it before ;)

@v6ak
Copy link

@v6ak v6ak commented Mar 1, 2020

@Linnxam
Copy link

@Linnxam Linnxam commented Mar 1, 2020

@marmarek @v6ak
Thanks for the suggestions. I made the private, root & volatile volumes read-only and it wouldn't even start a terminal (although the VM itself booted normally).
Leaving private & volatile rw and root read-only resulted in the desired effect, at least superficially.
qvm-volume config Android:root rw False

Let me give you an example:
One of the many good proprietary apps would be Google Translate, it's very interesting to be able to read foreign forums & comments but I can do without Google tracking translation in real time and calling home constantly
Easy: download offline languages, cut network VM
Next problem: I don't necessarily want all those translation to dwell somewhere on my disk as I obviously don't know what I'm going to translate.
So I tested it out, translated a couple sentences from Korean to English, shut down the VM, started it again and then Google Translate showed me my last translations so obviously it's getting stored on the disk.

I made the root volume read-only and tried again. When translating Korean sentences to English and closing the app, then starting it again, it shows my last translations, but when rebooting the VM they're gone now.
Is there a way to verify if they still get written to the fs somewhere or if everything is done in RAM now?

@adrelanos
Copy link
Member

@adrelanos adrelanos commented Mar 2, 2020

@adrelanos
Copy link
Member

@adrelanos adrelanos commented Mar 2, 2020

@adrelanos
The solution was to enable software rendering because Qubes doesn't support OpenGL within VMs. I'm surprised that you managed to run it, the Whonix guide doesn't instruct to enable software rendering.
snap set anbox software-rendering.enable=true
snap restart anbox.container-manager

I didn't need to enable software rendering.

The command from Instructions for enabling Anbox software rendering probably won't work for Whonix Anbox guide as it does not use snap and it shouldn't use snap either because that would break recommendation always verify signatures since snap does not verify gpg signatures (yet?).

@v6ak
Copy link

@v6ak v6ak commented Mar 4, 2020

I have looked at Anbox under Debian. I have tried the snap version (which is OK for just trying in a DVM and is likely more fresh than the package in Debian repo). It has security updates from Jan 2017, i.e., over 3 years old. That's no go for me. Maybe there can be few reasonable exceptions, but other than that, it is simply unacceptable.

@thestinger
Copy link

@thestinger thestinger commented Mar 4, 2020

Worth noting that Anbox has the Android sandbox and security model mostly disabled, and security within a Qubes guest is still important. It's contained from impacting the rest of the system but I don't think you should abandon having a proper security model in guests. It makes far more sense to run an x86 build of AOSP in the guest with the security model intact and full app compatibility rather than a partial implementation.

@thestinger
Copy link

@thestinger thestinger commented Mar 4, 2020

I'd also recommend using AOSP, which has x86 support, and just needs a properly configured kernel, etc. rather than a fork based on an out-of-date version that's rolling back the security model. I'm sure Android x86 has some useful patches, but unfortunately, it's largely focused on things counter to the goals and needs for this. An installer isn't needed and as people mentioned earlier the images could be updated elsewhere rather than trying to be compatible with the proper A/B update system + verified boot, or hacking together something much worse. Just keep things simple, use the latest code, and work on getting input devices and then further integration working smoothly. It's going to involve forking a few AOSP repositories to do the integration and it's really best if it's based on the latest stable release and sticks as close as possible to that to accomplish the goal of compatibility and QubesOS integration rather than starting from a fork not aligned with the goals.

This is still on the radar of the GrapheneOS project, and we've done some initial research and work on it, but we need to find developers that can be hired to work on it. If you're interested in the development and maintenance of a production-oriented port of AOSP to QubesOS including putting together integration for clipboard support, etc. and you have a record of open source contributions and relevant experience (not necessarily with AOSP itself but Java / Kotlin app development experience would be good), contact us at contact@grapheneos.org. There is funding available to get someone a nice workstation and a small stipend to work on it, but we can't currently fully fund the develop work ourselves. There are likely organizations interested in this among those that we're in contact with though and I would expect that I could get someone full-time funding to work on this if they got things rolling. This is a target that we want to support for GrapheneOS, but the support for it would be maintained as a separate project without the other changes to AOSP, which would then be used by GrapheneOS to build for the QubesOS guest target.

@Linnxam
Copy link

@Linnxam Linnxam commented Mar 9, 2020

@adrelanos
Are you using an iGPU or external? Some people are encountering the same issues, last resort was to install it using snap.
@v6ak @thestinger
I'm aware of the security issues, that's why I cut networking to the VM. For those requiring connectivity this obviously won't work.
Unfortunately it seems for now there's no other working Android VM solution where file and clipboard sharing work.

Now I'd like to verify whether or not the usage as outlined above would leave traces of VM content on disk, in my case e.g. a "Google Translate" app history.
The 2 measures I took so far were setting the root volume to read-only and having the VM use its maximal capacity. I can't even create an empty text file within the VM anymore.
Could I hash & compare whole VM volumes before and after execution, or how could this verification issue be approached? Not sure if this is on topic @marmarek.
But I think some people opting for this temporary solution might be asking themselves the same questions.

@tabbyrobin
Copy link

@tabbyrobin tabbyrobin commented Oct 15, 2020

This is still on the radar of the GrapheneOS project, and we've done some initial research and work on it, but we need to find developers that can be hired to work on it. If you're interested in the development and maintenance of a production-oriented port of AOSP to QubesOS including putting together integration for clipboard support, etc. [...] This is a target that we want to support for GrapheneOS, but the support for it would be maintained as a separate project without the other changes to AOSP, which would then be used by GrapheneOS to build for the QubesOS guest target.

I really hope this happens. I think GrapheneOS on Qubes could be very useful for low budget/low memory Qubes users (4-8gb RAM). Because android has a decent internal security model/compartmentalization, this way users could use fewer Qubes domains.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet