Usage instructions

[bharadwajy]

Hi,

@tspink I read your recent Usenix paper about Captive. I would like to build and reproduce the setup described in the paper with ARM VM running on a Linux OS running on bare metal using KVM to get a good understanding of your work.

Can you please provide steps to do so using Captive and Gensim?

Thanks!

[JG3779]

Would appreciate the instructions as well.

[tspink]

Hello, and thanks for getting in touch!

Just to keep you updated - I'm working on a README that should help you get started.

Tom

[bharadwajy]

Great! Thanks for your response. I'll look forward to the document.

[tspink]

I've uploaded a README that should get you started - let me know if it can be improved, or if you need any help!

Tom

[superDong1998]

hi! @tspink can you please give some more detailed requirements or steps about ARM userspace?

thanks!

[tspink]

Hi @superDong1998,

Probably the easiest way to get started is to use the Arch Linux AArch64 system. You need to do the following:

1. Create a block device file
2. Format it
3. Mount it
4. Downloand and extract the distribution into the filesystem
5. Unmount it
6. Load it!

A typical sequence of commands (you'll need to be root to do most of these) might be:

# truncate -s 8G arm64-linux.img
# losetup -f --show arm64-linux.img
/dev/loop0
# mkfs.ext4 /dev/loop0
# mount /dev/loop0 /mnt
# wget http://os.archlinuxarm.org/os/ArchLinuxARM-aarch64-latest.tar.gz
# bsdtar -xpf ArchLinuxARM-aarch64-latest.tar.gz -C /mnt
# umount /mnt
# losetup -d /dev/loop0

Note: I'm writing these commands off the top of my head, so YMMV and be very careful when executing commands as root. In particular, make sure the loopback device is correct when you start formatting and mount it. I take no responsibility for erased hard-drives.

What's going on here is that we're using truncate to create an 8G sparse file, then losetup is creating a loopback block device, backed by the new file. We then format it (mkfs.ext4) with an ext4 filesystem, mount the resulting filesystem, and extract the Arch Linux AArch64 userspace into it. Finally, we unmount it, and remove the loopback device.

Now, the file arm64-linux.img contains an Arch Linux AArch64 userspace, that be used to boot captive.

Hope this helps! Let me know if you need further clarification!

-- Tom

[superDong1998]

hi! Tom, sorry to bother you again. Because I was just getting started with full-system simulation, I ran into a problem when running captive. What puzzled me was that I didn't know if it was a machine problem or an environment problem. By the way, my userspace is not ready yet, could you please check this for me, thank you very much! here is my command: bin/captive --block-dev-file /root/dbt/buildroot/output/images/rootfs.ext4 --engine arch/aarch64.arch --device-tree ./captive-platform.dtb --kernel /root/dbt/arm64-kernel-image --platform virt and my machine info: Intel(R) Xeon(R) CPU E7- 4807 @ 1.87GHz with centos7 linux, kernel version is 5.6 and outputs are: warning: ?: Unable to find parent logging context 'AIO' for 'LinuxAIO' warning: ?: Unable to find parent logging context 'AIO' for 'POSIXAIO' debug: ?: KVM read-only mem: 1 loading arm64 kernel to 0x80080000 *** guest starting error: CPU: Unable to update SREGS: Invalid argument -------------------------------------------------------------------- [core 0] instruction count: 0 [core 0] exec time: 0 [core 0] comp time: 0 -------------------------------------------------------------------- device reads: device writes: *** guest stopped 在2020-04-07 17:07:33，Tom Spinknotifications@github.com写道： Hi @superDong1998, Probably the easiest way to get started is to use the Arch Linux AArch64 system. You need to do the following: Create a block device file Format it Mount it Downloand and extract the distribution into the filesystem Unmount it Load it! A typical sequence of commands (you'll need to be root to do most of these) might be: # truncate -s 8G arm64-linux.img # losetup -f --show arm64-linux.img /dev/loop0 # mkfs.ext4 /dev/loop0 # mount /dev/loop0 /mnt # wget http://os.archlinuxarm.org/os/ArchLinuxARM-aarch64-latest.tar.gz # bsdtar -xpf ArchLinuxARM-aarch64-latest.tar.gz -C /mnt # umount /mnt # losetup -d /dev/loop0 Note: I'm writing these commands off the top of my head, so YMMV and be very careful when executing commands as root. In particular, make sure the loopback device is correct when you start formatting and mount it. I take no responsibility for erased hard-drives. What's going on here is that we're using truncate to create an 8G sparse file, then losetup is creating a loopback block device, backed by the new file. We then format it (mkfs.ext4) with an ext4 filesystem, mount the resulting filesystem, and extract the Arch Linux AArch64 userspace into it. Finally, we unmount it, and remove the loopback device. Now, the file arm64-linux.img contains an Arch Linux AArch64 userspace, that be used to boot captive. Hope this helps! Let me know if you need further clarification!

…

-- Tom — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[tspink]

Thanks for the feedback! Could you try commenting out this line:

https://github.com/tspink/captive/blob/ef80250b1502fa6e372c1692e68c2d1cb21f6a0d/src/hypervisor/kvm/cpu.cpp#L224

And see if you get any further?

[superDong1998]

Hi, tom!
Thanks for the advice, but it seems doesn't work either, it still same question as I mentioned before.

Sorry to bother you anther time.

-alex

[tspink]

That's OK - it just means Captive is trying to enable a feature that is not supported on your host CPU. Try also commenting out line 238 - enabling the NX bit.

[superDong1998]

hi! Tom,
I have installed captive successfully, thanks very much for your previous help.

Recently I did some experiments on captive, but I was a little confused about the result performed on my machine compared the data in your paper published in ATC19. Here are my evaluation details:

host machine:

Intel Xeon Gold 5218 with 16 cores

OS: CentOS 7.7 with kernel 3.10

captive configurations:

Linux AArch64 kernel 5.7

full-fledged user-space with ArchLinuxARM-aarch64-latest.tar.gz

benchmark configurations :

SPEC2006 INT benchmark suite which compiled by aarch64-linux-gnu, version gcc 7.5.0 with opt flag -O2 static both on gcc and g++

I have tested SPEC2006 INT benchmarks both on captive and native(native benchmarks compiled with same version of gcc and same opt flag, and runs on my host machine). It turns out captive has about 25X slowdown compared to native, while this result is 7.24X in the paper.

So I was wondering if I missed some other details during the installation of captive which caused this result, or there is something wrong with the machine or environment I used, could you please help me take a look?

Many thanks!

-alex