gicv2 support #1235

bchalios · 2019-08-30T14:21:34Z

Reason for This PR

This PR is meant to address #1196 for supporting GICv2 devices on aarch64 architecture.

Description of Changes

This makes the GIC-related code more flexible and implements a GICv2 device. It introduces a GICDevice Trait which needs to be implement by GIC device implementations. We implement the Trait for GICv3 and GICv2 devices. When creating the GIC device for the microVM we initially try to create a v3 device and if that fails, we fall back to creating a v2 device.

I have tried this on a Hikey970 board and it works, so far. I 'm trying this out on a RPi4 as well to double check everything is working. In any case, I'm not greatly familiar with the GIC architecture (I did some research for this PR), so any comments on what I 've done so far is more that welcome.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license.

PR Checklist

[Author TODO: Meet these criteria. Where there are two options, keep one.]
[Reviewer TODO: Verify that these criteria are met. Request changes if not]

All commits in this PR are signed (git commit -s).
Either this PR is linked to an issue, or, the reason for this PR is
clearly provided.
The description of changes is clear and encompassing.
Either no docs need to be updated as part of this PR, or, the required
doc changes are included in this PR. Docs in scope are all *.md files
located either in the repository root, or in the docs/ directory.
Either no code has been touched, or, code-level documentation for touched
code is included in this PR.
Either no API changes are included in this PR, or, the API changes are
reflected in firecracker/swagger.yaml.
Either the changes in this PR have no user impact, or, the changes in
this PR have user impact and have been added to the CHANGELOG.md file.

bchalios · 2019-09-01T17:40:49Z

Ok, I did a re-write of the PR which tries to do what was discussed in the issue comments, i.e. the GIC-related code tries to setup a GIC-v3 device and if it doesn't succeed it fails back to create a GIC-v2 device.

Moreover, the GIC code returns the version of the GIC chip that was created and that info is used in order to create the correct FDT later on.

Moreover, I tried to update the relevant tests to take these changes into account.

Disclaimer: my rust knowledge is very limited, so comments are more than welcome :)

luxas · 2019-09-19T08:09:29Z

Thanks @bchalios! Could we get a review of this PR from any of the maintainers please?
cc @dianpopa @andreeaflorescu
Thanks in advance 👏!

dianpopa · 2019-09-19T08:43:31Z

Thanks @bchalios! Could we get a review of this PR from any of the maintainers please?
cc @dianpopa @andreeaflorescu
Thanks in advance 👏!

Yes I will definitely review this. It has been a busy period in terms of PR reviews but I will prioritize this and come back to you asap.

serban300

I didn't have time to take a detailed look on this, and I don't know this area of our code very well, but at first sight I would prefer a more extensible and object oriented design.

I think it would make sense to have a class for GicV2 that contains all the gicv2 specific logic and a different class for GicV3 that contains all the gicv3 specific logic. Also they can implement a common trait. This way it would be easier to add support for gicv4 in the future for example. Also the code would probably be more readable.

bchalios · 2019-09-29T16:49:32Z

I didn't have time to take a detailed look on this, and I don't know this area of our code very well, but at first sight I would prefer a more extensible and object oriented design.

I think it would make sense to have a class for GicV2 that contains all the gicv2 specific logic and a different class for GicV3 that contains all the gicv3 specific logic. Also they can implement a common trait. This way it would be easier to add support for gicv4 in the future for example. Also the code would probably be more readable.

Thanks for your review @serban300!

I 've tried to refactor the PR to implement the feature in the way you suggested. Please, take a look and let me know if the changes make sense to you.

Essentially, I am using this PR as a way to, as well, practice Rust (I 've never written anything before in Rust), so please let me know if something strikes badly to you.

serban300 · 2019-10-01T08:34:24Z

I didn't have time to take a detailed look on this, and I don't know this area of our code very well, but at first sight I would prefer a more extensible and object oriented design.
I think it would make sense to have a class for GicV2 that contains all the gicv2 specific logic and a different class for GicV3 that contains all the gicv3 specific logic. Also they can implement a common trait. This way it would be easier to add support for gicv4 in the future for example. Also the code would probably be more readable.

Thanks for your review @serban300!

I 've tried to refactor the PR to implement the feature in the way you suggested. Please, take a look and let me know if the changes make sense to you.

Essentially, I am using this PR as a way to, as well, practice Rust (I 've never written anything before in Rust), so please let me know if something strikes badly to you.

Thanks for addressing the comments. I will try to take a look on the code as soon as possible. Worst case scenario I will do this at the beginning of the next week. I am super busy right now.

acatangiu

@bchalios the PR looks good functionally, but we'd like some more code deduplication if you have time/availability.

arch/src/aarch64/gic.rs

arch/src/aarch64/gicv2.rs

bchalios · 2019-10-15T17:10:32Z

@acatangiu I pushed a commit that tries to apply the requested changes. The new functionality is implemented through static functions, as I could not figure out a clean way to incorporate it in the common Trait. If you see any alternative, let me know!

Cheers,
Babis

serban300 · 2019-10-16T08:23:38Z

arch/src/aarch64/fdt.rs

+    create_gic_v2_node(fdt, gic_device)
+}
+
+fn create_gic_v2_node(fdt: &mut Vec<u8>, gic_device: &Box<GICDevice>) -> Result<()> {


create_gic_v2_node and create_gic_v3_node look very similar. Looks like the only differences are:

append_property_string(fdt, "compatible", "arm,gic-400")?;
vs
append_property_string(fdt, "compatible", "arm,gic-v3")?;
and
ARCH_GIC_V2_MAINT_IRQ vs ARCH_GIC_V3_MAINT_IRQ

So I think we can have have only one method here. It could receives 2 paramaters: compatible: string and maint_irq: u32. Or it could get this parameters from the GicDevice. Whichever you think makes more sense. I think getting them from GicDevice would be cleaner.

I guess that is fine for now, but what happens when GICv4 (or any newer version) support comes in which would introduce more differences in the tree?

I don't know. Depends on the differences. We can probably make this more extensible but I don't know if it's worth it at this point.

serban300 · 2019-10-16T08:27:21Z

arch/src/aarch64/fdt.rs

@@ -355,15 +357,46 @@ fn create_chosen_node(fdt: &mut Vec<u8>, cmdline: &CStr) -> Result<()> {
    Ok(())
 }

-fn create_gic_node(fdt: &mut Vec<u8>, vcpu_count: u64) -> Result<()> {
+fn create_gic_node(fdt: &mut Vec<u8>, gic_device: &Box<GICDevice>) -> Result<()> {
+    if gic_device.version() == kvm_bindings::kvm_device_type_KVM_DEV_TYPE_ARM_VGIC_V3 {


I thoink that using a match block here would be more readable

serban300 · 2019-10-16T08:47:43Z

arch/src/aarch64/gic.rs

+}
+
+/// Initialize a GIC device
+pub fn init_device(vm: &VmFd, gic_device_type: u32) -> Result<DeviceFd> {


I would move this method inside the GICDevice trait and leverage the version() method which is already defined there:

// version() needs to be static fn version() -> u32; pub fn init_device(vm: &VmFd) -> Result<DeviceFd> { let mut gic_device = kvm_bindings::kvm_create_device { type_: Self::version(), fd: 0, flags: 0, }; let vgic_fd = vm.create_device(&mut gic_device).map_err(Error::CreateGIC); vgic_fd }

Please try to see if you could also leverage this pattern for the other methods defined here: set_device_attribute, finalize_device. This way maybe also the new() method could become smaller and simpler.

hmmm, how can init_device (and set_device_attribute, finalize_device, for that matter) be parts of the Trait object? They are used to create the object itself. How can we call the Trait method on an object that has not yet been constructed?

If the method is static you can call it even if the object hasn't been instantiated yet. For example something like this would work:

pub trait GICDevice { fn version() -> u32; fn init_device() -> bool { Self::version() == 1 } } pub struct GICv2 { initialized: bool } impl GICDevice for GICv2 { fn version() -> u32 { 1 } } impl GICv2 { fn new() -> GICv2 { GICv2 { initialized: Self::init_device() } } }

oh, great! I didn't know that. Thanks a lot.

Hmmm, I tried this out, but when I declare static Trait methods the compiler complains that the trait cannot be made into an object (meaning by the Box<GICDevice> I'm returning from the new method.

I have been looking around and it seems that there is no nice way around this issue. Please let me know, if I'm msising something

Ok, I dealt with it by marking the static methods of the Trait with where Self: Sized, e.g.:

pub trait GICDevice { fn init_device() -> bool where Self: Sized { true } }

essentially, saying that these will not be called on the object itself. Does that make sense to you?

Nice. Sounds good. It probably makes sense, but I have to see the entire commit anyway.

serban300 · 2019-10-16T09:03:39Z

arch/src/aarch64/gicv2.rs

+const KVM_VGIC_V2_CPU_SIZE: u64 = 0x2000;
+
+/// Get the address of the GICv2 distributor.
+fn get_dist_addr() -> u64 {


This can be declared in the GICDevice trait and defined separately for GICv2 and GICv3, just like suggested for init_device(). Also please check if you can do something similar for all the other methods defined here. I would prefer to see them either declare in the GICDevice trait, or if they don't belong there you can define them directly in the GICv2 struct. The same comment for all the methods defined in the gicv3.rs file outside of the GICv3 struct.

I sort of disagree on that. This is an implementation-specific attribute. It could be that another GIC implementation does not have a distributor address attribute, which would not allow it to implement the Trait

Ok. I'm not very familiar with ARM and GIC. Feel free to move the methods where you think that it makes sense. If they are GIC generic, move them to the trait. If not move them inside GICv2 or GICv3 struct.

serban300 · 2019-10-16T09:04:40Z

arch/src/aarch64/gicv2.rs

+const KVM_VGIC_V2_DIST_SIZE: u64 = 0x1000;
+const KVM_VGIC_V2_CPU_SIZE: u64 = 0x2000;


Please move this constants in the impl GICv2 { block since they are GicV2 related. Please try to define all the methods / constants where they belong.

bchalios · 2019-10-30T21:20:21Z

Ok, I think that I've addressed the requested changes. Please, take a look and let me know if the changes make sense

serban300

Looks much better, but I still have some comments. Please take a look.

serban300 · 2019-10-31T09:45:26Z

arch/src/aarch64/gic.rs

+/// Trait for GIC devices.
+pub trait GICDevice {
+    /// Returns the GIC version of the device
+    fn version(&self) -> u32;


It would make sense for this method to be static as well:

fn version() -> u32 where Self: Sized;

Right, I couldn't do that before because it was used from the fdt code, but now yup.

serban300 · 2019-10-31T09:46:06Z

arch/src/aarch64/gic.rs

+        Self: Sized,
+    {
+        let mut gic_device = kvm_bindings::kvm_create_device {
+            type_: version,


If yo transform version() into a static method, you can use Self::version() here. And yo ucan remove the version parameter.

serban300 · 2019-10-31T11:47:17Z

arch/src/aarch64/gicv2.rs

+    pub fn new(vm: &VmFd) -> Result<Box<dyn GICDevice>> {
+        let vgic_fd =
+            Self::init_device(vm, kvm_bindings::kvm_device_type_KVM_DEV_TYPE_ARM_VGIC_V2)?;
+
+        /* Setting up the distributor attribute.
+        We are placing the GIC below 1GB so we need to substract the size of the distributor. */
+        Self::set_device_attribute(
+            &vgic_fd,
+            kvm_bindings::KVM_DEV_ARM_VGIC_GRP_ADDR,
+            u64::from(kvm_bindings::KVM_VGIC_V2_ADDR_TYPE_DIST),
+            &GICv2::get_dist_addr() as *const u64 as u64,
+            0,
+        )?;
+
+        /* Setting up the CPU attribute. */
+        Self::set_device_attribute(
+            &vgic_fd,
+            kvm_bindings::KVM_DEV_ARM_VGIC_GRP_ADDR,
+            u64::from(kvm_bindings::KVM_VGIC_V2_ADDR_TYPE_CPU),
+            &GICv2::get_cpu_addr() as *const u64 as u64,
+            0,
+        )?;
+
+        Self::finalize_device(&vgic_fd)?;
+
+        let vgic_device = GICv2 {
+            fd: vgic_fd,
+            properties: [
+                GICv2::get_dist_addr(),
+                GICv2::get_dist_size(),
+                GICv2::get_cpu_addr(),
+                GICv2::get_cpu_size(),
+            ],
+        };
+
+        Ok(Box::new(vgic_device))
+    }


I think that this logic can be nicely abstracted and moved in the trait:

init_device() is already a trait method

for the logic related to setting the device attributes we can have a trait method

fn init_device_attributes(fd: &DeviceFd, vcpu_count: u64) -> Result<()> where Self: Sized;

and implement it differently for GICv2 vs GICv3. v2 impl for example:

finalize_device is already defined in the trait

the part that actually creates the device can also be abstracted in a method that is declared in the trait and defined separately for GICv2 and GICv3:

trait :

fn create_device(fd: DeviceFd, vcpu_count: u64) -> Box<dyn GICDevice> where Self: Sized;

v2 impl:

fn create_device(fd: DeviceFd, vcpu_count: u64) -> Box<GICDevice> { Box::new(GICv2 { fd: fd, properties: [ GICv2::get_dist_addr(), GICv2::get_dist_size(), GICv2::get_cpu_addr(), GICv2::get_cpu_size(), ], }) }

And then we can have new defined in the trait:

fn new(vm: &VmFd, vcpu_count: u64) -> Result<Box<dyn GICDevice>> where Self: Sized, { let vgic_fd = Self::init_device(vm)?; Self::init_device_attributes(&vgic_fd, vcpu_count); Self::finalize_device(&vgic_fd)?; Ok(Self::create_device(vgic_fd, vcpu_count)) }

The only problem here is that we would have to pass all the creation params for any GIC implementation. For example GICv2 doesn't need the vcpu_count but we have to pass it anyway. So it's not ideal. Also there would be some coupling. Which for the moment seems ok but I don't know how it will affect future GIC devices support.

Do you think it would make sense to refactor it like this ? Considering extensibility and future GIC devices support.

serban300 · 2019-10-31T11:48:35Z

arch/src/aarch64/gic.rs

+    fn set_device_attribute(
+        fd: &DeviceFd,
+        group: u32,
+        attr: u64,
+        addr: u64,
+        flags: u32,
+    ) -> Result<()>
+    where
+        Self: Sized,
+    {
+        let attr = kvm_bindings::kvm_device_attr {
+            group: group,
+            attr: attr,
+            addr: addr,
+            flags: flags,
+        };
+        fd.set_device_attr(&attr)
+            .map_err(Error::SetDeviceAttribute)?;
+
+        Ok(())
+    }


This method seems to only forward the parameters to fd.set_device_attr. I don't know if it brings a lot of value.

bchalios · 2019-10-31T15:03:41Z

Ok, could you take a look now? I don't think we can avoid the problem with multiple arguments per version, since the external code creating the device cannot know what device is supported.

serban300

In general it looks good to me. I just have a couple of nits left.

serban300 · 2019-11-01T08:41:44Z

arch/src/aarch64/gic.rs

+        let vgic_fd = vm.create_device(&mut gic_device).map_err(Error::CreateGIC);
+
+        vgic_fd


nit: here you can just do vm.create_device(&mut gic_device).map_err(Error::CreateGIC) . There's no need for the ssignment.

serban300 · 2019-11-01T08:42:58Z

arch/src/aarch64/gicv2.rs

+    }
+    fn device_fd(&self) -> &DeviceFd {


nit: I think it would be nice to have a new line between the end of a definition and the beginning of the next one. Please fix this in all the structs.

I completely agree :) I just saw that at other places, for example vmm/src/device_manager/mmio.rs that it is not used an empty line.

bchalios · 2019-11-01T11:51:03Z

Ok, done with the last comments, as well

serban300

Everything looks ok. It's as good as approved on my side, but please sign your commits using git commit --ammend -s. Also I think it would make sense to reduce the number of commits. You can fixup everything in one commit.

bchalios · 2019-11-01T13:18:04Z

Everything looks ok. It's as good as approved on my side, but please sign your commits using git commit --ammend -s. Also I think it would make sense to reduce the number of commits. You can fixup everything in one commit.

Hmmm, I thought the commits were signed already. GitHub marks them as verified. Am I missing something?

serban300 · 2019-11-01T13:51:48Z

Everything looks ok. It's as good as approved on my side, but please sign your commits using git commit --ammend -s. Also I think it would make sense to reduce the number of commits. You can fixup everything in one commit.

Hmmm, I thought the commits were signed already. GitHub marks them as verified. Am I missing something?

I am not sure why github marks them as verified, maybe that's something else. But there should be a sign-off message at the end of each commit. Something like:

Signed-off-by: Serban Iorga <seriorga@amazon.com>

and git commit --ammend -s will add it automatically.

bchalios · 2019-11-01T13:59:20Z

Everything looks ok. It's as good as approved on my side, but please sign your commits using git commit --ammend -s. Also I think it would make sense to reduce the number of commits. You can fixup everything in one commit.

Hmmm, I thought the commits were signed already. GitHub marks them as verified. Am I missing something?

I am not sure why github marks them as verified, maybe that's something else. But there should be a sign-off message at the end of each commit. Something like:
Signed-off-by: Serban Iorga <seriorga@amazon.com>
and git commit --ammend -s will add it automatically.

Ok, I 'm generally using git commit -S, which actually signs the commit with GPG but doesn't add the "Signed-off-by" bit in the commit message.

We refactor the GIC-related code, to define a GICDevice Trait and then create a GICv3 object that implements this Trait. This way, it is more straightforward to introduce an implementations for different GIC versions. Moreover, we implement the Trait for GICv2 interrupt controllers. At the moment, the creation of the GICDevice tries to create a version 3 device. If that fails, we fallback to GICv2. Signed-off-by: Babis Chalios <babis.chalios@gmail.com>

luxas · 2019-11-04T12:52:52Z

Thanks @bchalios @serban300 @acatangiu 🎉!

bchalios force-pushed the gicv2 branch 5 times, most recently from b135d55 to cecaaf3 Compare September 1, 2019 17:01

bchalios marked this pull request as ready for review September 1, 2019 17:41

alxiord assigned bchalios Sep 2, 2019

dianpopa self-requested a review September 2, 2019 08:36

bchalios force-pushed the gicv2 branch 2 times, most recently from be1f019 to 2f55520 Compare September 15, 2019 15:45

bchalios changed the title ~~WIP: gicv2 support~~ gicv2 support Sep 15, 2019

serban300 suggested changes Sep 19, 2019

View reviewed changes

dianpopa added Codebase: Usability labels Sep 23, 2019

bchalios force-pushed the gicv2 branch 4 times, most recently from 4978947 to a3771ec Compare September 29, 2019 15:36

acatangiu reviewed Oct 3, 2019

View reviewed changes

arch/src/aarch64/gic.rs Outdated Show resolved Hide resolved

arch/src/aarch64/gicv2.rs Outdated Show resolved Hide resolved

bchalios force-pushed the gicv2 branch from a3771ec to fd27ae3 Compare October 14, 2019 07:37

serban300 suggested changes Oct 16, 2019

View reviewed changes

bchalios force-pushed the gicv2 branch from fd27ae3 to 0e5fa62 Compare October 30, 2019 20:35

serban300 reviewed Oct 31, 2019

View reviewed changes

bchalios force-pushed the gicv2 branch from 43886ef to 901f0aa Compare October 31, 2019 15:01

bchalios force-pushed the gicv2 branch 4 times, most recently from cb90776 to 5a78a6c Compare October 31, 2019 17:32

serban300 reviewed Nov 1, 2019

View reviewed changes

serban300 self-requested a review November 1, 2019 08:54

bchalios force-pushed the gicv2 branch 2 times, most recently from d52195a to c341967 Compare November 1, 2019 11:50

serban300 reviewed Nov 1, 2019

View reviewed changes

serban300 self-requested a review November 1, 2019 13:02

bchalios force-pushed the gicv2 branch from c341967 to 378a6df Compare November 1, 2019 13:32

bchalios force-pushed the gicv2 branch from 378a6df to 4b488a6 Compare November 1, 2019 14:00

serban300 approved these changes Nov 1, 2019

View reviewed changes

acatangiu approved these changes Nov 1, 2019

View reviewed changes

acatangiu merged commit 763a80c into firecracker-microvm:master Nov 1, 2019

serban300 mentioned this pull request Nov 4, 2019

Add GICv2 support to support Raspberry Pi 4 #1196

Closed

		const KVM_VGIC_V2_DIST_SIZE: u64 = 0x1000;
		const KVM_VGIC_V2_CPU_SIZE: u64 = 0x2000;

		let vgic_fd = vm.create_device(&mut gic_device).map_err(Error::CreateGIC);

		vgic_fd

gicv2 support #1235

gicv2 support #1235

Conversation

bchalios commented Aug 30, 2019 • edited Loading

Reason for This PR

Description of Changes

License Acceptance

PR Checklist

bchalios commented Sep 1, 2019

luxas commented Sep 19, 2019

dianpopa commented Sep 19, 2019

serban300 left a comment

Choose a reason for hiding this comment

bchalios commented Sep 29, 2019 • edited Loading

serban300 commented Oct 1, 2019

acatangiu left a comment

Choose a reason for hiding this comment

bchalios commented Oct 15, 2019

Choose a reason for hiding this comment

bchalios Oct 18, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

serban300 Oct 16, 2019 • edited Loading

Choose a reason for hiding this comment

bchalios Oct 18, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bchalios commented Oct 30, 2019

serban300 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bchalios commented Oct 31, 2019

serban300 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bchalios commented Nov 1, 2019

serban300 left a comment

Choose a reason for hiding this comment

bchalios commented Nov 1, 2019

serban300 commented Nov 1, 2019

bchalios commented Nov 1, 2019

luxas commented Nov 4, 2019

bchalios commented Aug 30, 2019 •

edited

Loading

bchalios commented Sep 29, 2019 •

edited

Loading

bchalios Oct 18, 2019 •

edited

Loading

serban300 Oct 16, 2019 •

edited

Loading

bchalios Oct 18, 2019 •

edited

Loading