Spirv lookup refactor #88

Napokue · 2020-06-30T21:58:25Z

Fix for #85.

monocodus

This is an autogenerated code review, new suggestions: 5

src/back/spv/writer.rs

Napokue · 2020-06-30T22:02:40Z

src/back/spv/writer.rs

                instruction = self.instruction_type_vector(id, scalar_id, size);
-                self.lookup_type.insert(id, handle);
+                self.lookup_type.insert(id, lookup_type);


This still gives a compile error, because it expects a value type instead of the reference type. Problem is if I dereference it, I get the error: "Cannot move".

it looks like you are doing this self.lookup_type.insert(id, lookup_type); pretty much uniformly in this function.
Could it be moved outside of match?
And then, since this function needs it by value, we likely need to change the parameter to be passed by value

That doesn't work because you don't have ownership over lookup_type, only a shared reference. Handles implement the Copy trait which is why it worked before, because it automatically copied the value.

While I haven't looked much at the code it looks like you actually do need to copy the lookup type here, but that would require crate::Type to be Clone and that you write LookupType<T: Clone> everywhere, which is messy. All the problems stem from the LocalType::Local(T) part, so I would look at another way to represent that.

Did you consider Local(&'a T) instead?

I haven't tried using a lifetime yet, I am still exploring the options. But I indeed stumbled across having to implement Clone for all generics that will be using this.

Let's try to avoid IR changes for this an see if &'a T would be sufficient

Napokue · 2020-06-30T22:04:08Z

src/back/spv/writer.rs

+        let ty = match lookup_type {
+            LookupType::Handle(handle) => &arena[*handle],
+            LookupType::Local(value) => &value,
+        };


This feels pretty wrong to do it like this. Would it be better to let both LookupTypes implement a trait, where I can retrieve type T from, instead of matching and trying to get the Type (in this case).

it's fine if we don't do this many times :)

Napokue · 2020-06-30T22:05:15Z

src/back/spv/writer.rs

@@ -724,9 +769,13 @@ impl Writer {
    fn parse_type_declaration(
        &mut self,
        arena: &crate::Arena<crate::Type>,
-        handle: crate::Handle<crate::Type>,
+        lookup_type: &LookupType<crate::Type>,


I can change this to a value type, and deference it at the inserts in this function. Problem is that in get_type_id I get the "Cannot move" error.

please tell more!

kvark · 2020-06-30T22:58:31Z

src/back/spv/writer.rs

+    Local(T),
+}
+
+impl<T> PartialEq for LookupType<T>


Pretty sure derive(PartialEq) would give you the same implementation. Any reason we do it manually here?

Yes, because when I don't implement this, I get the error: "Binary operations '==' cannot be applied to type '&back::spv::LookupType<>'

kvark · 2020-06-30T22:59:07Z

src/back/spv/writer.rs

+}
+
+trait LookupHelperNew<T> {
+    fn get_id(&self, value: &LookupType<T>) -> Option<&Word>;


should return Word without a reference

kvark · 2020-06-30T23:01:10Z

src/back/spv/writer.rs

    ) -> (Instruction, Word) {
-        let ty = &arena[handle];
+        let ty = match lookup_type {


please don't match on a reference, instead let's match the type match *lookup_type

kvark · 2020-06-30T23:01:35Z

src/back/spv/writer.rs

+        let ty = match lookup_type {
+            LookupType::Handle(handle) => &arena[*handle],
+            LookupType::Local(value) => &value,
+        };


it's fine if we don't do this many times :)

kvark · 2020-06-30T23:05:57Z

src/back/spv/writer.rs

                instruction = self.instruction_type_vector(id, scalar_id, size);
-                self.lookup_type.insert(id, handle);
+                self.lookup_type.insert(id, lookup_type);


it looks like you are doing this self.lookup_type.insert(id, lookup_type); pretty much uniformly in this function.
Could it be moved outside of match?
And then, since this function needs it by value, we likely need to change the parameter to be passed by value

kvark · 2020-06-30T23:06:20Z

src/back/spv/writer.rs

@@ -724,9 +769,13 @@ impl Writer {
    fn parse_type_declaration(
        &mut self,
        arena: &crate::Arena<crate::Type>,
-        handle: crate::Handle<crate::Type>,
+        lookup_type: &LookupType<crate::Type>,


please tell more!

Napokue · 2020-08-04T18:00:45Z

src/back/spv/writer.rs

+            let new_ty: &'a crate::Type = &crate::Type {
+                name: None,
+                inner: crate::TypeInner::Scalar {
+                    width,
+                    kind
+                },
+            };


I try to create a new type here, but I am not sure why it does not work. Getting this lifetime error:

You are constructing the type on the stack, and then getting a reference to it. This doesn't fulfill "`a".

I think your current LookupType::Local variant is dysfunctional: the only thing in "'a" that owns the types is the module's arenat, and if you have the type there, you might as well use LookupType::Handle.

kvark · 2020-07-13T01:21:40Z

src/back/spv/writer.rs

+{
+    fn get_id(&self, value: &LookupType<'a, T>) -> Option<&Word> {
+        let mut result = None;
+        for (k, v) in self.iter() {


this could be expressed a bit nicer, something like self.iter().find(|&(k, v)| v == value).map(|(k, _)| k)

kvark · 2020-08-04T19:53:21Z

src/back/spv/writer.rs

+            let new_ty: &'a crate::Type = &crate::Type {
+                name: None,
+                inner: crate::TypeInner::Scalar {
+                    width,
+                    kind
+                },
+            };


You are constructing the type on the stack, and then getting a reference to it. This doesn't fulfill "`a".

I think your current LookupType::Local variant is dysfunctional: the only thing in "'a" that owns the types is the module's arenat, and if you have the type there, you might as well use LookupType::Handle.

src/back/spv/writer.rs

Napokue · 2020-08-06T22:25:32Z

src/back/spv/writer.rs

+                LookupType::Local(local_ty) => {
+                    match local_ty {
+                        LocalType::Scalar(ty_inner) => &ty.inner == ty_inner,
+                        // TODO Pointers


Had an equivalent check for LocalType::Pointer, but it caused a weird behavior that a OpTypePointer referencing a result type of OpTypePointer would be created. Which is not allowed in SPIR-V.

src/back/spv/writer.rs

kvark

I think we are on the right track!

src/back/spv/writer.rs

kvark · 2020-08-06T22:31:45Z

src/back/spv/writer.rs

@@ -66,7 +78,7 @@ pub struct Writer {
    annotations: Vec<Instruction>,
    writer_flags: WriterFlags,
    void_type: Option<u32>,
-    lookup_type: FastHashMap<Word, crate::Handle<crate::Type>>,
+    lookup_type_new: FastHashMap<Word, LookupType>,


I believe all of the lookup tables here need to be reversed (were they copied from the front-end?). I.e. you should never need to lookup IR stuff by a spirv ID. You need the opposite: given IR handles and stuff, be able to produce spirv IDs that you are saving to the file. So this table (and all others) need to look like this:

lookup_type: FastHashMap<LocalType, Word>,

Done this, as result, I had to modify the lib.rs to include Hash and Eq on a couple of structs.

src/back/spv/writer.rs

kvark · 2020-08-09T13:56:37Z

src/back/spv/writer.rs

@@ -10,38 +11,19 @@ enum Signedness {
    Signed = 1,
 }

-#[derive(Debug)]
+#[derive(Hash, Eq, PartialEq, Debug)]
 enum LocalType {
    Scalar(crate::TypeInner),


why are we storing TypeInner here? I.e. if we are going to store TypeInner, we wouldn't need LocalType at all :) The idea was to have these types fully specified, i.e.

enum LocalType { Scalar { kind, width }, Vector { size, kind, width }, Pointer { base, class }, }

kvark · 2020-08-09T13:59:26Z

src/lib.rs

@@ -181,7 +181,7 @@ pub struct Type {

 /// Enum with additional information, depending on the kind of type.
 // Clone is used only for error reporting and is not intended for end users
-#[derive(Clone, Debug, PartialEq)]
+#[derive(Clone, Debug, PartialEq, Hash, Eq)]


I think with the changes to LocalType that I suggested, these derives are no longer going to be needed

kvark · 2020-08-09T13:59:38Z

src/lib.rs

@@ -134,15 +134,15 @@ pub enum MemberOrigin {

 /// Member of a user-defined structure.
 // Clone is used only for error reporting and is not intended for end users
-#[derive(Clone, Debug, PartialEq)]
+#[derive(Clone, Debug, PartialEq, Hash, Eq)]


these shouldn't be needed either

src/lib.rs

kvark

Alright, we are almost there :) thank you for your patience!

src/back/spv/writer.rs

kvark · 2020-08-09T18:02:34Z

src/back/spv/writer.rs

@@ -1145,20 +1230,24 @@ impl Writer {
                            crate::TypeInner::Vector { size, kind, width } => {
                                vector_id = Some(left_id);
                                for (k, v) in self.lookup_type.iter() {


why are we not just issuing a lookup into the map? we shouldn't need to iterate the arenas anywhere more than once, really

I completely agree, this is probably the most dirtiest piece of code in this back-end. I will take a look to fix it, so it won't iterate the arena again. This piece of code will be completely refactored in the future I presume, as it only supports very basic cases.

Oh and by the way, as I see now, LocalType's arent even supported here. I will fix that too.

kvark · 2020-08-09T18:03:02Z

src/back/spv/writer.rs

@@ -1169,20 +1258,24 @@ impl Writer {
                            crate::TypeInner::Vector { size, kind, width } => {
                                vector_id = Some(right_id);
                                for (k, v) in self.lookup_type.iter() {


src/lib.rs

Napokue · 2020-08-09T19:58:32Z

No worries at all, we are working towards the same goal. I will look into it later today

kvark

Almost there!

src/back/spv/writer.rs

kvark · 2020-08-10T15:20:58Z

src/back/spv/writer.rs

@@ -865,15 +858,21 @@ impl Writer {

        let instruction;

-        match ty.inner.clone() {
+        match ty.inner {
            crate::TypeInner::Scalar { kind, width } => {
                instruction = self.write_scalar(id, kind, width);


I think this should use get_or_create_scalar

No that is not possible, get_or_create_scalar and get_or_create_vector only returns a Word/Id, not an instruction.

kvark · 2020-08-10T15:21:26Z

src/back/spv/writer.rs

            crate::TypeInner::Scalar { kind, width } => {
                instruction = self.write_scalar(id, kind, width);
                self.lookup_type.insert(LookupType::Handle(handle), id);
+                self.lookup_type
+                    .insert(LookupType::Local(LocalType::Scalar { kind, width }), id);
            }
            crate::TypeInner::Vector { size, kind, width } => {
                let scalar_id = self.get_or_create_scalar(arena, kind, width);
                instruction = self.instruction_type_vector(id, scalar_id, size);


this should use get_or_create_vector

src/back/spv/writer.rs

kvark

Thank you for updating this!

I see the main concern in that additions to the lookup maps happen at different stack levels, in different places. It would be nice to coalesce them, and also never drop Entry::Vacant and just use it right away.

There is a problem you'll face though: if Entry::Vacant is alive, you can't mutate the lookup map for adding dependant entries. So we should collect them somewhere, i.e. a function would have a parameter of type impl Extend<(LocalType, spirv::Id)>, and after such function is called, the caller can deal with Entry::Vacant, then release the hold of the map, and then iterate these extra dependents and push insert to the map.

src/back/spv/writer.rs

kvark · 2020-08-16T18:40:17Z

src/back/spv/writer.rs

+        if let Entry::Occupied(e) = self.lookup_type.entry(lookup_ty) {
+            *e.get()
+        } else {
+            match lookup_ty {


when we add a new type, we have an ID. So we should write down it in the map, i.e.

match self.lookup_type(lookup_ty) { Entry::Occupied(e) => e.get(), Entry::Vacant(e) => { let id = ...; *e.insert(id) } }

kvark · 2020-08-16T18:40:44Z

src/back/spv/writer.rs

+        match self.lookup_constant.entry(handle) {
+            Entry::Occupied(e) => *e.get(),
+            _ => {
+                let (instruction, id) = self.write_constant_type(handle, ir_module);


same here, we should add it to the map with e.insert(id)

kvark · 2020-08-16T18:40:52Z

src/back/spv/writer.rs

-        match self.lookup_global_variable.lookup_id(handle) {
-            Some(word) => word,
-            None => {
+        match self.lookup_global_variable.entry(handle) {


kvark · 2020-08-16T18:42:07Z

src/back/spv/writer.rs

+                    })) {
+                    Entry::Occupied(e) => *e.get(),
+                    _ => {
+                        let pointer_id = self.generate_id();


and here, we should insert it into the map

kvark · 2020-08-16T18:44:32Z

src/back/spv/writer.rs

+            LocalType::Pointer { .. } => unimplemented!(),
+        };
+
+        self.lookup_type.insert(LookupType::Local(local_ty), id);


I see what you did here. Let's cut the loop. Instead of doing match (entry) -> call function -> insert entry just do match (entry) -> insert entry as we have the Entry::Vacant there.

Napokue · 2020-08-17T19:58:50Z

src/back/spv/writer.rs

+        let id = match self.lookup_type.entry(lookup_ty) {
+            Entry::Occupied(e) => *e.get(),
+            Entry::Vacant(e) => {
+                let id = match lookup_ty {
+                    LookupType::Handle(handle) => self.write_type_declaration_arena(arena, handle, &mut sink),
+                    LookupType::Local(local_ty) => self.write_type_declaration_local(arena, local_ty),
+                };
+                *e.insert(id)
            }
+        };


I am getting the "cannot borrow *self as mutable more than once at a time" over here. I understand why, but I am not sure yet how to resolve this yet.

The functions have to be mutable, even if we extract the part let id = self.generate_id() from these functions and use it here.

hmm, right. I suppose we need to refactor write_type_declaration_arena to not work with self. Or at least, that's one way to address this...
if it's too difficult, we can go back to your original code, which I think was correct, and I'll look at it later to try to improve

Yes I would prefer that. This means, the rest of the comments are also resolved?

sure! Are we all set for landing then?

Napokue · 2020-08-17T19:59:50Z

src/back/spv/writer.rs

+        };
+
+        for (k, v) in sink.iter() {
+          self.lookup_type.insert(*k, *v);


I think this works, but it is not clean, as the sink will always be initialized even though a LookupType is local.

kvark

Thank you for pushing this through!

Napokue requested a review from kvark June 30, 2020 21:58

monocodus bot suggested changes Jun 30, 2020

View reviewed changes

Napokue force-pushed the spirv-lookup-refactor branch from 6890e70 to 42f7979 Compare June 30, 2020 22:00

Napokue commented Jun 30, 2020

View reviewed changes

kvark reviewed Jun 30, 2020

View reviewed changes

Napokue force-pushed the spirv-lookup-refactor branch from f9310ef to e80f414 Compare July 11, 2020 16:22

Napokue force-pushed the spirv-lookup-refactor branch from e80f414 to 05da859 Compare July 29, 2020 18:33

Napokue commented Aug 4, 2020

View reviewed changes

kvark reviewed Aug 4, 2020

View reviewed changes

Napokue force-pushed the spirv-lookup-refactor branch from 26c97b5 to 4acb5cf Compare August 6, 2020 22:13

Napokue commented Aug 6, 2020

View reviewed changes

src/back/spv/writer.rs Outdated Show resolved Hide resolved

Napokue commented Aug 6, 2020

View reviewed changes

src/back/spv/writer.rs Outdated Show resolved Hide resolved

Napokue commented Aug 6, 2020

View reviewed changes

src/back/spv/writer.rs Outdated Show resolved Hide resolved

kvark requested changes Aug 6, 2020

View reviewed changes

Napokue marked this pull request as ready for review August 8, 2020 11:42

kvark reviewed Aug 9, 2020

View reviewed changes

Napokue force-pushed the spirv-lookup-refactor branch from eee5807 to a67419b Compare August 9, 2020 15:49

Napokue commented Aug 9, 2020

View reviewed changes

src/lib.rs Show resolved Hide resolved

Napokue force-pushed the spirv-lookup-refactor branch from 3ef675c to 70ac39f Compare August 9, 2020 16:58

kvark requested changes Aug 9, 2020

View reviewed changes

kvark reviewed Aug 10, 2020

View reviewed changes

kvark requested changes Aug 16, 2020

View reviewed changes

Napokue commented Aug 17, 2020

View reviewed changes

Napokue force-pushed the spirv-lookup-refactor branch 3 times, most recently from 31eecf0 to 4d71e1d Compare August 17, 2020 22:07

kvark approved these changes Aug 17, 2020

View reviewed changes

[spirv-out] Refactor lookup

e4bf0b0

Napokue force-pushed the spirv-lookup-refactor branch from 4d71e1d to e4bf0b0 Compare August 17, 2020 22:14

kvark merged commit 58dd54f into gfx-rs:master Aug 17, 2020

Napokue deleted the spirv-lookup-refactor branch August 17, 2020 22:21

Napokue mentioned this pull request Aug 17, 2020

Scalar type in Vector is not added to the IR #85

Closed

Spirv lookup refactor #88

Spirv lookup refactor #88

Conversation

Napokue commented Jun 30, 2020 • edited Loading

monocodus bot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kvark left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kvark left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Napokue Aug 9, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Napokue commented Aug 9, 2020

kvark left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kvark left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kvark left a comment

Choose a reason for hiding this comment

Napokue commented Jun 30, 2020 •

edited

Loading

Napokue Aug 9, 2020 •

edited

Loading