Support for quads

[pathquester]

Would it be within the scope of the project to also support quads for meshes? That would add tremendous utility.

[lucidrains]

@pathquester hey! yea, maybe i can slot that into my limited time left

what are quads? is it just building meshes out of rectangles?

[lucidrains]

as you can tell, i know nothing about the domain 🤣

[pathquester]

@lucidrains Hi, awesome!! Yeah instead of triangular meshes, quad meshes make modeling a joy since you can add edge loops easily and subdivide further, among other benefits. If it remains limited to triangular meshes, that means we'll very often have to retopologize the mesh generated afterwards.

[lucidrains]

@pathquester if it just means adding an extra vertex to each face (4 vertices instead of 3), that would be easy. do people ever mix quads and triangles in the same mesh? that may complicate things

[lucidrains]

@pathquester wait, i'm geometrically challenged, but isn't it true that every quad can be broken into two triangles?

[lucidrains]

@pathquester let me read up on the subject first to get a surface level understanding. will get back to this maybe near month's end, once i wrap up a project in flight

[pathquester]

@lucidrains They can be mixed yes although quads-only is arguably better than mixed, supporting just that should be ideal. That is fantastic, I would be very happy to help with this. Yes that's true also more quads thus increase the resolution, subdivision surface modifiers are mostly associated with quads.

[lucidrains]

@pathquester maybe the issue is that when you have the attention net spit out the vertices, with 4 it may be difficult for it to maintain that they are within the same plane. yeah i'm not sure if it will work

[lucidrains]

anyways, i know nothing lol

i'll get back to you on this

[pathquester]

@lucidrains Ah yeah there may be challenges. Anyhow, many thanks!!

[MarcusLoppe]

@lucidrains

I brought this up in previous conversations with you, I did some testing and it reduces the face/triangles count by about 50%.
So this would also increase the performance and hardware requirements, and who knows maybe the quads are easier for the model to learn.
Most of the 3D work seems to be quad based and all the 3d meshes can be converted to quads using blender so it would be a extra step to convert the meshes.
I think it would be worth the hassle since it reduces the triangle count to 50% but adds additional vertex per face, but the transformer would save 6 tokens per triangle, e.g from 19 200 tokens to 9 600 tokens.
I don't think you can improve the transformer that much since this would also half the required context length.

Below is example of a 3200 triangle mesh convert into 1620 quads
Triangles:

Quads:

Here is another example,
Face edges reduced from 5768 to 4336
Face count reduced from 3688 to 2256

Triangles

Quads

[lucidrains]

@MarcusLoppe oh yes, now I recall

ok, I think I should be able to do this before mid March

[MarcusLoppe]

@MarcusLoppe oh yes, now I recall

ok, I think I should be able to do this before mid March

@lucidrains
Is this something I can implement? Wouldn't this just require changing the einops expressions, 3's to 4 and the face edge generator?

[lucidrains]

yeah! and also when decoding from attention net, have to terminate on multiple of 4s

[lucidrains]

believe face edges logic stays the same

[lucidrains]

@MarcusLoppe you'll be doing something no one has ever done. top of the line research 🤣

[lucidrains]

have a feeling quads will require synthetic pretraining, however. also on sampling will need to snap malformed quads to nearest plane

[MarcusLoppe]

@MarcusLoppe you'll be doing something no one has ever done. top of the line research 🤣

@lucidrains
I'm sure that someone have tried this, I'll check and see if there is anyone that have compared triangle/quad generation.

have a feeling quads will require synthetic pretraining, however. also on sampling will need to snap malformed quads to nearest plane

Why synthetic pre-training and snapping? The data it generates would just contain a extra vertex, that doesn't seem such a dramatic change?

[lucidrains]

@MarcusLoppe i think with vertices, you can generate invalid quads (the 4 points do not lie in the same plane). but like i said before, i'm geometrically challenged lol

[lucidrains]

@MarcusLoppe you'll be doing something no one has ever done. top of the line research 🤣

@lucidrains I'm sure that someone have tried this, I'll check and see if there is anyone that have compared triangle/quad generation.

have a feeling quads will require synthetic pretraining, however. also on sampling will need to snap malformed quads to nearest plane

Why synthetic pre-training and snapping? The data it generates would just contain a extra vertex, that doesn't seem such a dramatic change?

I don't think so. this is the first work to autoregress triangles, so surely no one has tried quads

[ell-hol]

@lucidrains that's exactly why quads are interesting. By not lying on the same plane they allow you to describe more complex shapes.

[lucidrains]

forgive ignorance, but how do you render a quad that has vertices not in the same plane?

[MarcusLoppe]

forgive ignorance, but how do you render a quad that has vertices not in the same plane?

@lucidrains
Not sure but blender seems to handle it fine, otherwise you can just convert the non-planar to triangles. No biggie.

Thread about non-coplanar quads:
https://blenderartists.org/t/non-coplanar-quads-avoid/1113775
Converting:
https://developer.rhino3d.com/api/rhinocommon/rhino.geometry.collections.meshfacelist/convertnonplanarquadstotriangles?version=8.x

But one thing I'm not sure about if it's always possible to convert into quads from triangles, for example the image below; blender is quite a capable software but it looks like a triangle unless it's a quad with one duplicated vertex.
Is it possible to do some kind of padding if this is the case?

[lucidrains]

that's so cool! ok I'm more optimistic about quads now 🤣

[lucidrains]

@MarcusLoppe decided I'll just knock it out Sunday morning, if you don't get to it first

[MarcusLoppe]

@MarcusLoppe decided I'll just knock it out Sunday morning, if you don't get to it first

I'll probably have tried and failed a couple times till then, but we'll see 😄

Any idea regarding the issue about not all triangles can be converted to quads? Are you going for the idea about adding a extra dimension on the face mask for padding or just force the triangles into quads with a duplicated vertex?

[MarcusLoppe]

@MarcusLoppe decided I'll just knock it out Sunday morning, if you don't get to it first

@lucidrains
I'm trying to figure out the derive_face_edges_from_faces, is it supposed to generate face_edges to itself?
For example of a 6.2k triangle mesh, 6206 of 24706 of the edges are connected to itself like the below:

tensor([0, 0])
tensor([1, 1])
tensor([2, 2])
tensor([3, 3])
tensor([4, 4])

[lucidrains]

@MarcusLoppe hey, very good question! i think it should! investigated this before building out the autoencoder, and noticed this setting

[lucidrains]

without knowing graph neural net literature very well, i assume they usually add the self edges with that function during loading of the data. it would make sense, as the paper only had sageconvs, and usually in any neural network you need some sort of connection to self (ie feedforwards in transformer)

[lucidrains]

let me know if you find otherwise

[lucidrains]

@MarcusLoppe decided I'll just knock it out Sunday morning, if you don't get to it first

I'll probably have tried and failed a couple times till then, but we'll see 😄

Any idea regarding the issue about not all triangles can be converted to quads? Are you going for the idea about adding a extra dimension on the face mask for padding or just force the triangles into quads with a duplicated vertex?

i didn't think that far haha, but duplicated vertex sounds like a good idea. the correct way is probably to have a dummy pad vertex that the transformer predicts, so a triangle would be a quad with a pad vertex. that would require an extra token on the transformer and some other engineering

[MarcusLoppe]

@MarcusLoppe hey, very good question! i think it should! investigated this before building out the autoencoder, and noticed this setting

Ah, correct, it seems like the GCN's uses self-loops:
https://towardsdatascience.com/what-makes-graph-convolutional-networks-work-53badade0ce9

i didn't think that far haha, but duplicated vertex sounds like a good idea. the correct way is probably to have a dummy pad vertex that the transformer predicts, so a triangle would be a quad with a pad vertex. that would require an extra token on the transformer and some other engineering

I think I'd let you figure out if it's possible, I can't even get past the project_in layer 😢

[lucidrains]

got it done this morning, at the expense of another project. hope this goes somewhere! i'll fix the derived features before mid March, for quads it is a bit different.

import torch

from meshgpt_pytorch import (
    MeshAutoencoder,
    MeshTransformer
)

# autoencoder

autoencoder = MeshAutoencoder(
    num_discrete_coors = 128,
    quads = True
)

# mock inputs

vertices = torch.randn((2, 121, 3))            # (batch, num vertices, coor (3))
faces = torch.randint(0, 121, (2, 64, 4))      # (batch, num faces, vertices (4))

# make sure faces are padded with `-1` for variable lengthed meshes

# forward in the faces

loss = autoencoder(
    vertices = vertices,
    faces = faces
)

loss.backward()

# after much training...
# you can pass in the raw face data above to train a transformer to model this sequence of face vertices

transformer = MeshTransformer(
    autoencoder,
    dim = 512,
    max_seq_len = 768,
    quads = True
)

loss = transformer(
    vertices = vertices,
    faces = faces
)

loss.backward()

# after much training of transformer, you can now sample novel 3d assets

faces_coordinates, face_mask = transformer.generate()

[lucidrains]

@MarcusLoppe if you want to still contribute, def welcome you to look into breaking up the derived features for triangles vs quads. based on the way you analyze things (and the way you code), i think you are on your way to becoming a great ML engineer. i'm thinking, perhaps for quads, it should be good for the network to know whether the 4 points are coplanar or not. as for normals, think you can get away by breaking the quad into two triangles and just offering the discretized normal for both triangles

[lucidrains]

ok, i got to leave this project for reals now. already done enough for it

[MarcusLoppe]

ok, i got to leave this project for reals now. already done enough for it

Alright 😄 I thank you very much @lucidrains for the project and all that you have done, good luck on your other projects ❤️

@MarcusLoppe if you want to still contribute, def welcome you to look into breaking up the derived features for triangles vs quads. based on the way you analyze things (and the way you code), i think you are on your way to becoming a great ML engineer. i'm thinking, perhaps for quads, it should be good for the network to know whether the 4 points are coplanar or not. as for normals, think you can get away by breaking the quad into two triangles and just offering the discretized normal for both triangles

Can't promise too much but I'll see if I can manage to get it done.

But I can report that quads works 😄 Instead of using 5688 tokens it just used 3792 tokens which is about 36% token reduction 😮
So congrats! The first autoregressive quad generator! 🎉

Only downside is that using mixed quads and triangles doesn't work, it triggers some fatal cuda error so I can't provide any errors.
But I have around 60k meshes from Objaverse and Shapenet so the training dataset should be fine.

I got the autoencoder to 0.29 loss after 45 epochs, and the transformer to 0.03 after 10minutes.
I trained on 50 augmentation of the same model (jitter + scale), on the left is the ground truth and the middle is the autoencoder and on the right is the transformer generated

For the autoencoder I generated the codes and then let it decode them, it was very accurately (0.00002 MSE loss).

[lucidrains]

woohoo! thank you Marcus for the quick validation! more helpful to me than you could know

[pathquester]

That's really wonderful!! Kudos!