-
Notifications
You must be signed in to change notification settings - Fork 1.3k
-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Is there a differentiable inverse render function? #1074
Comments
Do you mean gather information on 2D image plane and back-project them onto 3D vertex/faces? I'm also interested in that, I think the shading after rasterization make things harder because a color can result from multiple lighing configurations. |
Good point. I would be happy to start out with a naive implementation that doesn't take lighting etc into consideration though. |
Thanks for the advice! I'm currently using a TexturesVertex but I've heard that TexturesAtlas might look better. It's just been using a lot of gpu when I tried to get it to work. |
I'm trying to implement this with lightning included (hard goraud) and it was pretty straight forward but now I'm a little stuck on a good way to calculate the inverse for this step:
from Do you have any suggestions for Edit: I misunderstood the purpose, aggregation of the K neighbours happens in Edit2: Doing this now (keeping it proportional to results from the render forward pass), does that seem reasonable?
|
In hindsight with a little more understanding, using the proportions of the previous texture obviously doesn't work. I haven't figured out how to do this nicely so I have a starting guess + optimization to calculate I optimize over this block to find some solution:
Trying to make this differentiable is under the assumption that the expression looks like this and that
If that's actually true then it should be testable I should be able to only calculate the "constant" once per camera. |
Got this working finally. It was a lot more work than I anticipated. Was initially thinking about creating a PR for it but I already spent too much time on this. |
@samedii Could you share a working notebook on how you solved it? I'm very interested. Thanks! |
I think this could be a very useful module. I think it is already possible to do but I'm not familiar enough with the "internals" of the renderer.
My thought is that I need to:
pix_to_face
?)Do you have any tips for me on what functions/variables I should use to do this?
I've done reverse rendering by backprop + optimization previously but would like to differentiate through the process now.
The text was updated successfully, but these errors were encountered: