-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Camera position optimization with differentiable rendering using depth image #207
Comments
@guanming001 your notebook is not readable. For us to be most effective with resolving issues, it'd be best if you could provide concrete errors from the codebase. We can't be consultants for people's projects or help them make their ideas work. However, if you find that a PyTorch3D op should output something but it outputs something else, then this is something we can help you with and we will debug this and try to fix it. |
Hi @gkioxari sorry for the inconvenience caused, I have extracted the code portion from the raw format of jupyter notebook and the full version of the python code is attached at the end. Below are a quick overview of the main changes I have made to test optimization with differentiable rendering using depth image: I added a
To extract the depth reference image, I used the
Inside the basic model class, I have modified the reference image to depth and set the initial camera position closer to the reference depth image (hoping for a simpler optimization):
When the forward method is called, the loss is computed based on the difference of the depth images:
I have also lowered the learning rate of Adam optimizer:
Even with the simpler starting position and lower learning rate, the optimization process seems to be unable to converge correctly (as shown in the .gif where the camera is continuously moving closer to the object) Full version of the python code: camera_position_optimization_with_differentiable_rendering_depth.py
Thank you for your help! |
Hi @guanming001 I want to emphasize that we are not here to help you train your network for the task at hand. We provide the nails and hammers, but you are in charge of building your own house. |
Hi @gkioxari thank you for the reply. I have managed to get the optimization working by setting the original invalid depth values (negative ones) to an arbitrary large positive number
Example of using back the original camera starting position and learning rate of Adam optimizer (0.05), the optimization can converge most of the time. |
I meet the same problem using phong_renderer is not stable for optimization |
Hi thanks for creating this amazing library!
I tried to modify the tutorial example
camera_position_optimization_with_differentiable_rendering.ipynb to optimize the camera position using rendered depth image instead of the silhouette image.
But the results seem weird as the camera position does not converge to the desired location but keeps zooming into the object. Did I miss out anything?
Thank you.
The modified jupyter notebook is attached below for reference:
camera_position_optimization_with_differentiable_rendering_depth.ipynb
The text was updated successfully, but these errors were encountered: