Skip to content

Commit

Permalink
Add lighting features (#663)
Browse files Browse the repository at this point in the history
Signed-off-by: Clement Fuji Tsang <cfujitsang@nvidia.com>

Signed-off-by: Clement Fuji Tsang <cfujitsang@nvidia.com>
  • Loading branch information
Caenorst committed Dec 6, 2022
1 parent 3c1038a commit 609671b
Show file tree
Hide file tree
Showing 73 changed files with 6,416 additions and 7 deletions.
27 changes: 27 additions & 0 deletions Acknowledgements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -51,3 +51,30 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

========================

Spherical Harmonics and Spherical Gaussians components
https://github.com/TheRealMJP/BakingLab

MIT License

Copyright (c) 2016 MJP

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
16 changes: 16 additions & 0 deletions ci/gitlab_jenkins_templates/ubuntu_test_CI.jenkins
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,22 @@ spec:
build_passed = false
echo e.toString()
}
try {
stage("Diffuse lighting Tutorial") {
sh 'cd /kaolin/examples/tutorial && ipython diffuse_lighting.ipynb'
}
} catch(e) {
build_passed = false
echo e.toString()
}
try {
stage("Spherical Gaussian lighting Tutorial") {
sh 'cd /kaolin/examples/tutorial && ipython sg_specular_lighting.ipynb'
}
} catch(e) {
build_passed = false
echo e.toString()
}
try {
stage("SPC from Pointcloud Recipe") {
sh 'cd /kaolin/examples/recipes/dataload/ && python spc_from_pointcloud.py'
Expand Down
6 changes: 3 additions & 3 deletions ci/gitlab_jenkins_templates/windows_build_CI.jenkins
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,9 @@ spec:
value: ""
resources:
requests:
memory: 16Gi
memory: 32Gi
limits:
memory: 16Gi
memory: 32Gi
nodeSelector:
kubernetes.io/os: windows
''')
Expand Down Expand Up @@ -72,7 +72,7 @@ spec:
cudaUrl = cuda_version_url[cudaVer]
targetImage = docker.build(
"${targetImageTag}",
"""-m 16g --no-cache -f ./tools/windows/Dockerfile \
"""-m 32g --no-cache -f ./tools/windows/Dockerfile \
--build-arg CUDA_VERSION=${cudaVer} \
--build-arg CUDA_URL=${cudaUrl} \
--build-arg PYTHON_VERSION=${pythonVer} \
Expand Down
16 changes: 16 additions & 0 deletions ci/gitlab_jenkins_templates/windows_test_CI.jenkins
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,22 @@ spec:
'''
}
}
stage("Diffuse lighting Tutorial") {
catchError(stageResult: "failure") {
powershell '''
cd c:\\kaolin\\examples\\tutorial
ipython diffuse_lighting.ipynb
'''
}
}
stage("Spherical Gaussian lighting Tutorial") {
catchError(stageResult: "failure") {
powershell '''
cd c:\\kaolin\\examples\\tutorial
ipython sg_specular_lighting.ipynb
'''
}
}
// requires nvdiffrast. not currently supported on Windows
// stage("Camera and Rasterization Tutorial") {
// catchError(stageResult: "failure") {
Expand Down
2 changes: 2 additions & 0 deletions docs/kaolin_ext.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ def run_apidoc(_):
"kaolin/ops/spc/convolution.py",
"kaolin/ops/spc/points.py",
"kaolin/ops/spc/uint8.py",
"kaolin/render/lighting/sg.py",
"kaolin/render/lighting/sh.py",
"kaolin/render/mesh/deftet.py",
"kaolin/render/mesh/dibr.py",
"kaolin/render/mesh/rasterization.py",
Expand Down
3 changes: 2 additions & 1 deletion docs/modules/kaolin.render.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ kaolin.render
:maxdepth: 2
:titlesonly:

kaolin.render.mesh
kaolin.render.camera
kaolin.render.lighting
kaolin.render.mesh
kaolin.render.spc
12 changes: 12 additions & 0 deletions docs/notes/tutorial_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,18 @@ Detailed Tutorials
* Create a camera with ``from_args()`` general constructor
* Render a mesh with multiple materials with nvdiffrast
* Move camera and see the resulting rendering
* `Optimizing Diffuse Lighting <https://github.com/NVIDIAGameWorks/kaolin/blob/master/examples/tutorial/diffuse_lighting.ipynb>`_: Optimize lighting parameters with spherical gaussians and spherical harmonics:
* Load an obj mesh with normals and materials
* Rasterize the diffuse and specular albedo
* Render and optimize diffuse lighting:
* Spherical harmonics
* Spherical gaussian with inner product implementation
* Spherical gaussian with fitted approximation
* `Optimize Diffuse and Specular Lighting with Spherical Gaussians <https://github.com/NVIDIAGameWorks/kaolin/blob/master/examples/tutorial/sg_specular_lighting.ipynb>`_:
* Load an obj mesh with normals and materials
* Generate view rays from camera
* Rasterize the diffuse and specular albedo
* Render and optimize diffuse and specular lighting with spherical gaussians

Simple Recipes
--------------
Expand Down
583 changes: 583 additions & 0 deletions examples/tutorial/diffuse_lighting.ipynb

Large diffs are not rendered by default.

430 changes: 430 additions & 0 deletions examples/tutorial/sg_specular_lighting.ipynb

Large diffs are not rendered by default.

8 changes: 7 additions & 1 deletion kaolin/csrc/bindings.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,9 @@
#include "./metrics/sided_distance.h"
#include "./metrics/unbatched_triangle_distance.h"
#include "./render/mesh/deftet.h"
#include "./render/mesh/rasterization.h"
#include "./render/mesh/dibr_soft_mask.h"
#include "./render/mesh/rasterization.h"
#include "./render/sg/unbatched_reduced_sg_inner_product.h"
#include "./ops/conversions/mesh_to_spc/mesh_to_spc.h"
#include "./ops/spc/spc.h"
#include "./ops/spc/feature_grids.h"
Expand Down Expand Up @@ -89,6 +90,11 @@ PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
render_spc.def("sum_reduce_cuda", &sum_reduce_cuda);
render_spc.def("cumsum_cuda", &cumsum_cuda);
render_spc.def("cumprod_cuda", &cumprod_cuda);
py::module render_sg = render.def_submodule("sg");
render_sg.def("unbatched_reduced_sg_inner_product_forward_cuda",
&unbatched_reduced_sg_inner_product_forward_cuda);
render_sg.def("unbatched_reduced_sg_inner_product_backward_cuda",
&unbatched_reduced_sg_inner_product_backward_cuda);
}

} // namespace kaolin
220 changes: 220 additions & 0 deletions kaolin/csrc/render/sg/unbatched_reduced_sg_inner_product.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,220 @@
// Copyright (c) 2022 NVIDIA CORPORATION & AFFILIATES.
// All rights reserved.

// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at

// http://www.apache.org/licenses/LICENSE-2.0

// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

#include <ATen/ATen.h>
#include "../../check.h"

namespace kaolin {

#ifdef WITH_CUDA

void unbatched_reduced_sg_inner_product_forward_cuda_impl(
const at::Tensor intensity,
const at::Tensor direction,
const at::Tensor sharpness,
const at::Tensor other_intensity,
const at::Tensor other_direction,
const at::Tensor other_sharpness,
at::Tensor output);

void unbatched_reduced_sg_inner_product_backward_cuda_impl(
at::Tensor grad_out,
at::Tensor intensity,
at::Tensor direction,
at::Tensor sharpness,
at::Tensor other_intensity,
at::Tensor other_direction,
at::Tensor other_sharpness,
at::Tensor grad_intensity,
at::Tensor grad_direction,
at::Tensor grad_sharpness,
at::Tensor grad_other_intensity,
at::Tensor grad_other_direction,
at::Tensor grad_other_sharpness);

#endif

at::Tensor unbatched_reduced_sg_inner_product_forward_cuda(
at::Tensor intensity,
at::Tensor direction,
at::Tensor sharpness,
at::Tensor other_intensity,
at::Tensor other_direction,
at::Tensor other_sharpness) {

at::TensorArg intensity_arg{
intensity, "intensity", 1};
at::TensorArg direction_arg{
direction, "direction", 2};
at::TensorArg sharpness_arg{
sharpness, "sharpness", 3};
at::TensorArg other_intensity_arg{
other_intensity, "other_intensity", 4};
at::TensorArg other_direction_arg{
other_direction, "other_direction", 5};
at::TensorArg other_sharpness_arg{
other_sharpness, "other_sharpness", 6};

at::checkAllSameGPU(__func__, {
intensity_arg,
direction_arg,
sharpness_arg,
other_intensity_arg,
other_direction_arg,
other_sharpness_arg
});
at::checkAllContiguous(__func__, {
intensity_arg,
direction_arg,
sharpness_arg,
other_intensity_arg,
other_direction_arg,
other_sharpness_arg
});
at::checkAllSameType(__func__, {
intensity_arg,
direction_arg,
sharpness_arg,
other_intensity_arg,
other_direction_arg,
other_sharpness_arg
});

const int num_sg = intensity.size(0);
const int num_other = other_intensity.size(0);

at::checkSize(__func__, intensity_arg, {num_sg, 3});
at::checkSize(__func__, direction_arg, {num_sg, 3});
at::checkSize(__func__, sharpness_arg, {num_sg});
at::checkSize(__func__, other_intensity_arg, {num_other, 3});
at::checkSize(__func__, other_direction_arg, {num_other, 3});
at::checkSize(__func__, other_sharpness_arg, {num_other});

at::Tensor output = at::zeros_like(intensity);

#ifdef WITH_CUDA
if (num_sg > 0) {
unbatched_reduced_sg_inner_product_forward_cuda_impl(
intensity,
direction,
sharpness,
other_intensity,
other_direction,
other_sharpness,
output
);
}
#else
KAOLIN_NO_CUDA_ERROR(__func__);
#endif
return output;
}

std::vector<at::Tensor> unbatched_reduced_sg_inner_product_backward_cuda(
at::Tensor grad_out,
at::Tensor intensity,
at::Tensor direction,
at::Tensor sharpness,
at::Tensor other_intensity,
at::Tensor other_direction,
at::Tensor other_sharpness) {

at::TensorArg grad_out_arg{
grad_out, "grad_out", 1};
at::TensorArg intensity_arg{
intensity, "intensity", 2};
at::TensorArg direction_arg{
direction, "direction", 3};
at::TensorArg sharpness_arg{
sharpness, "sharpness", 4};
at::TensorArg other_intensity_arg{
other_intensity, "other_intensity", 5};
at::TensorArg other_direction_arg{
other_direction, "other_direction", 6};
at::TensorArg other_sharpness_arg{
other_sharpness, "other_sharpness", 7};

at::checkAllSameGPU(__func__, {
grad_out_arg,
intensity_arg,
direction_arg,
sharpness_arg,
other_intensity_arg,
other_direction_arg,
other_sharpness_arg
});
at::checkAllContiguous(__func__, {
grad_out_arg,
intensity_arg,
direction_arg,
sharpness_arg,
other_intensity_arg,
other_direction_arg,
other_sharpness_arg
});
at::checkAllSameType(__func__, {
grad_out_arg,
intensity_arg,
direction_arg,
sharpness_arg,
other_intensity_arg,
other_direction_arg,
other_sharpness_arg
});

const int num_sg = intensity.size(0);
const int num_other = other_intensity.size(0);

at::checkSize(__func__, grad_out_arg, {num_sg, 3});
at::checkSize(__func__, intensity_arg, {num_sg, 3});
at::checkSize(__func__, direction_arg, {num_sg, 3});
at::checkSize(__func__, sharpness_arg, {num_sg});
at::checkSize(__func__, other_intensity_arg, {num_other, 3});
at::checkSize(__func__, other_direction_arg, {num_other, 3});
at::checkSize(__func__, other_sharpness_arg, {num_other});

auto grad_intensity = at::zeros_like(intensity);
auto grad_direction = at::zeros_like(direction);
auto grad_sharpness = at::zeros_like(sharpness);
auto grad_other_intensity = at::zeros_like(other_intensity);
auto grad_other_direction = at::zeros_like(other_direction);
auto grad_other_sharpness = at::zeros_like(other_sharpness);

#ifdef WITH_CUDA
if (num_sg > 0) {
unbatched_reduced_sg_inner_product_backward_cuda_impl(
grad_out,
intensity,
direction,
sharpness,
other_intensity,
other_direction,
other_sharpness,
grad_intensity,
grad_direction,
grad_sharpness,
grad_other_intensity,
grad_other_direction,
grad_other_sharpness
);
}
#else
KAOLIN_NO_CUDA_ERROR(__func__);
#endif // WITH_CUDA
return {grad_intensity, grad_direction, grad_sharpness,
grad_other_intensity, grad_other_direction, grad_other_sharpness};
}

} // namespace kaolin

0 comments on commit 609671b

Please sign in to comment.