Get Ready to be Blown Away! NVIDIA's Crazy New Neural Engine is Redefining Realism in Graphics!

[FurkanGozukara]

Get Ready to be Blown Away! NVIDIA's Crazy New Neural Engine is Redefining Realism in Graphics!

Full tutorial: https://www.youtube.com/watch?v=RTWbCuDhC3c

Real-Time Neural Appearance Models. Hello everyone! Today, we're talking about the latest breakthrough from #NVIDIA. They've developed a new neural engine that can render ultra-realistic video in real-time. This means we're entering a new era of graphics and video processing that's faster and more advanced than ever before. In this video, we'll explore what this new technology can do, and how it's set to revolutionize a range of industries. So, without further ado, let's dive in!

Our Discord server ⤵️

https://bit.ly/SECoursesDiscord

If I have been of assistance to you and you would like to show your support for my work, please consider becoming a patron on 🥰 ⤵️

https://www.patreon.com/SECourses

Technology & Science: News, Tips, Tutorials, Tricks, Best Applications, Guides, Reviews ⤵️

https://www.youtube.com/playlist?list=PL_pbwdIyffsnkay6X91BWb9rrfLATUMr3

Playlist of StableDiffusion Tutorials, Automatic1111 and Google Colab Guides, DreamBooth, Textual Inversion / Embedding, LoRA, AI Upscaling, Pix2Pix, Img2Img ⤵️

https://www.youtube.com/playlist?list=PL_pbwdIyffsmclLl0O144nQRnezKlNdx3

Paper and the video link ⤵️

https://research.nvidia.com/labs/rtr/neural_appearance_models/

Info from the paper

Abstract

We present a complete system for real-time #rendering of scenes with complex appearance previously reserved for offline use. This is achieved with a combination of algorithmic and system level innovations.

Our appearance model utilizes learned hierarchical textures that are interpreted using neural decoders, which produce reflectance values and importance-sampled directions. To best utilize the modeling capacity of the decoders, we equip the decoders with two graphics priors. The first prior—transformation of directions into learned shading frames—facilitates accurate reconstruction of mesoscale effects. The second prior—a microfacet sampling distribution—allows the neural decoder to perform importance sampling efficiently. The resulting appearance model supports anisotropic sampling and level-of-detail rendering, and allows baking deeply layered material graphs into a compact unified neural representation.

By exposing hardware accelerated tensor operations to ray tracing shaders, we show that it is possible to inline and execute the neural decoders efficiently inside a real-time path tracer. We analyze scalability with increasing number of neural materials and propose to improve performance using code optimized for coherent and divergent execution. Our neural material shaders can be over an order of magnitude faster than non-neural layered materials. This opens up the door for using film-quality visuals in real-time applications such as games and live previews.

1 INTRODUCTION

Recent progress in rendering algorithms, light transport methods, and ray tracing hardware have pushed the limits of image quality that can be achieved in real time. However, progress in real-time material models has noticeably lagged behind. While deeply layered materials and sophisticated node graphs are commonplace in offline rendering, such approaches are often far too costly to be used in real-time applications. Aside from computational cost, sophisticated materials pose additional challenges for importance sampling and filtering: highly detailed materials will alias severely under minification, and the complex multi-lobe reflectance of layered materials causes high variance if not sampled properly.

Recent work in neural appearance modelling [Kuznetsov et al. 2022; Sztrajman et al. 2021; Zheng et al. 2021] has shown that multilayer perceptrons (MLPs) can be an effective tool for appearance modelling, importance sampling, and filtering. Nevertheless, these models do not support film-quality appearance; a scalable solution that can handle high-fidelity visuals at real time has yet to be demonstrated.

In this paper, we set our goal accordingly: to render film-quality materials, such as those used in the #VFX industry, in real time. These materials prioritize realism and visual fidelity, relying on very high-resolution textures. Layering of reflectance components, rather than an uber-shader, is used to generate material appearance yielding arbitrary BRDF combinations with hundreds of parameters. For these reasons, porting to real-time application is challenging.

In order to render film-quality appearance in real time we i) carefully cherry-pick components from prior works, ii) introduce algorithmic innovations, and iii) develop a scalable solution for inlining neural networks in the innermost rendering loop, both for classical rasterization and path tracing. We choose to forgo editability in favor of performance, effectively “baking” the reference material into a neural texture interpreted by neural networks. Our model can thus be viewed as an optimized representation for fast rendering, which is baked (via optimization) after editing has taken place.

Video Transcription

• 00:00:00 Greetings everyone. NVIDIA's New Neural Engine  is going to shock the entire Real-Time Graphics  

• 00:00:06 industry! The article and the original  video link will be in the description.  

• 00:00:10 Let see their supplementary amazing video while  listening the paper from a reporter style voice.

• 00:00:16 According to this paper, a complete  system for real-time rendering of scenes  

• 00:00:20 with complex appearance, which was previously  reserved for offline use, has been introduced.

• 00:00:26 This system is a result of a combination of  algorithmic and system level innovations.

• 00:00:31 The appearance model of the system  incorporates learned hierarchical textures,  

• 00:00:35 which are interpreted using neural decoders,  

• 00:00:38 producing reflectance values and  importance-sampled directions.

• 00:00:42 To enhance the modeling capacity of the decoders,  two graphics priors are equipped - the first one  

• 00:00:47 facilitates accurate reconstruction of mesoscale  effects by transforming directions into learned  

• 00:00:53 shading frames, and the second one allows  the neural decoder to perform importance  

• 00:00:58 sampling efficiently by incorporating  a microfacet sampling distribution.  

• 00:01:02 The resulting appearance model allows anisotropic  sampling and level-of-detail rendering,  

• 00:01:08 and enables baking deeply layered material graphs  into a compact unified neural representation.

• 00:01:14 By exposing hardware accelerated tensor operations  to ray tracing shaders, the system has made it  

• 00:01:20 possible to efficiently inline and execute the  neural decoders inside a real-time path tracer.

• 00:01:26 The scalability analysis of the  system with increasing number  

• 00:01:29 of neural materials has been carried out,  and a proposal to enhance the performance  

• 00:01:34 using code optimized for coherent and  divergent execution has been made.

• 00:01:40 The neural material shaders of  this system have been reported to  

• 00:01:43 be over an order of magnitude faster  than non-neural layered materials.

• 00:01:48 This development opens up the  possibility of using film-quality  

• 00:01:52 visuals in real-time applications  such as games and live previews.

• 00:01:56 In conclusion, the system utilizes  appearance models, neural networks,  

• 00:02:01 and real-time rendering to achieve a breakthrough  in the field of graphics and visual effects.

• 00:02:07 According to this research, advances in  rendering algorithms, light transport methods,  

• 00:02:12 and ray tracing hardware have significantly  improved real-time image quality .

• 00:02:18 However, real-time material models  have not kept up with the progress,  

• 00:02:22 as highly detailed materials and  complex node graphs used in offline  

• 00:02:27 rendering are often too expensive to  be used in real-time applications.

• 00:02:31 In addition to computational cost  , sophisticated materials pose  

• 00:02:36 challenges for importance sampling and filtering.

• 00:02:38 Recent work in neural appearance modeling has  shown promise in addressing these challenges.

• 00:02:45 Multi-layer perceptrons have been  effective in appearance modeling,  

• 00:02:49 importance sampling, and filtering.

• 00:02:51 However, these models do not yet support  film-quality appearance in real-time.

• 00:02:56 The goal of this paper is to  develop a scalable solution to  

• 00:03:00 render film-quality materials used  in the VFX industry in real-time,  

• 00:03:05 despite the challenges posed by high-resolution  textures and layered reflectance components.

• 00:03:10 The approach involves "baking"  reference material into a neural  

• 00:03:14 texture interpreted by neural networks,  sacrificing editability for performance.

• 00:03:19 The model consists of an encoder and two  decoders, with the neural texture in between.

• 00:03:25 The encoder maps BRDF  parameters to a latent space,  

• 00:03:30 and the latent texture is decoded using  two networks for evaluation and sampling.

• 00:03:35 Algorithmic innovations involve embedding  fixed-function elements or graphics priors  

• 00:03:41 in the two neural decoders to efficiently utilize  the limited expressive power of small networks.

• 00:03:47 A method for inlining fully fused neural  networks in rendering code is also presented.

• 00:03:53 This is the first complete and scalable system  

• 00:03:56 for running neural material shaders  inside real-time shading languages.

• 00:04:00 The execution model efficiently  handles divergent code paths and  

• 00:04:04 allows fast inferencing in any shader stage,  including ray tracing and fragment shaders.

• 00:04:11 The system can render complex materials  that are the norm in offline rendering  

• 00:04:15 and achieve the visual fidelity of highly  detailed assets with layered materials.

• 00:04:21 The joint evolution of models and systems is  crucial to bringing neural shaders to real-time,  

• 00:04:26 and this paper provides a solid  foundation for such developments.

• 00:04:31 Moreover, the authors aim to provide an optimized  representation for fast rendering, which is  

• 00:04:37 achieved by "baking" the reference material into  a neural texture interpreted by neural networks.

• 00:04:43 The resulting model is a system  that fits their criteria,  

• 00:04:46 but it naturally comes with limitations, which  the authors hope to address in future work.

• 00:04:51 The model consists of an encoder and two decoders,  with the neural (latent) texture in between.

• 00:04:58 The encoder maps BRDF parameters to a latent  space, which is then converted into a single  

• 00:05:05 multi-channel latent texture.

• 00:05:07 The latent texture is decoded using two networks,  

• 00:05:10 an evaluation network that infers the  BRDF value for a given pair of directions,  

• 00:05:14 and a sampling network that maps random  numbers to sampled (outgoing) directions.

• 00:05:20 The authors introduce several algorithmic  

• 00:05:22 innovations to support materials  with high-resolution textures.

• 00:05:26 For example, they embed fixed-function elements,  

• 00:05:30 graphics priors, in the two neural  decoders, such as a standard rotation  

• 00:05:35 operation between trainable components of the  BRDF decoder to handle normal mapped surfaces.

• 00:05:40 They also utilize a network-driven microfacet  distribution for importance sampling.

• 00:05:46 These priors are necessary to efficiently utilize  the limited expressive power of small networks.

• 00:05:52 At the system level, the authors present an  efficient method for inlining fully fused  

• 00:05:57 neural networks in rendering code, which  is the first complete and scalable system  

• 00:06:02 for running neural material shaders  inside real-time shading languages.

• 00:06:07 The authors' execution model utilizes tensor  operations whenever possible and efficiently  

• 00:06:12 handles divergent code paths, allowing  fast inferencing in any shader stage,  

• 00:06:18 including ray tracing and fragment shaders,  

• 00:06:20 which is essential for adoption in game  engines and interactive applications.

• 00:06:24 The authors demonstrate graceful  cost scaling in scenes with many  

• 00:06:28 different neural materials running  inside a real-time path tracer.

• 00:06:33 Their neural model has a fixed evaluation  cost, independent of the material complexity,  

• 00:06:39 allowing them to render complex materials  that are the norm in offline rendering.

• 00:06:44 To demonstrate this, they authored  several highly detailed assets with  

• 00:06:48 layered materials that provide visual detail  down to a 10 centimeters viewing distance.

• 00:06:53 They were able to reproduce the visual  fidelity of such complex assets,  

• 00:06:57 with shading being up to 10 times faster than the  original, moderately optimized shading models,  

• 00:07:03 while also providing additional  sampling and filtering facilities.

• 00:07:07 The authors' work represents  a significant step forward  

• 00:07:10 in rendering film-quality materials in real time.

• 00:07:13 By developing a system that is both  efficient and scalable, the authors  

• 00:07:17 have laid a solid foundation  for future work in this area.

• 00:07:21 They believe that the joint evolution of models  and systems is crucial to bringing neural shaders  

• 00:07:27 to real-time, and they have built their system  to serve as a platform for this evolution.